Oct 30 00:28:52.603479 ip-10-0-128-7 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Oct 30 00:28:52.603490 ip-10-0-128-7 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Oct 30 00:28:52.603497 ip-10-0-128-7 systemd[1]: kubelet.service: Failed with result 'resources'.
Oct 30 00:28:52.603791 ip-10-0-128-7 systemd[1]: Failed to start Kubernetes Kubelet.
Oct 30 00:29:03.951530 ip-10-0-128-7 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Oct 30 00:29:03.951551 ip-10-0-128-7 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot c43aa09fcd5a4eb6bc6e43c1a088b226 --
Oct 30 00:31:01.751714 ip-10-0-128-7 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 00:31:02.146714 ip-10-0-128-7 kubenswrapper[2579]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 00:31:02.146714 ip-10-0-128-7 kubenswrapper[2579]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 00:31:02.146714 ip-10-0-128-7 kubenswrapper[2579]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 00:31:02.146714 ip-10-0-128-7 kubenswrapper[2579]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 00:31:02.154231 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.154135    2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 00:31:02.154231 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.154225    2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 00:31:02.154231 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.154230    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 00:31:02.154231 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.154233    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 00:31:02.154231 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.154237    2579 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 00:31:02.154231 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.154240    2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 00:31:02.154231 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.154244    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 00:31:02.154513 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.154247    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 00:31:02.154513 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.154250    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 00:31:02.154513 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.154253    2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 00:31:02.154513 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.154258    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 00:31:02.154513 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.154261    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 00:31:02.154513 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.154264    2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 00:31:02.154513 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.154267    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 00:31:02.154513 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.154270    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 00:31:02.154513 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.154272    2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 00:31:02.154513 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.154275    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 00:31:02.154513 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.154278    2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 00:31:02.154513 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.154281    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 00:31:02.154513 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.154284    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 00:31:02.154513 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.154287    2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 00:31:02.154513 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.154290    2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 00:31:02.154513 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.154293    2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 00:31:02.154513 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.154295    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 00:31:02.154513 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.154298    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 00:31:02.154513 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.154300    2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 00:31:02.154513 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.154309    2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 00:31:02.154982 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.154312    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 00:31:02.154982 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.154315    2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 00:31:02.154982 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.154317    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 00:31:02.154982 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.154319    2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 00:31:02.154982 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.154322    2579 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 00:31:02.154982 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.154325    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 00:31:02.154982 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.154327    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 00:31:02.154982 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.154329    2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 00:31:02.154982 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.154332    2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 00:31:02.154982 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.154335    2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 00:31:02.154982 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.154337    2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 00:31:02.154982 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.154339    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 00:31:02.154982 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.154342    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 00:31:02.154982 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.154346    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 00:31:02.154982 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.154349    2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 00:31:02.154982 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.154351    2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 00:31:02.154982 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.154354    2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 00:31:02.154982 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.154357    2579 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 00:31:02.154982 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.154360    2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 00:31:02.155457 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.154363    2579 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 00:31:02.155457 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.154366    2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 00:31:02.155457 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.154369    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 00:31:02.155457 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.154371    2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 00:31:02.155457 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.154375    2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 00:31:02.155457 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.154378    2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 00:31:02.155457 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.154380    2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 00:31:02.155457 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.154383    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 00:31:02.155457 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.154386    2579 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 00:31:02.155457 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.154388    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 00:31:02.155457 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.154392    2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 00:31:02.155457 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.154395    2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 00:31:02.155457 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.154397    2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 00:31:02.155457 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.154400    2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 00:31:02.155457 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.154402    2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 00:31:02.155457 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.154406    2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 00:31:02.155457 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.154408    2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 00:31:02.155457 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.154411    2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 00:31:02.155457 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.154428    2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 00:31:02.155908 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.154433    2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 00:31:02.155908 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.154437    2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 00:31:02.155908 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.154440    2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 00:31:02.155908 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.154443    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 00:31:02.155908 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.154446    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 00:31:02.155908 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.154448    2579 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 00:31:02.155908 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.154451    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 00:31:02.155908 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.154454    2579 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 00:31:02.155908 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.154457    2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 00:31:02.155908 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.154460    2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 00:31:02.155908 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.154463    2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 00:31:02.155908 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.154465    2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 00:31:02.155908 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.154468    2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 00:31:02.155908 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.154471    2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 00:31:02.155908 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.154473    2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 00:31:02.155908 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.154476    2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 00:31:02.155908 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.154479    2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 00:31:02.155908 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.154482    2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 00:31:02.155908 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.154486    2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 00:31:02.155908 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.154490    2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 00:31:02.156444 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.154493    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 00:31:02.156444 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.154495    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 00:31:02.156444 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.154498    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 00:31:02.156444 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.154501    2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 00:31:02.156444 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.154503    2579 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 00:31:02.156444 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.154506    2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 00:31:02.156444 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.154509    2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 00:31:02.156444 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.154512    2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 00:31:02.156444 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.154516    2579 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 00:31:02.156444 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.154518    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 00:31:02.156444 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.154521    2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 00:31:02.156444 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.154524    2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 00:31:02.156444 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.154526    2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 00:31:02.156444 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.154529    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 00:31:02.156444 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.154531    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 00:31:02.156444 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.154534    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 00:31:02.156444 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.154537    2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 00:31:02.156444 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.154540    2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 00:31:02.156444 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.154542    2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 00:31:02.156444 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.154545    2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 00:31:02.156914 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.154548    2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 00:31:02.156914 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.155377    2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 00:31:02.156914 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.155385    2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 00:31:02.156914 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.155399    2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 00:31:02.156914 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.155403    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 00:31:02.156914 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.155406    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 00:31:02.156914 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.155409    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 00:31:02.156914 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.155411    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 00:31:02.156914 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.155432    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 00:31:02.156914 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.155436    2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 00:31:02.156914 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.155440    2579 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 00:31:02.156914 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.155445    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 00:31:02.156914 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.155448    2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 00:31:02.156914 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.155450    2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 00:31:02.156914 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.155453    2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 00:31:02.156914 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.155456    2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 00:31:02.156914 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.155460    2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 00:31:02.156914 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.155464    2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 00:31:02.156914 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.155468    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 00:31:02.157372 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.155471    2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 00:31:02.157372 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.155475    2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 00:31:02.157372 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.155478    2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 00:31:02.157372 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.155481    2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 00:31:02.157372 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.155483    2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 00:31:02.157372 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.155486    2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 00:31:02.157372 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.155489    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 00:31:02.157372 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.155492    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 00:31:02.157372 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.155495    2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 00:31:02.157372 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.155497    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 00:31:02.157372 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.155500    2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 00:31:02.157372 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.155502    2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 00:31:02.157372 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.155505    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 00:31:02.157372 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.155508    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 00:31:02.157372 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.155510    2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 00:31:02.157372 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.155513    2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 00:31:02.157372 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.155516    2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 00:31:02.157372 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.155518    2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 00:31:02.157372 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.155521    2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 00:31:02.157865 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.155523    2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 00:31:02.157865 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.155526    2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 00:31:02.157865 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.155529    2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 00:31:02.157865 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.155531    2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 00:31:02.157865 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.155534    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 00:31:02.157865 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.155536    2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 00:31:02.157865 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.155539    2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 00:31:02.157865 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.155542    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 00:31:02.157865 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.155545    2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 00:31:02.157865 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.155547    2579 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 00:31:02.157865 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.155550    2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 00:31:02.157865 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.155552    2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 00:31:02.157865 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.155555    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 00:31:02.157865 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.155558    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 00:31:02.157865 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.155561    2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 00:31:02.157865 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.155563    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 00:31:02.157865 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.155566    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 00:31:02.157865 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.155568    2579 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 00:31:02.157865 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.155571    2579 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 00:31:02.157865 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.155574    2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 00:31:02.157865 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.155576    2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 00:31:02.158367 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.155579    2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 00:31:02.158367 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.155581    2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 00:31:02.158367 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.155584    2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 00:31:02.158367 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.155586    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 00:31:02.158367 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.155589    2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 00:31:02.158367 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.155591    2579 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 00:31:02.158367 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.155594    2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 00:31:02.158367 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.155596    2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 00:31:02.158367 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.155601    2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 00:31:02.158367 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.155605    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 00:31:02.158367 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.155608    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 00:31:02.158367 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.155611    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 00:31:02.158367 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.155614    2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 00:31:02.158367 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.155617    2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 00:31:02.158367 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.155621    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 00:31:02.158367 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.155623    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 00:31:02.158367 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.155626    2579 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 00:31:02.158367 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.155628    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 00:31:02.158367 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.155632    2579 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 00:31:02.158838 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.155635    2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 00:31:02.158838 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.155637    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 00:31:02.158838 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.155640    2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 00:31:02.158838 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.155642    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 00:31:02.158838 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.155645    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 00:31:02.158838 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.155648    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 00:31:02.158838 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.155650    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 00:31:02.158838 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.155653    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 00:31:02.158838 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.155655    2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 00:31:02.158838 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.155661    2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 00:31:02.158838 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.155663    2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 00:31:02.158838 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.155666    2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 00:31:02.158838 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.155669    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 00:31:02.158838 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.155671    2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 00:31:02.158838 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.155674    2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 00:31:02.158838 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.155676    2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 00:31:02.158838 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.155679    2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 00:31:02.158838 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.155681    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 00:31:02.158838 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.155684    2579 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 00:31:02.158838 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.155687    2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 00:31:02.159322 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.155689    2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 00:31:02.159322 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.155692    2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 00:31:02.159322 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.155694    2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 00:31:02.159322 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.155696    2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 00:31:02.159322 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.155699    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 00:31:02.159322 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.155701    2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 00:31:02.159322 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.155704    2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 00:31:02.159322 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.155706    2579 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 00:31:02.159322 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.155710    2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 00:31:02.159322 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.156237    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 00:31:02.159322 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.156243    2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 00:31:02.159322 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.156247    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 00:31:02.159322 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.156250    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 00:31:02.159322 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.156253    2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 00:31:02.159322 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.156255    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 00:31:02.159322 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.156258    2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 00:31:02.159322 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.156261    2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 00:31:02.159322 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.156264    2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 00:31:02.159322 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.156267    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 00:31:02.159322 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.156269    2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 00:31:02.159832 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.156272    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 00:31:02.159832 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.156276    2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 00:31:02.159832 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.156279    2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 00:31:02.159832 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.156282    2579 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 00:31:02.159832 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.156285    2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 00:31:02.159832 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.156288    2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 00:31:02.159832 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.156290    2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 00:31:02.159832 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.156293    2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 00:31:02.159832 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.156296    2579 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 00:31:02.159832 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.156299    2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 00:31:02.159832 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.156302    2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 00:31:02.159832 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.156305    2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 00:31:02.159832 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.156307    2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 00:31:02.159832 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.156309    2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 00:31:02.159832 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.156312    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 00:31:02.159832 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.156315    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 00:31:02.159832 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.156317    2579 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 00:31:02.159832 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.156320    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 00:31:02.159832 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.156322    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 00:31:02.159832 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.156325    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 00:31:02.160307 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.156328    2579 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 00:31:02.160307 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.156330    2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 00:31:02.160307 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.156334    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 00:31:02.160307 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.156337    2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 00:31:02.160307 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.156340    2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 00:31:02.160307 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.156343    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 00:31:02.160307 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.156345    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 00:31:02.160307 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.156348    2579 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 00:31:02.160307 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.156350    2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 00:31:02.160307 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.156353    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 00:31:02.160307 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.156356    2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 00:31:02.160307 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.156358    2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 00:31:02.160307 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.156361    2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 00:31:02.160307 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.156365    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 00:31:02.160307 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.156368    2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 00:31:02.160307 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.156370    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 00:31:02.160307 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.156373    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 00:31:02.160307 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.156375    2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 00:31:02.160307 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.156378    2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 00:31:02.160307 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.156380    2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 00:31:02.160821 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.156383    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 00:31:02.160821 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.156385    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 00:31:02.160821 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.156388    2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 00:31:02.160821 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.156390    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 00:31:02.160821 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.156393    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 00:31:02.160821 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.156395    2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 00:31:02.160821 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.156399    2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 00:31:02.160821 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.156403    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 00:31:02.160821 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.156406    2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 00:31:02.160821 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.156408    2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 00:31:02.160821 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.156411    2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 00:31:02.160821 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.156427    2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 00:31:02.160821 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.156430    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 00:31:02.160821 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.156441    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 00:31:02.160821 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.156444    2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 00:31:02.160821 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.156447    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 00:31:02.160821 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.156449    2579 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 00:31:02.160821 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.156453    2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 00:31:02.160821 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.156456    2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 00:31:02.161314 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.156459    2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 00:31:02.161314 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.156461    2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 00:31:02.161314 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.156464    2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 00:31:02.161314 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.156466    2579 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 00:31:02.161314 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.156469    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 00:31:02.161314 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.156473    2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 00:31:02.161314 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.156479    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 00:31:02.161314 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.156482    2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 00:31:02.161314 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.156485    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 00:31:02.161314 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.156488    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 00:31:02.161314 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.156491    2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 00:31:02.161314 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.156494    2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 00:31:02.161314 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.156496    2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 00:31:02.161314 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.156499    2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 00:31:02.161314 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.156502    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 00:31:02.161314 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.156504    2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 00:31:02.161314 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.156507    2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 00:31:02.161314 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.156509    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 00:31:02.161314 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.156513    2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 00:31:02.161881 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.156515    2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 00:31:02.161881 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.156518    2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 00:31:02.161881 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.156520    2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 00:31:02.161881 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.156523    2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 00:31:02.161881 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.156525    2579 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 00:31:02.161881 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.156528    2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 00:31:02.161881 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.156530    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 00:31:02.161881 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.156533    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 00:31:02.161881 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.156536    2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 00:31:02.161881 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.156538    2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 00:31:02.161881 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.156541    2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 00:31:02.161881 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.156543    2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 00:31:02.161881 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.156546    2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 00:31:02.161881 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.156549    2579 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 00:31:02.161881 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.156551    2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 00:31:02.161881 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.156554    2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 00:31:02.161881 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.156556    2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 00:31:02.161881 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.157411    2579 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 00:31:02.161881 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.157434    2579 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 00:31:02.161881 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.157443    2579 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 00:31:02.161881 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.157448    2579 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 00:31:02.162381 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.157453    2579 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 00:31:02.162381 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.157456    2579 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 00:31:02.162381 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.157461    2579 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 00:31:02.162381 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.157465    2579 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 00:31:02.162381 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.157468    2579 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 00:31:02.162381 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.157471    2579 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 00:31:02.162381 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.157475    2579 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 00:31:02.162381 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.157478    2579 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 00:31:02.162381 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.157481    2579 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 00:31:02.162381 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.157484    2579 flags.go:64] FLAG: --cgroup-root=""
Oct 30 00:31:02.162381 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.157488    2579 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 00:31:02.162381 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.157491    2579 flags.go:64] FLAG: --client-ca-file=""
Oct 30 00:31:02.162381 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.157494    2579 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 00:31:02.162381 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.157497    2579 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 00:31:02.162381 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.157501    2579 flags.go:64] FLAG: --cluster-domain=""
Oct 30 00:31:02.162381 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.157504    2579 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 00:31:02.162381 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.157507    2579 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 00:31:02.162381 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.157510    2579 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 00:31:02.162381 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.157514    2579 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 00:31:02.162381 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.157518    2579 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 00:31:02.162381 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.157521    2579 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 00:31:02.162381 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.157524    2579 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 00:31:02.162381 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.157528    2579 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 00:31:02.162951 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.157531    2579 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 00:31:02.162951 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.157536    2579 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 00:31:02.162951 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.157541    2579 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 00:31:02.162951 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.157544    2579 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 00:31:02.162951 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.157547    2579 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 00:31:02.162951 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.157552    2579 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 00:31:02.162951 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.157555    2579 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 00:31:02.162951 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.157558    2579 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 00:31:02.162951 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.157561    2579 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 00:31:02.162951 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.157564    2579 flags.go:64] FLAG: --enable-server="true"
Oct 30 00:31:02.162951 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.157567    2579 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 00:31:02.162951 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.157572    2579 flags.go:64] FLAG: --event-burst="100"
Oct 30 00:31:02.162951 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.157575    2579 flags.go:64] FLAG: --event-qps="50"
Oct 30 00:31:02.162951 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.157578    2579 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 00:31:02.162951 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.157581    2579 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 00:31:02.162951 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.157584    2579 flags.go:64] FLAG: --eviction-hard=""
Oct 30 00:31:02.162951 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.157588    2579 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 00:31:02.162951 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.157591    2579 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 00:31:02.162951 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.157594    2579 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 00:31:02.162951 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.157597    2579 flags.go:64] FLAG: --eviction-soft=""
Oct 30 00:31:02.162951 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.157600    2579 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 00:31:02.162951 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.157603    2579 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 00:31:02.162951 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.157606    2579 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 00:31:02.162951 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.157609    2579 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 00:31:02.162951 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.157612    2579 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 00:31:02.163587 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.157615    2579 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 00:31:02.163587 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.157618    2579 flags.go:64] FLAG: --feature-gates=""
Oct 30 00:31:02.163587 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.157622    2579 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 00:31:02.163587 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.157625    2579 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 00:31:02.163587 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.157628    2579 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 00:31:02.163587 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.157631    2579 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 00:31:02.163587 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.157634    2579 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 00:31:02.163587 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.157638    2579 flags.go:64] FLAG: --help="false"
Oct 30 00:31:02.163587 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.157641    2579 flags.go:64] FLAG: --hostname-override="ip-10-0-128-7.ec2.internal"
Oct 30 00:31:02.163587 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.157646    2579 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 00:31:02.163587 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.157650    2579 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 00:31:02.163587 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.157653    2579 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 00:31:02.163587 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.157656    2579 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 00:31:02.163587 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.157660    2579 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 00:31:02.163587 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.157663    2579 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 00:31:02.163587 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.157666    2579 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 00:31:02.163587 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.157669    2579 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 00:31:02.163587 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.157671    2579 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 00:31:02.163587 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.157674    2579 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 00:31:02.163587 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.157677    2579 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 00:31:02.163587 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.157680    2579 flags.go:64] FLAG: --kube-reserved=""
Oct 30 00:31:02.163587 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.157683    2579 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 00:31:02.163587 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.157686    2579 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 00:31:02.163587 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.157689    2579 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 00:31:02.164648 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.157692    2579 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 00:31:02.164648 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.157694    2579 flags.go:64] FLAG: --lock-file=""
Oct 30 00:31:02.164648 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.157697    2579 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 00:31:02.164648 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.157700    2579 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 00:31:02.164648 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.157703    2579 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 00:31:02.164648 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.157709    2579 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 00:31:02.164648 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.157712    2579 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 00:31:02.164648 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.157715    2579 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 00:31:02.164648 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.157718    2579 flags.go:64] FLAG: --logging-format="text"
Oct 30 00:31:02.164648 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.157721    2579 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 00:31:02.164648 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.157724    2579 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 00:31:02.164648 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.157727    2579 flags.go:64] FLAG: --manifest-url=""
Oct 30 00:31:02.164648 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.157730    2579 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 00:31:02.164648 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.157734    2579 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 00:31:02.164648 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.157737    2579 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 00:31:02.164648 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.157741    2579 flags.go:64] FLAG: --max-pods="110"
Oct 30 00:31:02.164648 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.157744    2579 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 00:31:02.164648 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.157749    2579 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 00:31:02.164648 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.157752    2579 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 00:31:02.164648 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.157755    2579 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 00:31:02.164648 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.157758    2579 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 00:31:02.164648 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.157761    2579 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 00:31:02.164648 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.157764    2579 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 00:31:02.164648 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.157771    2579 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 00:31:02.164648 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.157775    2579 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 00:31:02.165776 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.157778    2579 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 00:31:02.165776 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.157781    2579 flags.go:64] FLAG: --pod-cidr=""
Oct 30 00:31:02.165776 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.157784    2579 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 00:31:02.165776 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.157787    2579 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 00:31:02.165776 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.157790    2579 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 00:31:02.165776 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.157793    2579 flags.go:64] FLAG: --port="10250"
Oct 30 00:31:02.165776 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.157796    2579 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 00:31:02.165776 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.157799    2579 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-090d92febc2f6edca"
Oct 30 00:31:02.165776 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.157802    2579 flags.go:64] FLAG: --qos-reserved=""
Oct 30 00:31:02.165776 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.157805    2579 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 00:31:02.165776 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.157808    2579 flags.go:64] FLAG: --register-node="true"
Oct 30 00:31:02.165776 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.157812    2579 flags.go:64] FLAG: --register-with-taints=""
Oct 30 00:31:02.165776 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.157815    2579 flags.go:64] FLAG: --registry-burst="10"
Oct 30 00:31:02.165776 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.157818    2579 flags.go:64] FLAG: --registry-qps="5"
Oct 30 00:31:02.165776 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.157821    2579 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 00:31:02.165776 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.157824    2579 flags.go:64] FLAG: --reserved-memory=""
Oct 30 00:31:02.165776 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.157828    2579 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 00:31:02.165776 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.157831    2579 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 00:31:02.165776 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.157834    2579 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 00:31:02.165776 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.157837    2579 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 00:31:02.165776 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.157840    2579 flags.go:64] FLAG: --runonce="false"
Oct 30 00:31:02.165776 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.157842    2579 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 00:31:02.165776 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.157847    2579 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 00:31:02.165776 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.157850    2579 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 00:31:02.165776 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.157855    2579 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 00:31:02.165776 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.157858    2579 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 00:31:02.165776 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.157861    2579 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 00:31:02.166655 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.157864    2579 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 00:31:02.166655 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.157868    2579 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 00:31:02.166655 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.157871    2579 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 00:31:02.166655 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.157874    2579 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 00:31:02.166655 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.157877    2579 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 00:31:02.166655 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.157880    2579 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 00:31:02.166655 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.157883    2579 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 00:31:02.166655 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.157886    2579 flags.go:64] FLAG: --system-cgroups=""
Oct 30 00:31:02.166655 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.157889    2579 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 00:31:02.166655 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.157894    2579 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 00:31:02.166655 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.157897    2579 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 00:31:02.166655 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.157900    2579 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 00:31:02.166655 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.157904    2579 flags.go:64] FLAG: --tls-min-version=""
Oct 30 00:31:02.166655 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.157907    2579 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 00:31:02.166655 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.157910    2579 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 00:31:02.166655 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.157913    2579 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 00:31:02.166655 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.157916    2579 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 00:31:02.166655 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.157919    2579 flags.go:64] FLAG: --v="2"
Oct 30 00:31:02.166655 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.157924    2579 flags.go:64] FLAG: --version="false"
Oct 30 00:31:02.166655 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.157928    2579 flags.go:64] FLAG: --vmodule=""
Oct 30 00:31:02.166655 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.157932    2579 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 00:31:02.166655 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.157936    2579 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 00:31:02.166655 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158074    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 00:31:02.166655 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158077    2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 00:31:02.167600 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158080    2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 00:31:02.167600 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158083    2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 00:31:02.167600 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158086    2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 00:31:02.167600 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158090    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 00:31:02.167600 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158092    2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 00:31:02.167600 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158098    2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 00:31:02.167600 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158101    2579 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 00:31:02.167600 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158103    2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 00:31:02.167600 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158107    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 00:31:02.167600 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158110    2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 00:31:02.167600 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158112    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 00:31:02.167600 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158115    2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 00:31:02.167600 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158118    2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 00:31:02.167600 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158120    2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 00:31:02.167600 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158123    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 00:31:02.167600 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158126    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 00:31:02.167600 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158128    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 00:31:02.167600 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158131    2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 00:31:02.167600 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158133    2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 00:31:02.167600 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158136    2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 00:31:02.168593 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158139    2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 00:31:02.168593 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158141    2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 00:31:02.168593 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158144    2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 00:31:02.168593 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158146    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 00:31:02.168593 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158149    2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 00:31:02.168593 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158152    2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 00:31:02.168593 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158154    2579 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 00:31:02.168593 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158157    2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 00:31:02.168593 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158159    2579 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 00:31:02.168593 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158162    2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 00:31:02.168593 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158164    2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 00:31:02.168593 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158167    2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 00:31:02.168593 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158169    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 00:31:02.168593 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158172    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 00:31:02.168593 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158176    2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 00:31:02.168593 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158181    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 00:31:02.168593 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158184    2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 00:31:02.168593 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158188    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 00:31:02.168593 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158191    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 00:31:02.169250 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158194    2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 00:31:02.169250 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158197    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 00:31:02.169250 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158200    2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 00:31:02.169250 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158203    2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 00:31:02.169250 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158205    2579 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 00:31:02.169250 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158208    2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 00:31:02.169250 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158210    2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 00:31:02.169250 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158213    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 00:31:02.169250 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158215    2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 00:31:02.169250 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158218    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 00:31:02.169250 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158221    2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 00:31:02.169250 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158223    2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 00:31:02.169250 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158226    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 00:31:02.169250 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158229    2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 00:31:02.169250 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158231    2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 00:31:02.169250 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158235    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 00:31:02.169250 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158237    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 00:31:02.169250 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158240    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 00:31:02.169250 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158243    2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 00:31:02.169250 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158246    2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 00:31:02.169973 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158248    2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 00:31:02.169973 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158251    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 00:31:02.169973 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158253    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 00:31:02.169973 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158256    2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 00:31:02.169973 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158258    2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 00:31:02.169973 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158262    2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 00:31:02.169973 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158266    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 00:31:02.169973 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158269    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 00:31:02.169973 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158273    2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 00:31:02.169973 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158276    2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 00:31:02.169973 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158280    2579 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 00:31:02.169973 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158283    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 00:31:02.169973 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158285    2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 00:31:02.169973 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158288    2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 00:31:02.169973 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158290    2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 00:31:02.169973 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158294    2579 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 00:31:02.169973 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158296    2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 00:31:02.169973 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158299    2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 00:31:02.169973 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158301    2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 00:31:02.170768 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158304    2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 00:31:02.170768 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158306    2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 00:31:02.170768 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158309    2579 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 00:31:02.170768 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158311    2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 00:31:02.170768 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158314    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 00:31:02.170768 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158317    2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 00:31:02.170768 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158319    2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 00:31:02.170768 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158322    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 00:31:02.170768 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158324    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 00:31:02.170768 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158327    2579 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 00:31:02.170768 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158329    2579 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 00:31:02.170768 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158332    2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 00:31:02.170768 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158334    2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 00:31:02.170768 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158337    2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 00:31:02.170768 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158339    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 00:31:02.170768 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158342    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 00:31:02.170768 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158344    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 00:31:02.170768 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158347    2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 00:31:02.170768 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158349    2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 00:31:02.170768 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158352    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 00:31:02.171348 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158354    2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 00:31:02.171348 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158358    2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 00:31:02.171348 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158360    2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 00:31:02.171348 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158364    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 00:31:02.171348 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158367    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 00:31:02.171348 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158370    2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 00:31:02.171348 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.158397    2579 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 00:31:02.171348 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158445    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 00:31:02.171348 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158450    2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 00:31:02.171348 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158454    2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 00:31:02.171348 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158458    2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 00:31:02.171348 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158461    2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 00:31:02.171348 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158463    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 00:31:02.171348 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158466    2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 00:31:02.171348 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158469    2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 00:31:02.171348 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158471    2579 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 00:31:02.171916 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158474    2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 00:31:02.171916 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158477    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 00:31:02.171916 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158480    2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 00:31:02.171916 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158482    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 00:31:02.171916 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158485    2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 00:31:02.171916 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158487    2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 00:31:02.171916 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158490    2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 00:31:02.171916 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158492    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 00:31:02.171916 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158495    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 00:31:02.171916 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158498    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 00:31:02.171916 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158500    2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 00:31:02.171916 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158503    2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 00:31:02.171916 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158505    2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 00:31:02.171916 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158508    2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 00:31:02.171916 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158511    2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 00:31:02.171916 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158513    2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 00:31:02.171916 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158516    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 00:31:02.171916 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158518    2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 00:31:02.171916 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158522    2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 00:31:02.172375 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158526    2579 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 00:31:02.172375 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158528    2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 00:31:02.172375 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158531    2579 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 00:31:02.172375 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158533    2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 00:31:02.172375 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158536    2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 00:31:02.172375 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158538    2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 00:31:02.172375 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158541    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 00:31:02.172375 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158544    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 00:31:02.172375 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158548    2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 00:31:02.172375 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158552    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 00:31:02.172375 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158556    2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 00:31:02.172375 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158560    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 00:31:02.172375 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158562    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 00:31:02.172375 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158565    2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 00:31:02.172375 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158568    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 00:31:02.172375 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158570    2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 00:31:02.172375 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158573    2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 00:31:02.172375 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158575    2579 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 00:31:02.172375 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158578    2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 00:31:02.172375 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158580    2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 00:31:02.172937 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158583    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 00:31:02.172937 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158585    2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 00:31:02.172937 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158588    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 00:31:02.172937 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158590    2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 00:31:02.172937 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158593    2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 00:31:02.172937 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158596    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 00:31:02.172937 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158598    2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 00:31:02.172937 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158601    2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 00:31:02.172937 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158603    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 00:31:02.172937 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158606    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 00:31:02.172937 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158609    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 00:31:02.172937 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158613    2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 00:31:02.172937 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158615    2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 00:31:02.172937 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158618    2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 00:31:02.172937 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158620    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 00:31:02.172937 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158623    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 00:31:02.172937 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158626    2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 00:31:02.172937 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158628    2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 00:31:02.172937 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158632    2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 00:31:02.173528 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158636    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 00:31:02.173528 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158639    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 00:31:02.173528 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158642    2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 00:31:02.173528 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158644    2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 00:31:02.173528 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158647    2579 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 00:31:02.173528 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158650    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 00:31:02.173528 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158652    2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 00:31:02.173528 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158655    2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 00:31:02.173528 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158657    2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 00:31:02.173528 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158660    2579 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 00:31:02.173528 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158663    2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 00:31:02.173528 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158665    2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 00:31:02.173528 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158668    2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 00:31:02.173528 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158671    2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 00:31:02.173528 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158673    2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 00:31:02.173528 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158676    2579 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 00:31:02.173528 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158678    2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 00:31:02.173528 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158681    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 00:31:02.173528 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158684    2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 00:31:02.173994 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158686    2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 00:31:02.173994 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158689    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 00:31:02.173994 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158692    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 00:31:02.173994 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158694    2579 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 00:31:02.173994 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158697    2579 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 00:31:02.173994 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158701    2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 00:31:02.173994 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158704    2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 00:31:02.173994 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158706    2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 00:31:02.173994 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158709    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 00:31:02.173994 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158711    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 00:31:02.173994 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158714    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 00:31:02.173994 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158717    2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 00:31:02.173994 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158719    2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 00:31:02.173994 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158722    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 00:31:02.173994 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158724    2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 00:31:02.173994 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158727    2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 00:31:02.173994 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158730    2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 00:31:02.173994 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158734    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 00:31:02.173994 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158736    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 00:31:02.174753 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.158739    2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 00:31:02.174753 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.166185    2579 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 00:31:02.174753 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.166208    2579 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 00:31:02.174753 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.166868    2579 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 00:31:02.174753 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.166881    2579 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 00:31:02.174753 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.166974    2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 00:31:02.174753 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.166984    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 00:31:02.174753 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.166989    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 00:31:02.174753 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.166994    2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 00:31:02.174753 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.166998    2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 00:31:02.174753 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.167003    2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 00:31:02.174753 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.167007    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 00:31:02.174753 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.167011    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 00:31:02.174753 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.167016    2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 00:31:02.174753 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.167020    2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 00:31:02.174753 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.167025    2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 00:31:02.174753 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.167029    2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 00:31:02.174753 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.167034    2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 00:31:02.174753 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.167039    2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 00:31:02.175370 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.167043    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 00:31:02.175370 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.167047    2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 00:31:02.175370 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.167052    2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 00:31:02.175370 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.167056    2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 00:31:02.175370 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.167061    2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 00:31:02.175370 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.167065    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 00:31:02.175370 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.167070    2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 00:31:02.175370 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.167074    2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 00:31:02.175370 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.167079    2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 00:31:02.175370 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.167083    2579 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 00:31:02.175370 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.167088    2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 00:31:02.175370 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.167092    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 00:31:02.175370 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.167096    2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 00:31:02.175370 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.167101    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 00:31:02.175370 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.167105    2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 00:31:02.175370 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.167116    2579 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 00:31:02.175370 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.167121    2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 00:31:02.175370 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.167126    2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 00:31:02.175370 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.167130    2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 00:31:02.175868 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.167134    2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 00:31:02.175868 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.167139    2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 00:31:02.175868 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.167145    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 00:31:02.175868 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.167149    2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 00:31:02.175868 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.167154    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 00:31:02.175868 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.167158    2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 00:31:02.175868 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.167163    2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 00:31:02.175868 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.167167    2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 00:31:02.175868 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.167172    2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 00:31:02.175868 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.167176    2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 00:31:02.175868 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.167181    2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 00:31:02.175868 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.167185    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 00:31:02.175868 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.167189    2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 00:31:02.175868 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.167193    2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 00:31:02.175868 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.167198    2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 00:31:02.175868 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.167202    2579 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 00:31:02.175868 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.167206    2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 00:31:02.175868 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.167211    2579 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 00:31:02.175868 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.167215    2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 00:31:02.175868 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.167219    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 00:31:02.176367 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.167223    2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 00:31:02.176367 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.167228    2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 00:31:02.176367 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.167232    2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 00:31:02.176367 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.167238    2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 00:31:02.176367 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.167246    2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 00:31:02.176367 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.167251    2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 00:31:02.176367 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.167256    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 00:31:02.176367 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.167261    2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 00:31:02.176367 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.167265    2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 00:31:02.176367 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.167271    2579 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 00:31:02.176367 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.167275    2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 00:31:02.176367 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.167280    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 00:31:02.176367 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.167284    2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 00:31:02.176367 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.167289    2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 00:31:02.176367 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.167293    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 00:31:02.176367 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.167297    2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 00:31:02.176367 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.167301    2579 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 00:31:02.176367 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.167305    2579 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 00:31:02.176367 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.167309    2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 00:31:02.176367 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.167314    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 00:31:02.177033 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.167318    2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 00:31:02.177033 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.167322    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 00:31:02.177033 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.167326    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 00:31:02.177033 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.167330    2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 00:31:02.177033 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.167335    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 00:31:02.177033 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.167339    2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 00:31:02.177033 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.167344    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 00:31:02.177033 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.167348    2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 00:31:02.177033 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.167352    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 00:31:02.177033 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.167356    2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 00:31:02.177033 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.167360    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 00:31:02.177033 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.167365    2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 00:31:02.177033 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.167369    2579 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 00:31:02.177033 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.167373    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 00:31:02.177033 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.167378    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 00:31:02.177033 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.167382    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 00:31:02.177033 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.167394    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 00:31:02.177033 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.167399    2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 00:31:02.177033 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.167403    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 00:31:02.177033 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.167408    2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 00:31:02.177819 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.167412    2579 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 00:31:02.177819 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.167432    2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 00:31:02.177819 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.167439    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 00:31:02.177819 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.167444    2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 00:31:02.177819 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.167448    2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 00:31:02.177819 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.167453    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 00:31:02.177819 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.167457    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 00:31:02.177819 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.167461    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 00:31:02.177819 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.167466    2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 00:31:02.177819 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.167470    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 00:31:02.177819 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.167474    2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 00:31:02.177819 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.167478    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 00:31:02.177819 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.167483    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 00:31:02.177819 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.167526    2579 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 00:31:02.177819 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.167759    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 00:31:02.177819 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.167767    2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 00:31:02.178528 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.167771    2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 00:31:02.178528 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.167776    2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 00:31:02.178528 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.167780    2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 00:31:02.178528 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.167784    2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 00:31:02.178528 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.167788    2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 00:31:02.178528 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.167792    2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 00:31:02.178528 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.167797    2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 00:31:02.178528 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.167802    2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 00:31:02.178528 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.167807    2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 00:31:02.178528 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.167811    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 00:31:02.178528 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.167816    2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 00:31:02.178528 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.167820    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 00:31:02.178528 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.167824    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 00:31:02.178528 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.167828    2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 00:31:02.178528 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.167832    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 00:31:02.178528 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.167838    2579 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 00:31:02.178528 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.167842    2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 00:31:02.178528 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.167847    2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 00:31:02.178528 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.167851    2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 00:31:02.179124 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.167857    2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 00:31:02.179124 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.167861    2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 00:31:02.179124 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.167866    2579 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 00:31:02.179124 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.167871    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 00:31:02.179124 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.167874    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 00:31:02.179124 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.167879    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 00:31:02.179124 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.167883    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 00:31:02.179124 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.167887    2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 00:31:02.179124 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.167891    2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 00:31:02.179124 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.167896    2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 00:31:02.179124 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.167900    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 00:31:02.179124 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.167904    2579 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 00:31:02.179124 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.167910    2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 00:31:02.179124 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.167916    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 00:31:02.179124 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.167921    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 00:31:02.179124 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.167926    2579 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 00:31:02.179124 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.167930    2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 00:31:02.179124 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.167935    2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 00:31:02.179124 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.167940    2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 00:31:02.179124 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.167944    2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 00:31:02.179670 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.167948    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 00:31:02.179670 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.167953    2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 00:31:02.179670 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.167958    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 00:31:02.179670 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.167962    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 00:31:02.179670 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.167966    2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 00:31:02.179670 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.167970    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 00:31:02.179670 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.167977    2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 00:31:02.179670 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.167983    2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 00:31:02.179670 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.167988    2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 00:31:02.179670 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.167992    2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 00:31:02.179670 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.167997    2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 00:31:02.179670 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.168001    2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 00:31:02.179670 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.168005    2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 00:31:02.179670 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.168011    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 00:31:02.179670 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.168016    2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 00:31:02.179670 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.168020    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 00:31:02.179670 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.168024    2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 00:31:02.179670 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.168028    2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 00:31:02.179670 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.168032    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 00:31:02.180117 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.168037    2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 00:31:02.180117 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.168041    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 00:31:02.180117 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.168045    2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 00:31:02.180117 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.168049    2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 00:31:02.180117 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.168054    2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 00:31:02.180117 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.168058    2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 00:31:02.180117 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.168062    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 00:31:02.180117 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.168066    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 00:31:02.180117 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.168070    2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 00:31:02.180117 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.168075    2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 00:31:02.180117 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.168079    2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 00:31:02.180117 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.168083    2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 00:31:02.180117 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.168087    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 00:31:02.180117 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.168091    2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 00:31:02.180117 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.168096    2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 00:31:02.180117 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.168100    2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 00:31:02.180117 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.168105    2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 00:31:02.180117 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.168110    2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 00:31:02.180117 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.168114    2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 00:31:02.180642 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.168118    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 00:31:02.180642 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.168122    2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 00:31:02.180642 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.168127    2579 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 00:31:02.180642 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.168131    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 00:31:02.180642 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.168135    2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 00:31:02.180642 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.168139    2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 00:31:02.180642 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.168143    2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 00:31:02.180642 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.168148    2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 00:31:02.180642 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.168153    2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 00:31:02.180642 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.168158    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 00:31:02.180642 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.168162    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 00:31:02.180642 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.168166    2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 00:31:02.180642 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.168170    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 00:31:02.180642 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.168174    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 00:31:02.180642 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.168178    2579 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 00:31:02.180642 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.168183    2579 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 00:31:02.180642 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.168187    2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 00:31:02.180642 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.168191    2579 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 00:31:02.180642 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.168195    2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 00:31:02.180642 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.168200    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 00:31:02.181117 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.168204    2579 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 00:31:02.181117 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.168208    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 00:31:02.181117 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.168212    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 00:31:02.181117 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.168216    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 00:31:02.181117 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.168220    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 00:31:02.181117 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.168224    2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 00:31:02.181117 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.168228    2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 00:31:02.181117 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.168267    2579 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 00:31:02.181117 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.168320    2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 00:31:02.181117 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.168327    2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 00:31:02.181117 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.168332    2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 00:31:02.181117 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.168336    2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 00:31:02.181117 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.168340    2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 00:31:02.181117 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.168344    2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 00:31:02.181117 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.168349    2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 00:31:02.181117 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.168354    2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 00:31:02.181536 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.168359    2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 00:31:02.181536 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.168363    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 00:31:02.181536 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.168368    2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 00:31:02.181536 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.168372    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 00:31:02.181536 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.168376    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 00:31:02.181536 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.168381    2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 00:31:02.181536 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.168385    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 00:31:02.181536 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.168390    2579 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 00:31:02.181536 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.168394    2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 00:31:02.181536 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.168398    2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 00:31:02.181536 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.168402    2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 00:31:02.181536 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.168406    2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 00:31:02.181536 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.168411    2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 00:31:02.181536 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.168431    2579 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 00:31:02.181536 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.168435    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 00:31:02.181536 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.168439    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 00:31:02.181536 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.168443    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 00:31:02.181536 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.168446    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 00:31:02.181536 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.168450    2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 00:31:02.182032 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.168454    2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 00:31:02.182032 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.168458    2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 00:31:02.182032 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.168462    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 00:31:02.182032 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.168467    2579 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 00:31:02.182032 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.168476    2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 00:31:02.182032 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.168481    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 00:31:02.182032 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.168486    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 00:31:02.182032 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.168491    2579 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 00:31:02.182032 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.168494    2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 00:31:02.182032 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.168498    2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 00:31:02.182032 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.168502    2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 00:31:02.182032 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.168506    2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 00:31:02.182032 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.168510    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 00:31:02.182032 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.168514    2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 00:31:02.182032 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.168518    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 00:31:02.182032 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.168522    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 00:31:02.182032 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.168527    2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 00:31:02.182032 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.168531    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 00:31:02.182032 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.168537    2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 00:31:02.182622 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.168544    2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 00:31:02.182622 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.168551    2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 00:31:02.182622 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.168556    2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 00:31:02.182622 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.168560    2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 00:31:02.182622 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.168565    2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 00:31:02.182622 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.168569    2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 00:31:02.182622 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.168574    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 00:31:02.182622 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.168579    2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 00:31:02.182622 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.168583    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 00:31:02.182622 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.168587    2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 00:31:02.182622 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.168591    2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 00:31:02.182622 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.168596    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 00:31:02.182622 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.168599    2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 00:31:02.182622 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.168603    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 00:31:02.182622 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.168607    2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 00:31:02.182622 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.168612    2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 00:31:02.182622 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.168617    2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 00:31:02.182622 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.168621    2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 00:31:02.182622 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.168625    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 00:31:02.183166 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.168630    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 00:31:02.183166 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.168634    2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 00:31:02.183166 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.168639    2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 00:31:02.183166 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.168643    2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 00:31:02.183166 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.168647    2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 00:31:02.183166 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.168651    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 00:31:02.183166 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.168655    2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 00:31:02.183166 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.168659    2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 00:31:02.183166 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.168664    2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 00:31:02.183166 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.168668    2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 00:31:02.183166 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.168672    2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 00:31:02.183166 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.168676    2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 00:31:02.183166 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.168681    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 00:31:02.183166 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.168685    2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 00:31:02.183166 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.168690    2579 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 00:31:02.183166 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.168694    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 00:31:02.183166 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.168699    2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 00:31:02.183166 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.168703    2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 00:31:02.183166 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.168707    2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 00:31:02.183665 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.168711    2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 00:31:02.183665 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.168715    2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 00:31:02.183665 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.168720    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 00:31:02.183665 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.168724    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 00:31:02.183665 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.168729    2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 00:31:02.183665 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.168733    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 00:31:02.183665 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.168737    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 00:31:02.183665 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.168741    2579 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 00:31:02.183665 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.168746    2579 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 00:31:02.183665 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.168749    2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 00:31:02.183665 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.168754    2579 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 00:31:02.183665 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.168758    2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 00:31:02.183665 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.168764    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 00:31:02.183665 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.168769    2579 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 00:31:02.183665 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.168773    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 00:31:02.183665 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.168777    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 00:31:02.183665 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.168781    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 00:31:02.183665 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.168785    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 00:31:02.183665 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.168789    2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 00:31:02.183665 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.168793    2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 00:31:02.184195 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.168797    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 00:31:02.184195 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:02.168802    2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 00:31:02.184195 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.168898    2579 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 00:31:02.184195 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.171527    2579 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Oct 30 00:31:02.184195 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.172505    2579 server.go:1014] "Starting client certificate rotation"
Oct 30 00:31:02.184195 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.172611    2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 00:31:02.184195 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.172652    2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 00:31:02.184195 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.174804    2579 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 00:31:02.184195 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.179055    2579 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 00:31:02.190186 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.190169    2579 log.go:25] "Validated CRI v1 runtime API"
Oct 30 00:31:02.193722 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.193705    2579 log.go:25] "Validated CRI v1 image API"
Oct 30 00:31:02.195026 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.194999    2579 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 00:31:02.197600 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.197573    2579 fs.go:135] Filesystem UUIDs: map[4444e55e-7462-4627-8c3d-fd7f95cef134:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 89f2892b-758d-4770-80e8-4d7019907450:/dev/nvme0n1p4]
Oct 30 00:31:02.197664 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.197600    2579 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Oct 30 00:31:02.202960 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.202839    2579 manager.go:217] Machine: {Timestamp:2026-10-30 00:31:02.201315786 +0000 UTC m=+0.330586854 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100808 MemoryCapacity:33164333056 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2f5f3fefea6c82c96c9899e764860b SystemUUID:ec2f5f3f-efea-6c82-c96c-9899e764860b BootID:c43aa09f-cd5a-4eb6-bc6e-43c1a088b226 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:d8:58:65:ed Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:d8:58:65:ed Speed:0 Mtu:9001} {Name:ovs-system MacAddress:16:80:d1:87:0b:00 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164333056 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 00:31:02.202960 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.202956    2579 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 00:31:02.203103 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.203039    2579 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 00:31:02.206211 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.206188    2579 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 00:31:02.206363 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.206212    2579 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-7.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 00:31:02.206404 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.206365    2579 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 00:31:02.206404 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.206373    2579 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 00:31:02.206476 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.206405    2579 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 00:31:02.206476 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.206433    2579 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 00:31:02.207154 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.207144    2579 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 00:31:02.210672 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.210661    2579 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 00:31:02.210798 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.210788    2579 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 00:31:02.211441 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.211431    2579 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 00:31:02.211473 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.211445    2579 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 00:31:02.211473 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.211455    2579 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 00:31:02.211473 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.211465    2579 kubelet.go:405] "Adding apiserver pod source"
Oct 30 00:31:02.211473 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.211473    2579 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 00:31:02.211830 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.211817    2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 00:31:02.214193 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.214176    2579 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 00:31:02.218947 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.218931    2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 00:31:02.219802 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.219788    2579 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 00:31:02.219876 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.219818    2579 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 00:31:02.219973 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.219964    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 00:31:02.220005 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.219979    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 00:31:02.220005 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.219986    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 00:31:02.220005 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.219996    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 00:31:02.220005 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.220002    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 00:31:02.220116 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.220007    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 00:31:02.220438 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.220428    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 00:31:02.220470 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.220440    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 00:31:02.220470 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.220447    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 00:31:02.220470 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.220453    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 00:31:02.220470 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.220468    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 00:31:02.220589 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.220478    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 00:31:02.221849 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.221840    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 00:31:02.221849 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.221850    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 00:31:02.225703 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.225690    2579 server.go:1263] "Started kubelet"
Oct 30 00:31:02.226661 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.226614    2579 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 00:31:02.227120 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.227072    2579 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 00:31:02.227173 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.227154    2579 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 00:31:02.227440 ip-10-0-128-7 systemd[1]: Started Kubernetes Kubelet.
Oct 30 00:31:02.230247 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.230147    2579 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 00:31:02.231547 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.231122    2579 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 00:31:02.234210 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:02.234186    2579 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Oct 30 00:31:02.235321 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.235302    2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 00:31:02.236000 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.235984    2579 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 00:31:02.236098 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:02.236080    2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-7.ec2.internal\" not found"
Oct 30 00:31:02.236551 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.236533    2579 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 00:31:02.236621 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.236555    2579 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 00:31:02.236621 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.236530    2579 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 00:31:02.236697 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.236656    2579 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 00:31:02.236697 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.236663    2579 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 00:31:02.236787 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.236772    2579 factory.go:55] Registering systemd factory
Oct 30 00:31:02.236819 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.236790    2579 factory.go:223] Registration of the systemd container factory successfully
Oct 30 00:31:02.236983 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.236970    2579 factory.go:153] Registering CRI-O factory
Oct 30 00:31:02.237044 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.236987    2579 factory.go:223] Registration of the crio container factory successfully
Oct 30 00:31:02.237106 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.237094    2579 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 00:31:02.237157 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.237118    2579 factory.go:103] Registering Raw factory
Oct 30 00:31:02.237157 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.237133    2579 manager.go:1199] Started watching for new ooms in manager
Oct 30 00:31:02.237978 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.237962    2579 manager.go:319] Starting recovery of all containers
Oct 30 00:31:02.247244 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.247225    2579 manager.go:324] Recovery completed
Oct 30 00:31:02.252544 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.252525    2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 00:31:02.254317 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.254294    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-7.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 00:31:02.254387 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.254330    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-7.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 00:31:02.254387 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.254342    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-7.ec2.internal" event="NodeHasSufficientPID"
Oct 30 00:31:02.257232 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.257212    2579 cpu_manager.go:226] "Starting" policy="none"
Oct 30 00:31:02.257232 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.257231    2579 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 00:31:02.257366 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.257253    2579 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 00:31:02.257366 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.257255    2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 00:31:02.258066 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.258047    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-7.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 00:31:02.258133 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.258078    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-7.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 00:31:02.258133 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.258094    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-7.ec2.internal" event="NodeHasSufficientPID"
Oct 30 00:31:02.259587 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.259572    2579 policy_none.go:50] "Start"
Oct 30 00:31:02.259587 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.259587    2579 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 00:31:02.259739 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.259597    2579 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 00:31:02.260277 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.260256    2579 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-128-7.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Oct 30 00:31:02.260342 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:02.260256    2579 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-128-7.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Oct 30 00:31:02.261567 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:02.260268    2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-7.ec2.internal.18e32746ee788952  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-7.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-128-7.ec2.internal,},FirstTimestamp:2026-10-30 00:31:02.225660242 +0000 UTC m=+0.354931310,LastTimestamp:2026-10-30 00:31:02.225660242 +0000 UTC m=+0.354931310,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-7.ec2.internal,}"
Oct 30 00:31:02.261761 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.261751    2579 policy_none.go:44] "Start"
Oct 30 00:31:02.267654 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.267635    2579 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 00:31:02.278712 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:02.278644    2579 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-128-7.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node"
Oct 30 00:31:02.278712 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:02.278597    2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-7.ec2.internal.18e32746f02dcc14  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-7.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-128-7.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-128-7.ec2.internal,},FirstTimestamp:2026-10-30 00:31:02.254316564 +0000 UTC m=+0.383587633,LastTimestamp:2026-10-30 00:31:02.254316564 +0000 UTC m=+0.383587633,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-7.ec2.internal,}"
Oct 30 00:31:02.278712 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:02.278685    2579 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service"
Oct 30 00:31:02.278896 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:02.278715    2579 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver"
Oct 30 00:31:02.290856 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.290827    2579 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 00:31:02.292268 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.292249    2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-7rd49"
Oct 30 00:31:02.293595 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.293579    2579 manager.go:341] "Starting Device Plugin manager"
Oct 30 00:31:02.293687 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:02.293614    2579 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 00:31:02.293687 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.293626    2579 server.go:87] "Starting device plugin registration server"
Oct 30 00:31:02.293898 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.293887    2579 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 00:31:02.293946 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.293901    2579 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 00:31:02.294002 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.293987    2579 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 00:31:02.294340 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.294083    2579 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 00:31:02.294340 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.294095    2579 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 00:31:02.294820 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:02.294801    2579 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 00:31:02.294896 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:02.294856    2579 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-128-7.ec2.internal\" not found"
Oct 30 00:31:02.303872 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.303856    2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-7rd49"
Oct 30 00:31:02.304989 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:02.304922    2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-7.ec2.internal.18e32746f02e1817  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-7.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-128-7.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-128-7.ec2.internal,},FirstTimestamp:2026-10-30 00:31:02.254336023 +0000 UTC m=+0.383607091,LastTimestamp:2026-10-30 00:31:02.254336023 +0000 UTC m=+0.383607091,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-7.ec2.internal,}"
Oct 30 00:31:02.367701 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.367657    2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 00:31:02.368898 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.368881    2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 00:31:02.368962 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.368903    2579 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 00:31:02.368962 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.368921    2579 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 00:31:02.368962 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:02.368956    2579 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Oct 30 00:31:02.385384 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.385361    2579 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 00:31:02.394361 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.394347    2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 00:31:02.395232 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.395215    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-7.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 00:31:02.395312 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.395245    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-7.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 00:31:02.395312 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.395257    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-7.ec2.internal" event="NodeHasSufficientPID"
Oct 30 00:31:02.395312 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.395280    2579 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-128-7.ec2.internal"
Oct 30 00:31:02.419482 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.419437    2579 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-128-7.ec2.internal"
Oct 30 00:31:02.419482 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:02.419463    2579 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-128-7.ec2.internal\": node \"ip-10-0-128-7.ec2.internal\" not found"
Oct 30 00:31:02.458026 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:02.458001    2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-7.ec2.internal\" not found"
Oct 30 00:31:02.469253 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.469221    2579 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-7.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-128-7.ec2.internal"]
Oct 30 00:31:02.469327 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.469285    2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 00:31:02.470133 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.470118    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-7.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 00:31:02.470219 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.470150    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-7.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 00:31:02.470219 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.470164    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-7.ec2.internal" event="NodeHasSufficientPID"
Oct 30 00:31:02.472370 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.472355    2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 00:31:02.472527 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.472511    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-7.ec2.internal"
Oct 30 00:31:02.472576 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.472546    2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 00:31:02.473036 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.473022    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-7.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 00:31:02.473102 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.473047    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-7.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 00:31:02.473102 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.473056    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-7.ec2.internal" event="NodeHasSufficientPID"
Oct 30 00:31:02.473102 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.473057    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-7.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 00:31:02.473102 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.473076    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-7.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 00:31:02.473102 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.473085    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-7.ec2.internal" event="NodeHasSufficientPID"
Oct 30 00:31:02.474983 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.474967    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-7.ec2.internal"
Oct 30 00:31:02.475063 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.474991    2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 00:31:02.475631 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.475615    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-7.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 00:31:02.475710 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.475645    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-7.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 00:31:02.475710 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.475660    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-7.ec2.internal" event="NodeHasSufficientPID"
Oct 30 00:31:02.501718 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:02.501698    2579 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-7.ec2.internal\" not found" node="ip-10-0-128-7.ec2.internal"
Oct 30 00:31:02.504640 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:02.504623    2579 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-7.ec2.internal\" not found" node="ip-10-0-128-7.ec2.internal"
Oct 30 00:31:02.538743 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.538721    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/70c88bf4a25fa0f049c2cc951e3cbb56-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-7.ec2.internal\" (UID: \"70c88bf4a25fa0f049c2cc951e3cbb56\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-7.ec2.internal"
Oct 30 00:31:02.538857 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.538748    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/70c88bf4a25fa0f049c2cc951e3cbb56-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-7.ec2.internal\" (UID: \"70c88bf4a25fa0f049c2cc951e3cbb56\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-7.ec2.internal"
Oct 30 00:31:02.538857 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.538764    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/74c81293f9236d86a7e367354c3aa34d-config\") pod \"kube-apiserver-proxy-ip-10-0-128-7.ec2.internal\" (UID: \"74c81293f9236d86a7e367354c3aa34d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-7.ec2.internal"
Oct 30 00:31:02.558488 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:02.558470    2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-7.ec2.internal\" not found"
Oct 30 00:31:02.639513 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.639479    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/70c88bf4a25fa0f049c2cc951e3cbb56-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-7.ec2.internal\" (UID: \"70c88bf4a25fa0f049c2cc951e3cbb56\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-7.ec2.internal"
Oct 30 00:31:02.639513 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.639510    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/74c81293f9236d86a7e367354c3aa34d-config\") pod \"kube-apiserver-proxy-ip-10-0-128-7.ec2.internal\" (UID: \"74c81293f9236d86a7e367354c3aa34d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-7.ec2.internal"
Oct 30 00:31:02.639736 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.639525    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/70c88bf4a25fa0f049c2cc951e3cbb56-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-7.ec2.internal\" (UID: \"70c88bf4a25fa0f049c2cc951e3cbb56\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-7.ec2.internal"
Oct 30 00:31:02.639736 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.639592    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/70c88bf4a25fa0f049c2cc951e3cbb56-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-7.ec2.internal\" (UID: \"70c88bf4a25fa0f049c2cc951e3cbb56\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-7.ec2.internal"
Oct 30 00:31:02.639736 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.639617    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/74c81293f9236d86a7e367354c3aa34d-config\") pod \"kube-apiserver-proxy-ip-10-0-128-7.ec2.internal\" (UID: \"74c81293f9236d86a7e367354c3aa34d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-7.ec2.internal"
Oct 30 00:31:02.639736 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.639592    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/70c88bf4a25fa0f049c2cc951e3cbb56-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-7.ec2.internal\" (UID: \"70c88bf4a25fa0f049c2cc951e3cbb56\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-7.ec2.internal"
Oct 30 00:31:02.659509 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:02.659461    2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-7.ec2.internal\" not found"
Oct 30 00:31:02.760202 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:02.760123    2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-7.ec2.internal\" not found"
Oct 30 00:31:02.804838 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.804811    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-7.ec2.internal"
Oct 30 00:31:02.808071 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:02.808055    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-7.ec2.internal"
Oct 30 00:31:02.860428 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:02.860249    2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-7.ec2.internal\" not found"
Oct 30 00:31:02.961117 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:02.961086    2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-7.ec2.internal\" not found"
Oct 30 00:31:03.061614 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:03.061551    2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-7.ec2.internal\" not found"
Oct 30 00:31:04.190003 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:04.189976    2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-7.ec2.internal\" not found"
Oct 30 00:31:04.200158 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:04.200132    2579 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Oct 30 00:31:04.200311 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:04.200292    2579 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 30 00:31:04.290789 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:04.290757    2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-7.ec2.internal\" not found"
Oct 30 00:31:04.336160 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:04.336101    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 00:26:03 +0000 UTC" deadline="2028-07-16 10:18:03.019057466 +0000 UTC"
Oct 30 00:31:04.336295 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:04.336158    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15009h46m58.682907907s"
Oct 30 00:31:04.378636 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:04.378584    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70c88bf4a25fa0f049c2cc951e3cbb56.slice/crio-453fcdc0a3bdbd70d0b6a86ba356202ebd1abd1c2e08b95bc5fd8533b3550ac8 WatchSource:0}: Error finding container 453fcdc0a3bdbd70d0b6a86ba356202ebd1abd1c2e08b95bc5fd8533b3550ac8: Status 404 returned error can't find the container with id 453fcdc0a3bdbd70d0b6a86ba356202ebd1abd1c2e08b95bc5fd8533b3550ac8
Oct 30 00:31:04.378920 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:04.378895    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74c81293f9236d86a7e367354c3aa34d.slice/crio-c72433bacc952d570b3676b773c9b174fec945357ecf73c909e453c4a02d6577 WatchSource:0}: Error finding container c72433bacc952d570b3676b773c9b174fec945357ecf73c909e453c4a02d6577: Status 404 returned error can't find the container with id c72433bacc952d570b3676b773c9b174fec945357ecf73c909e453c4a02d6577
Oct 30 00:31:04.384633 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:04.384618    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 00:31:04.391861 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:04.391844    2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-7.ec2.internal\" not found"
Oct 30 00:31:04.400479 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:04.400437    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-7.ec2.internal" event={"ID":"70c88bf4a25fa0f049c2cc951e3cbb56","Type":"ContainerStarted","Data":"453fcdc0a3bdbd70d0b6a86ba356202ebd1abd1c2e08b95bc5fd8533b3550ac8"}
Oct 30 00:31:04.401480 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:04.401459    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-7.ec2.internal" event={"ID":"74c81293f9236d86a7e367354c3aa34d","Type":"ContainerStarted","Data":"c72433bacc952d570b3676b773c9b174fec945357ecf73c909e453c4a02d6577"}
Oct 30 00:31:04.492393 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:04.492359    2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-7.ec2.internal\" not found"
Oct 30 00:31:04.498503 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:04.498486    2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 00:31:04.499624 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:04.499603    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-7.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 00:31:04.499713 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:04.499634    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-7.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 00:31:04.499713 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:04.499644    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-7.ec2.internal" event="NodeHasSufficientPID"
Oct 30 00:31:04.500665 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:04.500650    2579 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 00:31:04.534267 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:04.534242    2579 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 00:31:04.534375 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:04.534297    2579 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 00:31:04.564384 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:04.564362    2579 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-7.ec2.internal"
Oct 30 00:31:04.588791 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:04.588771    2579 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 00:31:04.589716 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:04.589704    2579 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-7.ec2.internal"
Oct 30 00:31:04.609832 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:04.609810    2579 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 00:31:05.240686 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.240656    2579 apiserver.go:52] "Watching apiserver"
Oct 30 00:31:05.264549 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.264511    2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Oct 30 00:31:05.280048 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.280018    2579 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 00:31:05.280361 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.280338    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-6gjtz","openshift-multus/multus-h4hbh","kube-system/kube-apiserver-proxy-ip-10-0-128-7.ec2.internal","openshift-image-registry/node-ca-8b6d6","openshift-multus/network-metrics-daemon-z58bb","openshift-ovn-kubernetes/ovnkube-node-lxg8k","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-7.ec2.internal","openshift-multus/multus-additional-cni-plugins-zvcpp","openshift-network-diagnostics/network-check-target-lngrr","kube-system/global-pull-secret-syncer-tz6g8","kube-system/konnectivity-agent-m5qgr","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v24xq","openshift-network-operator/iptables-alerter-kph7f","openshift-cluster-node-tuning-operator/tuned-5hvvb"]
Oct 30 00:31:05.282937 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.282910    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tz6g8"
Oct 30 00:31:05.283043 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:05.282986    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tz6g8" podUID="80603987-09a1-4ca1-b8b4-3c9ac8d0efc6"
Oct 30 00:31:05.285136 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.285113    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-kph7f"
Oct 30 00:31:05.287359 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.287333    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-5hvvb"
Oct 30 00:31:05.290107 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.290090    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-8b6d6"
Oct 30 00:31:05.292496 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.292373    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z58bb"
Oct 30 00:31:05.292496 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:05.292458    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z58bb" podUID="af2d6f84-aea3-4698-be12-11bd6767caec"
Oct 30 00:31:05.294758 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.294736    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-lxg8k"
Oct 30 00:31:05.297103 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.297080    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lngrr"
Oct 30 00:31:05.297201 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:05.297151    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lngrr" podUID="d27ee5b1-d1aa-418c-9169-12cac37dcc2c"
Oct 30 00:31:05.302024 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.301637    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-zvcpp"
Oct 30 00:31:05.302024 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.301710    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 00:31:05.302024 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.301978    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 00:31:05.302024 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.302021    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-8njw4\""
Oct 30 00:31:05.302263 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.302074    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-z22kq\""
Oct 30 00:31:05.302263 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.302112    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 00:31:05.302263 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.302128    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 00:31:05.302263 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.302213    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 00:31:05.305758 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.305732    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-h4hbh"
Oct 30 00:31:05.308198 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.308178    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 00:31:05.308592 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.308579    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-6gjtz"
Oct 30 00:31:05.309369 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.309346    2579 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 00:31:05.309537 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.309411    2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-px25j"
Oct 30 00:31:05.311643 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.311626    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-m5qgr"
Oct 30 00:31:05.314176 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.314156    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v24xq"
Oct 30 00:31:05.319365 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.319340    2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-px25j"
Oct 30 00:31:05.366434 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.366408    2579 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 00:31:05.377468 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.377436    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/80603987-09a1-4ca1-b8b4-3c9ac8d0efc6-original-pull-secret\") pod \"global-pull-secret-syncer-tz6g8\" (UID: \"80603987-09a1-4ca1-b8b4-3c9ac8d0efc6\") " pod="kube-system/global-pull-secret-syncer-tz6g8"
Oct 30 00:31:05.377599 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.377502    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6af42f7e-20f7-4b52-9759-0bf1af4d85b4-etc-openvswitch\") pod \"ovnkube-node-lxg8k\" (UID: \"6af42f7e-20f7-4b52-9759-0bf1af4d85b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxg8k"
Oct 30 00:31:05.377599 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.377587    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a707a5d7-313f-4bef-8baf-a99aeae40d54-host-run-k8s-cni-cncf-io\") pod \"multus-h4hbh\" (UID: \"a707a5d7-313f-4bef-8baf-a99aeae40d54\") " pod="openshift-multus/multus-h4hbh"
Oct 30 00:31:05.377707 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.377626    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a707a5d7-313f-4bef-8baf-a99aeae40d54-etc-kubernetes\") pod \"multus-h4hbh\" (UID: \"a707a5d7-313f-4bef-8baf-a99aeae40d54\") " pod="openshift-multus/multus-h4hbh"
Oct 30 00:31:05.377707 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.377653    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4606f172-d949-471e-842c-b340b50e179b-etc-sysctl-conf\") pod \"tuned-5hvvb\" (UID: \"4606f172-d949-471e-842c-b340b50e179b\") " pod="openshift-cluster-node-tuning-operator/tuned-5hvvb"
Oct 30 00:31:05.377707 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.377685    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rl4f\" (UniqueName: \"kubernetes.io/projected/4606f172-d949-471e-842c-b340b50e179b-kube-api-access-6rl4f\") pod \"tuned-5hvvb\" (UID: \"4606f172-d949-471e-842c-b340b50e179b\") " pod="openshift-cluster-node-tuning-operator/tuned-5hvvb"
Oct 30 00:31:05.377855 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.377748    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/88de9f3a-e7d0-400f-9091-46c0b89c1a3c-socket-dir\") pod \"aws-ebs-csi-driver-node-v24xq\" (UID: \"88de9f3a-e7d0-400f-9091-46c0b89c1a3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v24xq"
Oct 30 00:31:05.377855 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.377775    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c1e4fab8-6072-4fcd-9624-a60dee7aed86-iptables-alerter-script\") pod \"iptables-alerter-kph7f\" (UID: \"c1e4fab8-6072-4fcd-9624-a60dee7aed86\") " pod="openshift-network-operator/iptables-alerter-kph7f"
Oct 30 00:31:05.377855 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.377801    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6af42f7e-20f7-4b52-9759-0bf1af4d85b4-node-log\") pod \"ovnkube-node-lxg8k\" (UID: \"6af42f7e-20f7-4b52-9759-0bf1af4d85b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxg8k"
Oct 30 00:31:05.377855 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.377836    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a707a5d7-313f-4bef-8baf-a99aeae40d54-multus-socket-dir-parent\") pod \"multus-h4hbh\" (UID: \"a707a5d7-313f-4bef-8baf-a99aeae40d54\") " pod="openshift-multus/multus-h4hbh"
Oct 30 00:31:05.378031 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.377877    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6af42f7e-20f7-4b52-9759-0bf1af4d85b4-ovnkube-script-lib\") pod \"ovnkube-node-lxg8k\" (UID: \"6af42f7e-20f7-4b52-9759-0bf1af4d85b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxg8k"
Oct 30 00:31:05.378031 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.377923    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a707a5d7-313f-4bef-8baf-a99aeae40d54-cni-binary-copy\") pod \"multus-h4hbh\" (UID: \"a707a5d7-313f-4bef-8baf-a99aeae40d54\") " pod="openshift-multus/multus-h4hbh"
Oct 30 00:31:05.378031 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.377964    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4606f172-d949-471e-842c-b340b50e179b-etc-sysconfig\") pod \"tuned-5hvvb\" (UID: \"4606f172-d949-471e-842c-b340b50e179b\") " pod="openshift-cluster-node-tuning-operator/tuned-5hvvb"
Oct 30 00:31:05.378031 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.377996    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4606f172-d949-471e-842c-b340b50e179b-sys\") pod \"tuned-5hvvb\" (UID: \"4606f172-d949-471e-842c-b340b50e179b\") " pod="openshift-cluster-node-tuning-operator/tuned-5hvvb"
Oct 30 00:31:05.378031 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.378017    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4606f172-d949-471e-842c-b340b50e179b-lib-modules\") pod \"tuned-5hvvb\" (UID: \"4606f172-d949-471e-842c-b340b50e179b\") " pod="openshift-cluster-node-tuning-operator/tuned-5hvvb"
Oct 30 00:31:05.378250 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.378045    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4606f172-d949-471e-842c-b340b50e179b-tmp\") pod \"tuned-5hvvb\" (UID: \"4606f172-d949-471e-842c-b340b50e179b\") " pod="openshift-cluster-node-tuning-operator/tuned-5hvvb"
Oct 30 00:31:05.378250 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.378075    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/88de9f3a-e7d0-400f-9091-46c0b89c1a3c-sys-fs\") pod \"aws-ebs-csi-driver-node-v24xq\" (UID: \"88de9f3a-e7d0-400f-9091-46c0b89c1a3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v24xq"
Oct 30 00:31:05.378250 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.378114    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/80603987-09a1-4ca1-b8b4-3c9ac8d0efc6-kubelet-config\") pod \"global-pull-secret-syncer-tz6g8\" (UID: \"80603987-09a1-4ca1-b8b4-3c9ac8d0efc6\") " pod="kube-system/global-pull-secret-syncer-tz6g8"
Oct 30 00:31:05.378250 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.378151    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jzc7\" (UniqueName: \"kubernetes.io/projected/c1e4fab8-6072-4fcd-9624-a60dee7aed86-kube-api-access-2jzc7\") pod \"iptables-alerter-kph7f\" (UID: \"c1e4fab8-6072-4fcd-9624-a60dee7aed86\") " pod="openshift-network-operator/iptables-alerter-kph7f"
Oct 30 00:31:05.378250 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.378186    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6af42f7e-20f7-4b52-9759-0bf1af4d85b4-log-socket\") pod \"ovnkube-node-lxg8k\" (UID: \"6af42f7e-20f7-4b52-9759-0bf1af4d85b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxg8k"
Oct 30 00:31:05.378250 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.378210    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6af42f7e-20f7-4b52-9759-0bf1af4d85b4-host-cni-bin\") pod \"ovnkube-node-lxg8k\" (UID: \"6af42f7e-20f7-4b52-9759-0bf1af4d85b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxg8k"
Oct 30 00:31:05.378250 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.378235    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c5c3a55b-e259-4550-9b7e-f666a28d217e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zvcpp\" (UID: \"c5c3a55b-e259-4550-9b7e-f666a28d217e\") " pod="openshift-multus/multus-additional-cni-plugins-zvcpp"
Oct 30 00:31:05.378567 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.378262    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/14285dfa-dd38-490e-a486-a1fcfc3d0214-host\") pod \"node-ca-8b6d6\" (UID: \"14285dfa-dd38-490e-a486-a1fcfc3d0214\") " pod="openshift-image-registry/node-ca-8b6d6"
Oct 30 00:31:05.378567 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.378290    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c5c3a55b-e259-4550-9b7e-f666a28d217e-system-cni-dir\") pod \"multus-additional-cni-plugins-zvcpp\" (UID: \"c5c3a55b-e259-4550-9b7e-f666a28d217e\") " pod="openshift-multus/multus-additional-cni-plugins-zvcpp"
Oct 30 00:31:05.378567 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.378315    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a707a5d7-313f-4bef-8baf-a99aeae40d54-host-run-netns\") pod \"multus-h4hbh\" (UID: \"a707a5d7-313f-4bef-8baf-a99aeae40d54\") " pod="openshift-multus/multus-h4hbh"
Oct 30 00:31:05.378567 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.378362    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a707a5d7-313f-4bef-8baf-a99aeae40d54-host-var-lib-cni-bin\") pod \"multus-h4hbh\" (UID: \"a707a5d7-313f-4bef-8baf-a99aeae40d54\") " pod="openshift-multus/multus-h4hbh"
Oct 30 00:31:05.378567 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.378420    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6af42f7e-20f7-4b52-9759-0bf1af4d85b4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lxg8k\" (UID: \"6af42f7e-20f7-4b52-9759-0bf1af4d85b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxg8k"
Oct 30 00:31:05.378567 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.378461    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6af42f7e-20f7-4b52-9759-0bf1af4d85b4-systemd-units\") pod \"ovnkube-node-lxg8k\" (UID: \"6af42f7e-20f7-4b52-9759-0bf1af4d85b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxg8k"
Oct 30 00:31:05.378567 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.378486    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6af42f7e-20f7-4b52-9759-0bf1af4d85b4-host-run-netns\") pod \"ovnkube-node-lxg8k\" (UID: \"6af42f7e-20f7-4b52-9759-0bf1af4d85b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxg8k"
Oct 30 00:31:05.378567 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.378522    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6af42f7e-20f7-4b52-9759-0bf1af4d85b4-env-overrides\") pod \"ovnkube-node-lxg8k\" (UID: \"6af42f7e-20f7-4b52-9759-0bf1af4d85b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxg8k"
Oct 30 00:31:05.378567 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.378547    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6af42f7e-20f7-4b52-9759-0bf1af4d85b4-ovn-node-metrics-cert\") pod \"ovnkube-node-lxg8k\" (UID: \"6af42f7e-20f7-4b52-9759-0bf1af4d85b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxg8k"
Oct 30 00:31:05.378967 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.378588    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6af42f7e-20f7-4b52-9759-0bf1af4d85b4-host-cni-netd\") pod \"ovnkube-node-lxg8k\" (UID: \"6af42f7e-20f7-4b52-9759-0bf1af4d85b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxg8k"
Oct 30 00:31:05.378967 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.378614    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r998s\" (UniqueName: \"kubernetes.io/projected/d27ee5b1-d1aa-418c-9169-12cac37dcc2c-kube-api-access-r998s\") pod \"network-check-target-lngrr\" (UID: \"d27ee5b1-d1aa-418c-9169-12cac37dcc2c\") " pod="openshift-network-diagnostics/network-check-target-lngrr"
Oct 30 00:31:05.378967 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.378669    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a707a5d7-313f-4bef-8baf-a99aeae40d54-system-cni-dir\") pod \"multus-h4hbh\" (UID: \"a707a5d7-313f-4bef-8baf-a99aeae40d54\") " pod="openshift-multus/multus-h4hbh"
Oct 30 00:31:05.378967 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.378716    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a707a5d7-313f-4bef-8baf-a99aeae40d54-multus-conf-dir\") pod \"multus-h4hbh\" (UID: \"a707a5d7-313f-4bef-8baf-a99aeae40d54\") " pod="openshift-multus/multus-h4hbh"
Oct 30 00:31:05.378967 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.378742    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4606f172-d949-471e-842c-b340b50e179b-host\") pod \"tuned-5hvvb\" (UID: \"4606f172-d949-471e-842c-b340b50e179b\") " pod="openshift-cluster-node-tuning-operator/tuned-5hvvb"
Oct 30 00:31:05.378967 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.378783    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4606f172-d949-471e-842c-b340b50e179b-etc-tuned\") pod \"tuned-5hvvb\" (UID: \"4606f172-d949-471e-842c-b340b50e179b\") " pod="openshift-cluster-node-tuning-operator/tuned-5hvvb"
Oct 30 00:31:05.378967 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.378834    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4d7j7\" (UniqueName: \"kubernetes.io/projected/af2d6f84-aea3-4698-be12-11bd6767caec-kube-api-access-4d7j7\") pod \"network-metrics-daemon-z58bb\" (UID: \"af2d6f84-aea3-4698-be12-11bd6767caec\") " pod="openshift-multus/network-metrics-daemon-z58bb"
Oct 30 00:31:05.378967 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.378870    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/88de9f3a-e7d0-400f-9091-46c0b89c1a3c-device-dir\") pod \"aws-ebs-csi-driver-node-v24xq\" (UID: \"88de9f3a-e7d0-400f-9091-46c0b89c1a3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v24xq"
Oct 30 00:31:05.378967 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.378899    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6af42f7e-20f7-4b52-9759-0bf1af4d85b4-host-kubelet\") pod \"ovnkube-node-lxg8k\" (UID: \"6af42f7e-20f7-4b52-9759-0bf1af4d85b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxg8k"
Oct 30 00:31:05.378967 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.378923    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lbm4\" (UniqueName: \"kubernetes.io/projected/6af42f7e-20f7-4b52-9759-0bf1af4d85b4-kube-api-access-5lbm4\") pod \"ovnkube-node-lxg8k\" (UID: \"6af42f7e-20f7-4b52-9759-0bf1af4d85b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxg8k"
Oct 30 00:31:05.378967 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.378946    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a707a5d7-313f-4bef-8baf-a99aeae40d54-cnibin\") pod \"multus-h4hbh\" (UID: \"a707a5d7-313f-4bef-8baf-a99aeae40d54\") " pod="openshift-multus/multus-h4hbh"
Oct 30 00:31:05.378967 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.378973    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4cc6\" (UniqueName: \"kubernetes.io/projected/a707a5d7-313f-4bef-8baf-a99aeae40d54-kube-api-access-h4cc6\") pod \"multus-h4hbh\" (UID: \"a707a5d7-313f-4bef-8baf-a99aeae40d54\") " pod="openshift-multus/multus-h4hbh"
Oct 30 00:31:05.379511 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.378992    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4606f172-d949-471e-842c-b340b50e179b-run\") pod \"tuned-5hvvb\" (UID: \"4606f172-d949-471e-842c-b340b50e179b\") " pod="openshift-cluster-node-tuning-operator/tuned-5hvvb"
Oct 30 00:31:05.379511 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.379040    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/88de9f3a-e7d0-400f-9091-46c0b89c1a3c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-v24xq\" (UID: \"88de9f3a-e7d0-400f-9091-46c0b89c1a3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v24xq"
Oct 30 00:31:05.379511 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.379080    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c5c3a55b-e259-4550-9b7e-f666a28d217e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zvcpp\" (UID: \"c5c3a55b-e259-4550-9b7e-f666a28d217e\") " pod="openshift-multus/multus-additional-cni-plugins-zvcpp"
Oct 30 00:31:05.379511 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.379125    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a707a5d7-313f-4bef-8baf-a99aeae40d54-host-var-lib-kubelet\") pod \"multus-h4hbh\" (UID: \"a707a5d7-313f-4bef-8baf-a99aeae40d54\") " pod="openshift-multus/multus-h4hbh"
Oct 30 00:31:05.379511 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.379156    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4606f172-d949-471e-842c-b340b50e179b-etc-sysctl-d\") pod \"tuned-5hvvb\" (UID: \"4606f172-d949-471e-842c-b340b50e179b\") " pod="openshift-cluster-node-tuning-operator/tuned-5hvvb"
Oct 30 00:31:05.379511 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.379181    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0d1b347f-c22e-489b-986d-d3b3e01fa36b-konnectivity-ca\") pod \"konnectivity-agent-m5qgr\" (UID: \"0d1b347f-c22e-489b-986d-d3b3e01fa36b\") " pod="kube-system/konnectivity-agent-m5qgr"
Oct 30 00:31:05.379511 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.379217    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxcbj\" (UniqueName: \"kubernetes.io/projected/88de9f3a-e7d0-400f-9091-46c0b89c1a3c-kube-api-access-nxcbj\") pod \"aws-ebs-csi-driver-node-v24xq\" (UID: \"88de9f3a-e7d0-400f-9091-46c0b89c1a3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v24xq"
Oct 30 00:31:05.379511 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.379268    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6llvc\" (UniqueName: \"kubernetes.io/projected/14285dfa-dd38-490e-a486-a1fcfc3d0214-kube-api-access-6llvc\") pod \"node-ca-8b6d6\" (UID: \"14285dfa-dd38-490e-a486-a1fcfc3d0214\") " pod="openshift-image-registry/node-ca-8b6d6"
Oct 30 00:31:05.379511 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.379297    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6af42f7e-20f7-4b52-9759-0bf1af4d85b4-run-systemd\") pod \"ovnkube-node-lxg8k\" (UID: \"6af42f7e-20f7-4b52-9759-0bf1af4d85b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxg8k"
Oct 30 00:31:05.379511 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.379330    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c5c3a55b-e259-4550-9b7e-f666a28d217e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zvcpp\" (UID: \"c5c3a55b-e259-4550-9b7e-f666a28d217e\") " pod="openshift-multus/multus-additional-cni-plugins-zvcpp"
Oct 30 00:31:05.379511 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.379356    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a707a5d7-313f-4bef-8baf-a99aeae40d54-host-var-lib-cni-multus\") pod \"multus-h4hbh\" (UID: \"a707a5d7-313f-4bef-8baf-a99aeae40d54\") " pod="openshift-multus/multus-h4hbh"
Oct 30 00:31:05.379511 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.379375    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a707a5d7-313f-4bef-8baf-a99aeae40d54-host-run-multus-certs\") pod \"multus-h4hbh\" (UID: \"a707a5d7-313f-4bef-8baf-a99aeae40d54\") " pod="openshift-multus/multus-h4hbh"
Oct 30 00:31:05.379511 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.379411    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/078fd0e8-1393-45d0-baee-bbcd88095b2b-hosts-file\") pod \"node-resolver-6gjtz\" (UID: \"078fd0e8-1393-45d0-baee-bbcd88095b2b\") " pod="openshift-dns/node-resolver-6gjtz"
Oct 30 00:31:05.379511 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.379472    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6af42f7e-20f7-4b52-9759-0bf1af4d85b4-var-lib-openvswitch\") pod \"ovnkube-node-lxg8k\" (UID: \"6af42f7e-20f7-4b52-9759-0bf1af4d85b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxg8k"
Oct 30 00:31:05.380055 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.379509    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c5c3a55b-e259-4550-9b7e-f666a28d217e-cni-binary-copy\") pod \"multus-additional-cni-plugins-zvcpp\" (UID: \"c5c3a55b-e259-4550-9b7e-f666a28d217e\") " pod="openshift-multus/multus-additional-cni-plugins-zvcpp"
Oct 30 00:31:05.380055 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.379552    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a707a5d7-313f-4bef-8baf-a99aeae40d54-multus-daemon-config\") pod \"multus-h4hbh\" (UID: \"a707a5d7-313f-4bef-8baf-a99aeae40d54\") " pod="openshift-multus/multus-h4hbh"
Oct 30 00:31:05.380055 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.379597    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/078fd0e8-1393-45d0-baee-bbcd88095b2b-tmp-dir\") pod \"node-resolver-6gjtz\" (UID: \"078fd0e8-1393-45d0-baee-bbcd88095b2b\") " pod="openshift-dns/node-resolver-6gjtz"
Oct 30 00:31:05.380055 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.379625    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6af42f7e-20f7-4b52-9759-0bf1af4d85b4-host-slash\") pod \"ovnkube-node-lxg8k\" (UID: \"6af42f7e-20f7-4b52-9759-0bf1af4d85b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxg8k"
Oct 30 00:31:05.380055 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.379647    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a707a5d7-313f-4bef-8baf-a99aeae40d54-hostroot\") pod \"multus-h4hbh\" (UID: \"a707a5d7-313f-4bef-8baf-a99aeae40d54\") " pod="openshift-multus/multus-h4hbh"
Oct 30 00:31:05.380055 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.379670    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crppr\" (UniqueName: \"kubernetes.io/projected/078fd0e8-1393-45d0-baee-bbcd88095b2b-kube-api-access-crppr\") pod \"node-resolver-6gjtz\" (UID: \"078fd0e8-1393-45d0-baee-bbcd88095b2b\") " pod="openshift-dns/node-resolver-6gjtz"
Oct 30 00:31:05.380055 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.379694    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqmpd\" (UniqueName: \"kubernetes.io/projected/c5c3a55b-e259-4550-9b7e-f666a28d217e-kube-api-access-mqmpd\") pod \"multus-additional-cni-plugins-zvcpp\" (UID: \"c5c3a55b-e259-4550-9b7e-f666a28d217e\") " pod="openshift-multus/multus-additional-cni-plugins-zvcpp"
Oct 30 00:31:05.380055 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.379726    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/88de9f3a-e7d0-400f-9091-46c0b89c1a3c-registration-dir\") pod \"aws-ebs-csi-driver-node-v24xq\" (UID: \"88de9f3a-e7d0-400f-9091-46c0b89c1a3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v24xq"
Oct 30 00:31:05.380055 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.379753    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c5c3a55b-e259-4550-9b7e-f666a28d217e-cnibin\") pod \"multus-additional-cni-plugins-zvcpp\" (UID: \"c5c3a55b-e259-4550-9b7e-f666a28d217e\") " pod="openshift-multus/multus-additional-cni-plugins-zvcpp"
Oct 30 00:31:05.380055 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.379781    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6af42f7e-20f7-4b52-9759-0bf1af4d85b4-ovnkube-config\") pod \"ovnkube-node-lxg8k\" (UID: \"6af42f7e-20f7-4b52-9759-0bf1af4d85b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxg8k"
Oct 30 00:31:05.380055 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.379804    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6af42f7e-20f7-4b52-9759-0bf1af4d85b4-run-ovn\") pod \"ovnkube-node-lxg8k\" (UID: \"6af42f7e-20f7-4b52-9759-0bf1af4d85b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxg8k"
Oct 30 00:31:05.380055 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.379827    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4606f172-d949-471e-842c-b340b50e179b-etc-modprobe-d\") pod \"tuned-5hvvb\" (UID: \"4606f172-d949-471e-842c-b340b50e179b\") " pod="openshift-cluster-node-tuning-operator/tuned-5hvvb"
Oct 30 00:31:05.380055 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.379857    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af2d6f84-aea3-4698-be12-11bd6767caec-metrics-certs\") pod \"network-metrics-daemon-z58bb\" (UID: \"af2d6f84-aea3-4698-be12-11bd6767caec\") " pod="openshift-multus/network-metrics-daemon-z58bb"
Oct 30 00:31:05.380055 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.379879    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/80603987-09a1-4ca1-b8b4-3c9ac8d0efc6-dbus\") pod \"global-pull-secret-syncer-tz6g8\" (UID: \"80603987-09a1-4ca1-b8b4-3c9ac8d0efc6\") " pod="kube-system/global-pull-secret-syncer-tz6g8"
Oct 30 00:31:05.380055 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.379902    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c1e4fab8-6072-4fcd-9624-a60dee7aed86-host-slash\") pod \"iptables-alerter-kph7f\" (UID: \"c1e4fab8-6072-4fcd-9624-a60dee7aed86\") " pod="openshift-network-operator/iptables-alerter-kph7f"
Oct 30 00:31:05.380055 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.379923    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/14285dfa-dd38-490e-a486-a1fcfc3d0214-serviceca\") pod \"node-ca-8b6d6\" (UID: \"14285dfa-dd38-490e-a486-a1fcfc3d0214\") " pod="openshift-image-registry/node-ca-8b6d6"
Oct 30 00:31:05.380812 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.379945    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6af42f7e-20f7-4b52-9759-0bf1af4d85b4-host-run-ovn-kubernetes\") pod \"ovnkube-node-lxg8k\" (UID: \"6af42f7e-20f7-4b52-9759-0bf1af4d85b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxg8k"
Oct 30 00:31:05.380812 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.379965    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a707a5d7-313f-4bef-8baf-a99aeae40d54-multus-cni-dir\") pod \"multus-h4hbh\" (UID: \"a707a5d7-313f-4bef-8baf-a99aeae40d54\") " pod="openshift-multus/multus-h4hbh"
Oct 30 00:31:05.380812 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.379987    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4606f172-d949-471e-842c-b340b50e179b-etc-systemd\") pod \"tuned-5hvvb\" (UID: \"4606f172-d949-471e-842c-b340b50e179b\") " pod="openshift-cluster-node-tuning-operator/tuned-5hvvb"
Oct 30 00:31:05.380812 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.380042    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4606f172-d949-471e-842c-b340b50e179b-var-lib-kubelet\") pod \"tuned-5hvvb\" (UID: \"4606f172-d949-471e-842c-b340b50e179b\") " pod="openshift-cluster-node-tuning-operator/tuned-5hvvb"
Oct 30 00:31:05.380812 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.380066    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/88de9f3a-e7d0-400f-9091-46c0b89c1a3c-etc-selinux\") pod \"aws-ebs-csi-driver-node-v24xq\" (UID: \"88de9f3a-e7d0-400f-9091-46c0b89c1a3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v24xq"
Oct 30 00:31:05.380812 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.380102    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6af42f7e-20f7-4b52-9759-0bf1af4d85b4-run-openvswitch\") pod \"ovnkube-node-lxg8k\" (UID: \"6af42f7e-20f7-4b52-9759-0bf1af4d85b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxg8k"
Oct 30 00:31:05.380812 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.380124    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4606f172-d949-471e-842c-b340b50e179b-etc-kubernetes\") pod \"tuned-5hvvb\" (UID: \"4606f172-d949-471e-842c-b340b50e179b\") " pod="openshift-cluster-node-tuning-operator/tuned-5hvvb"
Oct 30 00:31:05.380812 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.380145    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0d1b347f-c22e-489b-986d-d3b3e01fa36b-agent-certs\") pod \"konnectivity-agent-m5qgr\" (UID: \"0d1b347f-c22e-489b-986d-d3b3e01fa36b\") " pod="kube-system/konnectivity-agent-m5qgr"
Oct 30 00:31:05.400315 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.400292    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 00:31:05.400433 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.400375    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 00:31:05.400537 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.400523    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 00:31:05.400976 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.400870    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 00:31:05.400976 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.400906    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 00:31:05.400976 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.400925    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 00:31:05.400976 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.400939    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-mt5kv\""
Oct 30 00:31:05.400976 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.400949    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 00:31:05.400976 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.400931    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 00:31:05.400976 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.400919    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 00:31:05.401348 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.401039    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 00:31:05.401447 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.401429    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 00:31:05.401512 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.401430    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 00:31:05.410562 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.410543    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 00:31:05.410562 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.410551    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 00:31:05.410701 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.410553    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 00:31:05.412410 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.412376    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-8g889\""
Oct 30 00:31:05.413046 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.413029    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-ttql7\""
Oct 30 00:31:05.413120 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.413097    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 00:31:05.413172 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.413129    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 00:31:05.413172 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.413153    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 00:31:05.413741 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.413720    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-xlz4l\""
Oct 30 00:31:05.413834 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.413824    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 00:31:05.413992 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.413974    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 00:31:05.414292 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.414276    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-h6pr4\""
Oct 30 00:31:05.414707 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.414687    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-6rb57\""
Oct 30 00:31:05.415024 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.415005    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 00:31:05.415134 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.415115    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-d7h4r\""
Oct 30 00:31:05.480847 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.480806    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c5c3a55b-e259-4550-9b7e-f666a28d217e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zvcpp\" (UID: \"c5c3a55b-e259-4550-9b7e-f666a28d217e\") " pod="openshift-multus/multus-additional-cni-plugins-zvcpp"
Oct 30 00:31:05.481012 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.480855    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/14285dfa-dd38-490e-a486-a1fcfc3d0214-host\") pod \"node-ca-8b6d6\" (UID: \"14285dfa-dd38-490e-a486-a1fcfc3d0214\") " pod="openshift-image-registry/node-ca-8b6d6"
Oct 30 00:31:05.481012 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.480882    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c5c3a55b-e259-4550-9b7e-f666a28d217e-system-cni-dir\") pod \"multus-additional-cni-plugins-zvcpp\" (UID: \"c5c3a55b-e259-4550-9b7e-f666a28d217e\") " pod="openshift-multus/multus-additional-cni-plugins-zvcpp"
Oct 30 00:31:05.481012 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.480902    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a707a5d7-313f-4bef-8baf-a99aeae40d54-host-run-netns\") pod \"multus-h4hbh\" (UID: \"a707a5d7-313f-4bef-8baf-a99aeae40d54\") " pod="openshift-multus/multus-h4hbh"
Oct 30 00:31:05.481012 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.480920    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a707a5d7-313f-4bef-8baf-a99aeae40d54-host-var-lib-cni-bin\") pod \"multus-h4hbh\" (UID: \"a707a5d7-313f-4bef-8baf-a99aeae40d54\") " pod="openshift-multus/multus-h4hbh"
Oct 30 00:31:05.481012 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.480945    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6af42f7e-20f7-4b52-9759-0bf1af4d85b4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lxg8k\" (UID: \"6af42f7e-20f7-4b52-9759-0bf1af4d85b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxg8k"
Oct 30 00:31:05.481012 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.480969    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6af42f7e-20f7-4b52-9759-0bf1af4d85b4-systemd-units\") pod \"ovnkube-node-lxg8k\" (UID: \"6af42f7e-20f7-4b52-9759-0bf1af4d85b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxg8k"
Oct 30 00:31:05.481012 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.480992    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6af42f7e-20f7-4b52-9759-0bf1af4d85b4-host-run-netns\") pod \"ovnkube-node-lxg8k\" (UID: \"6af42f7e-20f7-4b52-9759-0bf1af4d85b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxg8k"
Oct 30 00:31:05.481012 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.481015    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6af42f7e-20f7-4b52-9759-0bf1af4d85b4-env-overrides\") pod \"ovnkube-node-lxg8k\" (UID: \"6af42f7e-20f7-4b52-9759-0bf1af4d85b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxg8k"
Oct 30 00:31:05.481333 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.481039    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6af42f7e-20f7-4b52-9759-0bf1af4d85b4-ovn-node-metrics-cert\") pod \"ovnkube-node-lxg8k\" (UID: \"6af42f7e-20f7-4b52-9759-0bf1af4d85b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxg8k"
Oct 30 00:31:05.481333 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.481062    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6af42f7e-20f7-4b52-9759-0bf1af4d85b4-host-cni-netd\") pod \"ovnkube-node-lxg8k\" (UID: \"6af42f7e-20f7-4b52-9759-0bf1af4d85b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxg8k"
Oct 30 00:31:05.481333 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.481083    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r998s\" (UniqueName: \"kubernetes.io/projected/d27ee5b1-d1aa-418c-9169-12cac37dcc2c-kube-api-access-r998s\") pod \"network-check-target-lngrr\" (UID: \"d27ee5b1-d1aa-418c-9169-12cac37dcc2c\") " pod="openshift-network-diagnostics/network-check-target-lngrr"
Oct 30 00:31:05.481333 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.481097    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a707a5d7-313f-4bef-8baf-a99aeae40d54-system-cni-dir\") pod \"multus-h4hbh\" (UID: \"a707a5d7-313f-4bef-8baf-a99aeae40d54\") " pod="openshift-multus/multus-h4hbh"
Oct 30 00:31:05.481333 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.481119    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a707a5d7-313f-4bef-8baf-a99aeae40d54-multus-conf-dir\") pod \"multus-h4hbh\" (UID: \"a707a5d7-313f-4bef-8baf-a99aeae40d54\") " pod="openshift-multus/multus-h4hbh"
Oct 30 00:31:05.481333 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.481132    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4606f172-d949-471e-842c-b340b50e179b-host\") pod \"tuned-5hvvb\" (UID: \"4606f172-d949-471e-842c-b340b50e179b\") " pod="openshift-cluster-node-tuning-operator/tuned-5hvvb"
Oct 30 00:31:05.481333 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.481147    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4606f172-d949-471e-842c-b340b50e179b-etc-tuned\") pod \"tuned-5hvvb\" (UID: \"4606f172-d949-471e-842c-b340b50e179b\") " pod="openshift-cluster-node-tuning-operator/tuned-5hvvb"
Oct 30 00:31:05.481333 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.481162    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4d7j7\" (UniqueName: \"kubernetes.io/projected/af2d6f84-aea3-4698-be12-11bd6767caec-kube-api-access-4d7j7\") pod \"network-metrics-daemon-z58bb\" (UID: \"af2d6f84-aea3-4698-be12-11bd6767caec\") " pod="openshift-multus/network-metrics-daemon-z58bb"
Oct 30 00:31:05.481333 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.481177    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/88de9f3a-e7d0-400f-9091-46c0b89c1a3c-device-dir\") pod \"aws-ebs-csi-driver-node-v24xq\" (UID: \"88de9f3a-e7d0-400f-9091-46c0b89c1a3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v24xq"
Oct 30 00:31:05.481333 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.481190    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6af42f7e-20f7-4b52-9759-0bf1af4d85b4-host-kubelet\") pod \"ovnkube-node-lxg8k\" (UID: \"6af42f7e-20f7-4b52-9759-0bf1af4d85b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxg8k"
Oct 30 00:31:05.481333 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.481204    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lbm4\" (UniqueName: \"kubernetes.io/projected/6af42f7e-20f7-4b52-9759-0bf1af4d85b4-kube-api-access-5lbm4\") pod \"ovnkube-node-lxg8k\" (UID: \"6af42f7e-20f7-4b52-9759-0bf1af4d85b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxg8k"
Oct 30 00:31:05.481333 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.481227    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a707a5d7-313f-4bef-8baf-a99aeae40d54-cnibin\") pod \"multus-h4hbh\" (UID: \"a707a5d7-313f-4bef-8baf-a99aeae40d54\") " pod="openshift-multus/multus-h4hbh"
Oct 30 00:31:05.481333 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.481247    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4cc6\" (UniqueName: \"kubernetes.io/projected/a707a5d7-313f-4bef-8baf-a99aeae40d54-kube-api-access-h4cc6\") pod \"multus-h4hbh\" (UID: \"a707a5d7-313f-4bef-8baf-a99aeae40d54\") " pod="openshift-multus/multus-h4hbh"
Oct 30 00:31:05.481333 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.481265    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4606f172-d949-471e-842c-b340b50e179b-run\") pod \"tuned-5hvvb\" (UID: \"4606f172-d949-471e-842c-b340b50e179b\") " pod="openshift-cluster-node-tuning-operator/tuned-5hvvb"
Oct 30 00:31:05.481333 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.481285    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/88de9f3a-e7d0-400f-9091-46c0b89c1a3c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-v24xq\" (UID: \"88de9f3a-e7d0-400f-9091-46c0b89c1a3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v24xq"
Oct 30 00:31:05.481333 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.481301    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c5c3a55b-e259-4550-9b7e-f666a28d217e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zvcpp\" (UID: \"c5c3a55b-e259-4550-9b7e-f666a28d217e\") " pod="openshift-multus/multus-additional-cni-plugins-zvcpp"
Oct 30 00:31:05.481333 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.481315    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a707a5d7-313f-4bef-8baf-a99aeae40d54-host-var-lib-kubelet\") pod \"multus-h4hbh\" (UID: \"a707a5d7-313f-4bef-8baf-a99aeae40d54\") " pod="openshift-multus/multus-h4hbh"
Oct 30 00:31:05.481845 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.481335    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4606f172-d949-471e-842c-b340b50e179b-etc-sysctl-d\") pod \"tuned-5hvvb\" (UID: \"4606f172-d949-471e-842c-b340b50e179b\") " pod="openshift-cluster-node-tuning-operator/tuned-5hvvb"
Oct 30 00:31:05.481845 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.481352    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0d1b347f-c22e-489b-986d-d3b3e01fa36b-konnectivity-ca\") pod \"konnectivity-agent-m5qgr\" (UID: \"0d1b347f-c22e-489b-986d-d3b3e01fa36b\") " pod="kube-system/konnectivity-agent-m5qgr"
Oct 30 00:31:05.481845 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.481367    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxcbj\" (UniqueName: \"kubernetes.io/projected/88de9f3a-e7d0-400f-9091-46c0b89c1a3c-kube-api-access-nxcbj\") pod \"aws-ebs-csi-driver-node-v24xq\" (UID: \"88de9f3a-e7d0-400f-9091-46c0b89c1a3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v24xq"
Oct 30 00:31:05.481845 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.481397    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6llvc\" (UniqueName: \"kubernetes.io/projected/14285dfa-dd38-490e-a486-a1fcfc3d0214-kube-api-access-6llvc\") pod \"node-ca-8b6d6\" (UID: \"14285dfa-dd38-490e-a486-a1fcfc3d0214\") " pod="openshift-image-registry/node-ca-8b6d6"
Oct 30 00:31:05.481845 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.481419    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6af42f7e-20f7-4b52-9759-0bf1af4d85b4-run-systemd\") pod \"ovnkube-node-lxg8k\" (UID: \"6af42f7e-20f7-4b52-9759-0bf1af4d85b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxg8k"
Oct 30 00:31:05.481845 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.481441    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c5c3a55b-e259-4550-9b7e-f666a28d217e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zvcpp\" (UID: \"c5c3a55b-e259-4550-9b7e-f666a28d217e\") " pod="openshift-multus/multus-additional-cni-plugins-zvcpp"
Oct 30 00:31:05.481845 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.481464    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a707a5d7-313f-4bef-8baf-a99aeae40d54-host-var-lib-cni-multus\") pod \"multus-h4hbh\" (UID: \"a707a5d7-313f-4bef-8baf-a99aeae40d54\") " pod="openshift-multus/multus-h4hbh"
Oct 30 00:31:05.481845 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.481482    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4606f172-d949-471e-842c-b340b50e179b-host\") pod \"tuned-5hvvb\" (UID: \"4606f172-d949-471e-842c-b340b50e179b\") " pod="openshift-cluster-node-tuning-operator/tuned-5hvvb"
Oct 30 00:31:05.481845 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.481495    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6af42f7e-20f7-4b52-9759-0bf1af4d85b4-host-run-netns\") pod \"ovnkube-node-lxg8k\" (UID: \"6af42f7e-20f7-4b52-9759-0bf1af4d85b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxg8k"
Oct 30 00:31:05.481845 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.481512    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c5c3a55b-e259-4550-9b7e-f666a28d217e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zvcpp\" (UID: \"c5c3a55b-e259-4550-9b7e-f666a28d217e\") " pod="openshift-multus/multus-additional-cni-plugins-zvcpp"
Oct 30 00:31:05.481845 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.481522    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a707a5d7-313f-4bef-8baf-a99aeae40d54-host-run-multus-certs\") pod \"multus-h4hbh\" (UID: \"a707a5d7-313f-4bef-8baf-a99aeae40d54\") " pod="openshift-multus/multus-h4hbh"
Oct 30 00:31:05.481845 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.481511    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6af42f7e-20f7-4b52-9759-0bf1af4d85b4-host-cni-netd\") pod \"ovnkube-node-lxg8k\" (UID: \"6af42f7e-20f7-4b52-9759-0bf1af4d85b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxg8k"
Oct 30 00:31:05.481845 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.481500    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a707a5d7-313f-4bef-8baf-a99aeae40d54-host-run-netns\") pod \"multus-h4hbh\" (UID: \"a707a5d7-313f-4bef-8baf-a99aeae40d54\") " pod="openshift-multus/multus-h4hbh"
Oct 30 00:31:05.481845 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.481552    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c5c3a55b-e259-4550-9b7e-f666a28d217e-system-cni-dir\") pod \"multus-additional-cni-plugins-zvcpp\" (UID: \"c5c3a55b-e259-4550-9b7e-f666a28d217e\") " pod="openshift-multus/multus-additional-cni-plugins-zvcpp"
Oct 30 00:31:05.481845 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.481487    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a707a5d7-313f-4bef-8baf-a99aeae40d54-host-run-multus-certs\") pod \"multus-h4hbh\" (UID: \"a707a5d7-313f-4bef-8baf-a99aeae40d54\") " pod="openshift-multus/multus-h4hbh"
Oct 30 00:31:05.481845 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.481574    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6af42f7e-20f7-4b52-9759-0bf1af4d85b4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lxg8k\" (UID: \"6af42f7e-20f7-4b52-9759-0bf1af4d85b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxg8k"
Oct 30 00:31:05.481845 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.481604    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/078fd0e8-1393-45d0-baee-bbcd88095b2b-hosts-file\") pod \"node-resolver-6gjtz\" (UID: \"078fd0e8-1393-45d0-baee-bbcd88095b2b\") " pod="openshift-dns/node-resolver-6gjtz"
Oct 30 00:31:05.482678 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.481616    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/14285dfa-dd38-490e-a486-a1fcfc3d0214-host\") pod \"node-ca-8b6d6\" (UID: \"14285dfa-dd38-490e-a486-a1fcfc3d0214\") " pod="openshift-image-registry/node-ca-8b6d6"
Oct 30 00:31:05.482678 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.481615    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6af42f7e-20f7-4b52-9759-0bf1af4d85b4-systemd-units\") pod \"ovnkube-node-lxg8k\" (UID: \"6af42f7e-20f7-4b52-9759-0bf1af4d85b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxg8k"
Oct 30 00:31:05.482678 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.481634    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a707a5d7-313f-4bef-8baf-a99aeae40d54-system-cni-dir\") pod \"multus-h4hbh\" (UID: \"a707a5d7-313f-4bef-8baf-a99aeae40d54\") " pod="openshift-multus/multus-h4hbh"
Oct 30 00:31:05.482678 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.481633    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6af42f7e-20f7-4b52-9759-0bf1af4d85b4-var-lib-openvswitch\") pod \"ovnkube-node-lxg8k\" (UID: \"6af42f7e-20f7-4b52-9759-0bf1af4d85b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxg8k"
Oct 30 00:31:05.482678 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.481659    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a707a5d7-313f-4bef-8baf-a99aeae40d54-multus-conf-dir\") pod \"multus-h4hbh\" (UID: \"a707a5d7-313f-4bef-8baf-a99aeae40d54\") " pod="openshift-multus/multus-h4hbh"
Oct 30 00:31:05.482678 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.481668    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6af42f7e-20f7-4b52-9759-0bf1af4d85b4-var-lib-openvswitch\") pod \"ovnkube-node-lxg8k\" (UID: \"6af42f7e-20f7-4b52-9759-0bf1af4d85b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxg8k"
Oct 30 00:31:05.482678 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.481668    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c5c3a55b-e259-4550-9b7e-f666a28d217e-cni-binary-copy\") pod \"multus-additional-cni-plugins-zvcpp\" (UID: \"c5c3a55b-e259-4550-9b7e-f666a28d217e\") " pod="openshift-multus/multus-additional-cni-plugins-zvcpp"
Oct 30 00:31:05.482678 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.481700    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a707a5d7-313f-4bef-8baf-a99aeae40d54-multus-daemon-config\") pod \"multus-h4hbh\" (UID: \"a707a5d7-313f-4bef-8baf-a99aeae40d54\") " pod="openshift-multus/multus-h4hbh"
Oct 30 00:31:05.482678 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.481708    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/078fd0e8-1393-45d0-baee-bbcd88095b2b-hosts-file\") pod \"node-resolver-6gjtz\" (UID: \"078fd0e8-1393-45d0-baee-bbcd88095b2b\") " pod="openshift-dns/node-resolver-6gjtz"
Oct 30 00:31:05.482678 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.481730    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/078fd0e8-1393-45d0-baee-bbcd88095b2b-tmp-dir\") pod \"node-resolver-6gjtz\" (UID: \"078fd0e8-1393-45d0-baee-bbcd88095b2b\") " pod="openshift-dns/node-resolver-6gjtz"
Oct 30 00:31:05.482678 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.481756    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6af42f7e-20f7-4b52-9759-0bf1af4d85b4-host-slash\") pod \"ovnkube-node-lxg8k\" (UID: \"6af42f7e-20f7-4b52-9759-0bf1af4d85b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxg8k"
Oct 30 00:31:05.482678 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.481779    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a707a5d7-313f-4bef-8baf-a99aeae40d54-hostroot\") pod \"multus-h4hbh\" (UID: \"a707a5d7-313f-4bef-8baf-a99aeae40d54\") " pod="openshift-multus/multus-h4hbh"
Oct 30 00:31:05.482678 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.481804    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crppr\" (UniqueName: \"kubernetes.io/projected/078fd0e8-1393-45d0-baee-bbcd88095b2b-kube-api-access-crppr\") pod \"node-resolver-6gjtz\" (UID: \"078fd0e8-1393-45d0-baee-bbcd88095b2b\") " pod="openshift-dns/node-resolver-6gjtz"
Oct 30 00:31:05.482678 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.481830    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqmpd\" (UniqueName: \"kubernetes.io/projected/c5c3a55b-e259-4550-9b7e-f666a28d217e-kube-api-access-mqmpd\") pod \"multus-additional-cni-plugins-zvcpp\" (UID: \"c5c3a55b-e259-4550-9b7e-f666a28d217e\") " pod="openshift-multus/multus-additional-cni-plugins-zvcpp"
Oct 30 00:31:05.482678 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.481854    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/88de9f3a-e7d0-400f-9091-46c0b89c1a3c-registration-dir\") pod \"aws-ebs-csi-driver-node-v24xq\" (UID: \"88de9f3a-e7d0-400f-9091-46c0b89c1a3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v24xq"
Oct 30 00:31:05.482678 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.481876    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c5c3a55b-e259-4550-9b7e-f666a28d217e-cnibin\") pod \"multus-additional-cni-plugins-zvcpp\" (UID: \"c5c3a55b-e259-4550-9b7e-f666a28d217e\") " pod="openshift-multus/multus-additional-cni-plugins-zvcpp"
Oct 30 00:31:05.482678 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.481903    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6af42f7e-20f7-4b52-9759-0bf1af4d85b4-ovnkube-config\") pod \"ovnkube-node-lxg8k\" (UID: \"6af42f7e-20f7-4b52-9759-0bf1af4d85b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxg8k"
Oct 30 00:31:05.482678 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.481896    2579 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 00:31:05.483469 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.481928    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6af42f7e-20f7-4b52-9759-0bf1af4d85b4-host-slash\") pod \"ovnkube-node-lxg8k\" (UID: \"6af42f7e-20f7-4b52-9759-0bf1af4d85b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxg8k"
Oct 30 00:31:05.483469 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.481934    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6af42f7e-20f7-4b52-9759-0bf1af4d85b4-run-ovn\") pod \"ovnkube-node-lxg8k\" (UID: \"6af42f7e-20f7-4b52-9759-0bf1af4d85b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxg8k"
Oct 30 00:31:05.483469 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.481969    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6af42f7e-20f7-4b52-9759-0bf1af4d85b4-run-ovn\") pod \"ovnkube-node-lxg8k\" (UID: \"6af42f7e-20f7-4b52-9759-0bf1af4d85b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxg8k"
Oct 30 00:31:05.483469 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.481978    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4606f172-d949-471e-842c-b340b50e179b-etc-modprobe-d\") pod \"tuned-5hvvb\" (UID: \"4606f172-d949-471e-842c-b340b50e179b\") " pod="openshift-cluster-node-tuning-operator/tuned-5hvvb"
Oct 30 00:31:05.483469 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.482012    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af2d6f84-aea3-4698-be12-11bd6767caec-metrics-certs\") pod \"network-metrics-daemon-z58bb\" (UID: \"af2d6f84-aea3-4698-be12-11bd6767caec\") " pod="openshift-multus/network-metrics-daemon-z58bb"
Oct 30 00:31:05.483469 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.482038    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/80603987-09a1-4ca1-b8b4-3c9ac8d0efc6-dbus\") pod \"global-pull-secret-syncer-tz6g8\" (UID: \"80603987-09a1-4ca1-b8b4-3c9ac8d0efc6\") " pod="kube-system/global-pull-secret-syncer-tz6g8"
Oct 30 00:31:05.483469 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.482063    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c1e4fab8-6072-4fcd-9624-a60dee7aed86-host-slash\") pod \"iptables-alerter-kph7f\" (UID: \"c1e4fab8-6072-4fcd-9624-a60dee7aed86\") " pod="openshift-network-operator/iptables-alerter-kph7f"
Oct 30 00:31:05.483469 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.482075    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4606f172-d949-471e-842c-b340b50e179b-etc-modprobe-d\") pod \"tuned-5hvvb\" (UID: \"4606f172-d949-471e-842c-b340b50e179b\") " pod="openshift-cluster-node-tuning-operator/tuned-5hvvb"
Oct 30 00:31:05.483469 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.482088    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/14285dfa-dd38-490e-a486-a1fcfc3d0214-serviceca\") pod \"node-ca-8b6d6\" (UID: \"14285dfa-dd38-490e-a486-a1fcfc3d0214\") " pod="openshift-image-registry/node-ca-8b6d6"
Oct 30 00:31:05.483469 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.482115    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6af42f7e-20f7-4b52-9759-0bf1af4d85b4-host-run-ovn-kubernetes\") pod \"ovnkube-node-lxg8k\" (UID: \"6af42f7e-20f7-4b52-9759-0bf1af4d85b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxg8k"
Oct 30 00:31:05.483469 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.482139    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a707a5d7-313f-4bef-8baf-a99aeae40d54-multus-cni-dir\") pod \"multus-h4hbh\" (UID: \"a707a5d7-313f-4bef-8baf-a99aeae40d54\") " pod="openshift-multus/multus-h4hbh"
Oct 30 00:31:05.483469 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.482165    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4606f172-d949-471e-842c-b340b50e179b-etc-systemd\") pod \"tuned-5hvvb\" (UID: \"4606f172-d949-471e-842c-b340b50e179b\") " pod="openshift-cluster-node-tuning-operator/tuned-5hvvb"
Oct 30 00:31:05.483469 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.482188    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4606f172-d949-471e-842c-b340b50e179b-var-lib-kubelet\") pod \"tuned-5hvvb\" (UID: \"4606f172-d949-471e-842c-b340b50e179b\") " pod="openshift-cluster-node-tuning-operator/tuned-5hvvb"
Oct 30 00:31:05.483469 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.482219    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/88de9f3a-e7d0-400f-9091-46c0b89c1a3c-etc-selinux\") pod \"aws-ebs-csi-driver-node-v24xq\" (UID: \"88de9f3a-e7d0-400f-9091-46c0b89c1a3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v24xq"
Oct 30 00:31:05.483469 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.482242    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a707a5d7-313f-4bef-8baf-a99aeae40d54-host-var-lib-cni-bin\") pod \"multus-h4hbh\" (UID: \"a707a5d7-313f-4bef-8baf-a99aeae40d54\") " pod="openshift-multus/multus-h4hbh"
Oct 30 00:31:05.483469 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.482259    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6af42f7e-20f7-4b52-9759-0bf1af4d85b4-run-openvswitch\") pod \"ovnkube-node-lxg8k\" (UID: \"6af42f7e-20f7-4b52-9759-0bf1af4d85b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxg8k"
Oct 30 00:31:05.483469 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.482259    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c5c3a55b-e259-4550-9b7e-f666a28d217e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zvcpp\" (UID: \"c5c3a55b-e259-4550-9b7e-f666a28d217e\") " pod="openshift-multus/multus-additional-cni-plugins-zvcpp"
Oct 30 00:31:05.483994 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.482266    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6af42f7e-20f7-4b52-9759-0bf1af4d85b4-env-overrides\") pod \"ovnkube-node-lxg8k\" (UID: \"6af42f7e-20f7-4b52-9759-0bf1af4d85b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxg8k"
Oct 30 00:31:05.483994 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.482283    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4606f172-d949-471e-842c-b340b50e179b-etc-kubernetes\") pod \"tuned-5hvvb\" (UID: \"4606f172-d949-471e-842c-b340b50e179b\") " pod="openshift-cluster-node-tuning-operator/tuned-5hvvb"
Oct 30 00:31:05.483994 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.482312    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0d1b347f-c22e-489b-986d-d3b3e01fa36b-agent-certs\") pod \"konnectivity-agent-m5qgr\" (UID: \"0d1b347f-c22e-489b-986d-d3b3e01fa36b\") " pod="kube-system/konnectivity-agent-m5qgr"
Oct 30 00:31:05.483994 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.482318    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c1e4fab8-6072-4fcd-9624-a60dee7aed86-host-slash\") pod \"iptables-alerter-kph7f\" (UID: \"c1e4fab8-6072-4fcd-9624-a60dee7aed86\") " pod="openshift-network-operator/iptables-alerter-kph7f"
Oct 30 00:31:05.483994 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.482353    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/80603987-09a1-4ca1-b8b4-3c9ac8d0efc6-original-pull-secret\") pod \"global-pull-secret-syncer-tz6g8\" (UID: \"80603987-09a1-4ca1-b8b4-3c9ac8d0efc6\") " pod="kube-system/global-pull-secret-syncer-tz6g8"
Oct 30 00:31:05.483994 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.482402    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6af42f7e-20f7-4b52-9759-0bf1af4d85b4-etc-openvswitch\") pod \"ovnkube-node-lxg8k\" (UID: \"6af42f7e-20f7-4b52-9759-0bf1af4d85b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxg8k"
Oct 30 00:31:05.483994 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.482430    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a707a5d7-313f-4bef-8baf-a99aeae40d54-host-run-k8s-cni-cncf-io\") pod \"multus-h4hbh\" (UID: \"a707a5d7-313f-4bef-8baf-a99aeae40d54\") " pod="openshift-multus/multus-h4hbh"
Oct 30 00:31:05.483994 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.482456    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a707a5d7-313f-4bef-8baf-a99aeae40d54-etc-kubernetes\") pod \"multus-h4hbh\" (UID: \"a707a5d7-313f-4bef-8baf-a99aeae40d54\") " pod="openshift-multus/multus-h4hbh"
Oct 30 00:31:05.483994 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.482468    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4606f172-d949-471e-842c-b340b50e179b-etc-sysctl-d\") pod \"tuned-5hvvb\" (UID: \"4606f172-d949-471e-842c-b340b50e179b\") " pod="openshift-cluster-node-tuning-operator/tuned-5hvvb"
Oct 30 00:31:05.483994 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.482481    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4606f172-d949-471e-842c-b340b50e179b-etc-sysctl-conf\") pod \"tuned-5hvvb\" (UID: \"4606f172-d949-471e-842c-b340b50e179b\") " pod="openshift-cluster-node-tuning-operator/tuned-5hvvb"
Oct 30 00:31:05.483994 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.482505    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/88de9f3a-e7d0-400f-9091-46c0b89c1a3c-device-dir\") pod \"aws-ebs-csi-driver-node-v24xq\" (UID: \"88de9f3a-e7d0-400f-9091-46c0b89c1a3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v24xq"
Oct 30 00:31:05.483994 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.482508    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rl4f\" (UniqueName: \"kubernetes.io/projected/4606f172-d949-471e-842c-b340b50e179b-kube-api-access-6rl4f\") pod \"tuned-5hvvb\" (UID: \"4606f172-d949-471e-842c-b340b50e179b\") " pod="openshift-cluster-node-tuning-operator/tuned-5hvvb"
Oct 30 00:31:05.483994 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.482551    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/88de9f3a-e7d0-400f-9091-46c0b89c1a3c-socket-dir\") pod \"aws-ebs-csi-driver-node-v24xq\" (UID: \"88de9f3a-e7d0-400f-9091-46c0b89c1a3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v24xq"
Oct 30 00:31:05.483994 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.482580    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c1e4fab8-6072-4fcd-9624-a60dee7aed86-iptables-alerter-script\") pod \"iptables-alerter-kph7f\" (UID: \"c1e4fab8-6072-4fcd-9624-a60dee7aed86\") " pod="openshift-network-operator/iptables-alerter-kph7f"
Oct 30 00:31:05.483994 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.482598    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/88de9f3a-e7d0-400f-9091-46c0b89c1a3c-registration-dir\") pod \"aws-ebs-csi-driver-node-v24xq\" (UID: \"88de9f3a-e7d0-400f-9091-46c0b89c1a3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v24xq"
Oct 30 00:31:05.483994 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.482606    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6af42f7e-20f7-4b52-9759-0bf1af4d85b4-node-log\") pod \"ovnkube-node-lxg8k\" (UID: \"6af42f7e-20f7-4b52-9759-0bf1af4d85b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxg8k"
Oct 30 00:31:05.483994 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.482646    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c5c3a55b-e259-4550-9b7e-f666a28d217e-cnibin\") pod \"multus-additional-cni-plugins-zvcpp\" (UID: \"c5c3a55b-e259-4550-9b7e-f666a28d217e\") " pod="openshift-multus/multus-additional-cni-plugins-zvcpp"
Oct 30 00:31:05.484520 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.482656    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6af42f7e-20f7-4b52-9759-0bf1af4d85b4-host-kubelet\") pod \"ovnkube-node-lxg8k\" (UID: \"6af42f7e-20f7-4b52-9759-0bf1af4d85b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxg8k"
Oct 30 00:31:05.484520 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.482673    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a707a5d7-313f-4bef-8baf-a99aeae40d54-multus-socket-dir-parent\") pod \"multus-h4hbh\" (UID: \"a707a5d7-313f-4bef-8baf-a99aeae40d54\") " pod="openshift-multus/multus-h4hbh"
Oct 30 00:31:05.484520 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.482686    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a707a5d7-313f-4bef-8baf-a99aeae40d54-multus-daemon-config\") pod \"multus-h4hbh\" (UID: \"a707a5d7-313f-4bef-8baf-a99aeae40d54\") " pod="openshift-multus/multus-h4hbh"
Oct 30 00:31:05.484520 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.482351    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a707a5d7-313f-4bef-8baf-a99aeae40d54-host-var-lib-kubelet\") pod \"multus-h4hbh\" (UID: \"a707a5d7-313f-4bef-8baf-a99aeae40d54\") " pod="openshift-multus/multus-h4hbh"
Oct 30 00:31:05.484520 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.482704    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6af42f7e-20f7-4b52-9759-0bf1af4d85b4-ovnkube-script-lib\") pod \"ovnkube-node-lxg8k\" (UID: \"6af42f7e-20f7-4b52-9759-0bf1af4d85b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxg8k"
Oct 30 00:31:05.484520 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.482320    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a707a5d7-313f-4bef-8baf-a99aeae40d54-hostroot\") pod \"multus-h4hbh\" (UID: \"a707a5d7-313f-4bef-8baf-a99aeae40d54\") " pod="openshift-multus/multus-h4hbh"
Oct 30 00:31:05.484520 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.482739    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a707a5d7-313f-4bef-8baf-a99aeae40d54-cni-binary-copy\") pod \"multus-h4hbh\" (UID: \"a707a5d7-313f-4bef-8baf-a99aeae40d54\") " pod="openshift-multus/multus-h4hbh"
Oct 30 00:31:05.484520 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.482754    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a707a5d7-313f-4bef-8baf-a99aeae40d54-host-run-k8s-cni-cncf-io\") pod \"multus-h4hbh\" (UID: \"a707a5d7-313f-4bef-8baf-a99aeae40d54\") " pod="openshift-multus/multus-h4hbh"
Oct 30 00:31:05.484520 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.482766    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4606f172-d949-471e-842c-b340b50e179b-etc-sysconfig\") pod \"tuned-5hvvb\" (UID: \"4606f172-d949-471e-842c-b340b50e179b\") " pod="openshift-cluster-node-tuning-operator/tuned-5hvvb"
Oct 30 00:31:05.484520 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.482787    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4606f172-d949-471e-842c-b340b50e179b-sys\") pod \"tuned-5hvvb\" (UID: \"4606f172-d949-471e-842c-b340b50e179b\") " pod="openshift-cluster-node-tuning-operator/tuned-5hvvb"
Oct 30 00:31:05.484520 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.482792    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a707a5d7-313f-4bef-8baf-a99aeae40d54-etc-kubernetes\") pod \"multus-h4hbh\" (UID: \"a707a5d7-313f-4bef-8baf-a99aeae40d54\") " pod="openshift-multus/multus-h4hbh"
Oct 30 00:31:05.484520 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.482799    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4606f172-d949-471e-842c-b340b50e179b-run\") pod \"tuned-5hvvb\" (UID: \"4606f172-d949-471e-842c-b340b50e179b\") " pod="openshift-cluster-node-tuning-operator/tuned-5hvvb"
Oct 30 00:31:05.484520 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:05.482866    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 00:31:05.484520 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.482894    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4606f172-d949-471e-842c-b340b50e179b-var-lib-kubelet\") pod \"tuned-5hvvb\" (UID: \"4606f172-d949-471e-842c-b340b50e179b\") " pod="openshift-cluster-node-tuning-operator/tuned-5hvvb"
Oct 30 00:31:05.484520 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:05.482928    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/80603987-09a1-4ca1-b8b4-3c9ac8d0efc6-original-pull-secret podName:80603987-09a1-4ca1-b8b4-3c9ac8d0efc6 nodeName:}" failed. No retries permitted until 2026-10-30 00:31:05.982908422 +0000 UTC m=+3.084215303 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/80603987-09a1-4ca1-b8b4-3c9ac8d0efc6-original-pull-secret") pod "global-pull-secret-syncer-tz6g8" (UID: "80603987-09a1-4ca1-b8b4-3c9ac8d0efc6") : object "kube-system"/"original-pull-secret" not registered
Oct 30 00:31:05.484520 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.482937    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4606f172-d949-471e-842c-b340b50e179b-lib-modules\") pod \"tuned-5hvvb\" (UID: \"4606f172-d949-471e-842c-b340b50e179b\") " pod="openshift-cluster-node-tuning-operator/tuned-5hvvb"
Oct 30 00:31:05.484520 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.482975    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4606f172-d949-471e-842c-b340b50e179b-sys\") pod \"tuned-5hvvb\" (UID: \"4606f172-d949-471e-842c-b340b50e179b\") " pod="openshift-cluster-node-tuning-operator/tuned-5hvvb"
Oct 30 00:31:05.484520 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.483075    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4606f172-d949-471e-842c-b340b50e179b-etc-sysctl-conf\") pod \"tuned-5hvvb\" (UID: \"4606f172-d949-471e-842c-b340b50e179b\") " pod="openshift-cluster-node-tuning-operator/tuned-5hvvb"
Oct 30 00:31:05.485429 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.482867    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4606f172-d949-471e-842c-b340b50e179b-lib-modules\") pod \"tuned-5hvvb\" (UID: \"4606f172-d949-471e-842c-b340b50e179b\") " pod="openshift-cluster-node-tuning-operator/tuned-5hvvb"
Oct 30 00:31:05.485429 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.483170    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4606f172-d949-471e-842c-b340b50e179b-tmp\") pod \"tuned-5hvvb\" (UID: \"4606f172-d949-471e-842c-b340b50e179b\") " pod="openshift-cluster-node-tuning-operator/tuned-5hvvb"
Oct 30 00:31:05.485429 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.483196    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/88de9f3a-e7d0-400f-9091-46c0b89c1a3c-sys-fs\") pod \"aws-ebs-csi-driver-node-v24xq\" (UID: \"88de9f3a-e7d0-400f-9091-46c0b89c1a3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v24xq"
Oct 30 00:31:05.485429 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.483222    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/80603987-09a1-4ca1-b8b4-3c9ac8d0efc6-kubelet-config\") pod \"global-pull-secret-syncer-tz6g8\" (UID: \"80603987-09a1-4ca1-b8b4-3c9ac8d0efc6\") " pod="kube-system/global-pull-secret-syncer-tz6g8"
Oct 30 00:31:05.485429 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.483245    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jzc7\" (UniqueName: \"kubernetes.io/projected/c1e4fab8-6072-4fcd-9624-a60dee7aed86-kube-api-access-2jzc7\") pod \"iptables-alerter-kph7f\" (UID: \"c1e4fab8-6072-4fcd-9624-a60dee7aed86\") " pod="openshift-network-operator/iptables-alerter-kph7f"
Oct 30 00:31:05.485429 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.483266    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6af42f7e-20f7-4b52-9759-0bf1af4d85b4-log-socket\") pod \"ovnkube-node-lxg8k\" (UID: \"6af42f7e-20f7-4b52-9759-0bf1af4d85b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxg8k"
Oct 30 00:31:05.485429 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.483337    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6af42f7e-20f7-4b52-9759-0bf1af4d85b4-host-cni-bin\") pod \"ovnkube-node-lxg8k\" (UID: \"6af42f7e-20f7-4b52-9759-0bf1af4d85b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxg8k"
Oct 30 00:31:05.485429 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.482188    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/078fd0e8-1393-45d0-baee-bbcd88095b2b-tmp-dir\") pod \"node-resolver-6gjtz\" (UID: \"078fd0e8-1393-45d0-baee-bbcd88095b2b\") " pod="openshift-dns/node-resolver-6gjtz"
Oct 30 00:31:05.485429 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.483486    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6af42f7e-20f7-4b52-9759-0bf1af4d85b4-host-cni-bin\") pod \"ovnkube-node-lxg8k\" (UID: \"6af42f7e-20f7-4b52-9759-0bf1af4d85b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxg8k"
Oct 30 00:31:05.485429 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.483510    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/88de9f3a-e7d0-400f-9091-46c0b89c1a3c-etc-selinux\") pod \"aws-ebs-csi-driver-node-v24xq\" (UID: \"88de9f3a-e7d0-400f-9091-46c0b89c1a3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v24xq"
Oct 30 00:31:05.485429 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.483523    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6af42f7e-20f7-4b52-9759-0bf1af4d85b4-log-socket\") pod \"ovnkube-node-lxg8k\" (UID: \"6af42f7e-20f7-4b52-9759-0bf1af4d85b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxg8k"
Oct 30 00:31:05.485429 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.482189    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c5c3a55b-e259-4550-9b7e-f666a28d217e-cni-binary-copy\") pod \"multus-additional-cni-plugins-zvcpp\" (UID: \"c5c3a55b-e259-4550-9b7e-f666a28d217e\") " pod="openshift-multus/multus-additional-cni-plugins-zvcpp"
Oct 30 00:31:05.485429 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.483540    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0d1b347f-c22e-489b-986d-d3b3e01fa36b-konnectivity-ca\") pod \"konnectivity-agent-m5qgr\" (UID: \"0d1b347f-c22e-489b-986d-d3b3e01fa36b\") " pod="kube-system/konnectivity-agent-m5qgr"
Oct 30 00:31:05.485429 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:05.483550    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 00:31:05.485429 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.483573    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6af42f7e-20f7-4b52-9759-0bf1af4d85b4-run-openvswitch\") pod \"ovnkube-node-lxg8k\" (UID: \"6af42f7e-20f7-4b52-9759-0bf1af4d85b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxg8k"
Oct 30 00:31:05.485429 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.483617    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4606f172-d949-471e-842c-b340b50e179b-etc-kubernetes\") pod \"tuned-5hvvb\" (UID: \"4606f172-d949-471e-842c-b340b50e179b\") " pod="openshift-cluster-node-tuning-operator/tuned-5hvvb"
Oct 30 00:31:05.485429 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:05.483606    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/af2d6f84-aea3-4698-be12-11bd6767caec-metrics-certs podName:af2d6f84-aea3-4698-be12-11bd6767caec nodeName:}" failed. No retries permitted until 2026-10-30 00:31:05.983588424 +0000 UTC m=+3.084895299 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/af2d6f84-aea3-4698-be12-11bd6767caec-metrics-certs") pod "network-metrics-daemon-z58bb" (UID: "af2d6f84-aea3-4698-be12-11bd6767caec") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 00:31:05.486406 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.483668    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6af42f7e-20f7-4b52-9759-0bf1af4d85b4-etc-openvswitch\") pod \"ovnkube-node-lxg8k\" (UID: \"6af42f7e-20f7-4b52-9759-0bf1af4d85b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxg8k"
Oct 30 00:31:05.486406 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.483779    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/80603987-09a1-4ca1-b8b4-3c9ac8d0efc6-kubelet-config\") pod \"global-pull-secret-syncer-tz6g8\" (UID: \"80603987-09a1-4ca1-b8b4-3c9ac8d0efc6\") " pod="kube-system/global-pull-secret-syncer-tz6g8"
Oct 30 00:31:05.486406 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.483827    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/88de9f3a-e7d0-400f-9091-46c0b89c1a3c-sys-fs\") pod \"aws-ebs-csi-driver-node-v24xq\" (UID: \"88de9f3a-e7d0-400f-9091-46c0b89c1a3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v24xq"
Oct 30 00:31:05.486406 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.483859    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/80603987-09a1-4ca1-b8b4-3c9ac8d0efc6-dbus\") pod \"global-pull-secret-syncer-tz6g8\" (UID: \"80603987-09a1-4ca1-b8b4-3c9ac8d0efc6\") " pod="kube-system/global-pull-secret-syncer-tz6g8"
Oct 30 00:31:05.486406 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.483905    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6af42f7e-20f7-4b52-9759-0bf1af4d85b4-run-systemd\") pod \"ovnkube-node-lxg8k\" (UID: \"6af42f7e-20f7-4b52-9759-0bf1af4d85b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxg8k"
Oct 30 00:31:05.486406 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.484013    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6af42f7e-20f7-4b52-9759-0bf1af4d85b4-ovnkube-script-lib\") pod \"ovnkube-node-lxg8k\" (UID: \"6af42f7e-20f7-4b52-9759-0bf1af4d85b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxg8k"
Oct 30 00:31:05.486406 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.484082    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a707a5d7-313f-4bef-8baf-a99aeae40d54-multus-socket-dir-parent\") pod \"multus-h4hbh\" (UID: \"a707a5d7-313f-4bef-8baf-a99aeae40d54\") " pod="openshift-multus/multus-h4hbh"
Oct 30 00:31:05.486406 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.484152    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a707a5d7-313f-4bef-8baf-a99aeae40d54-host-var-lib-cni-multus\") pod \"multus-h4hbh\" (UID: \"a707a5d7-313f-4bef-8baf-a99aeae40d54\") " pod="openshift-multus/multus-h4hbh"
Oct 30 00:31:05.486406 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.484177    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6af42f7e-20f7-4b52-9759-0bf1af4d85b4-ovnkube-config\") pod \"ovnkube-node-lxg8k\" (UID: \"6af42f7e-20f7-4b52-9759-0bf1af4d85b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxg8k"
Oct 30 00:31:05.486406 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.484215    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a707a5d7-313f-4bef-8baf-a99aeae40d54-cnibin\") pod \"multus-h4hbh\" (UID: \"a707a5d7-313f-4bef-8baf-a99aeae40d54\") " pod="openshift-multus/multus-h4hbh"
Oct 30 00:31:05.486406 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.484253    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6af42f7e-20f7-4b52-9759-0bf1af4d85b4-node-log\") pod \"ovnkube-node-lxg8k\" (UID: \"6af42f7e-20f7-4b52-9759-0bf1af4d85b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxg8k"
Oct 30 00:31:05.486406 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.484305    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4606f172-d949-471e-842c-b340b50e179b-etc-systemd\") pod \"tuned-5hvvb\" (UID: \"4606f172-d949-471e-842c-b340b50e179b\") " pod="openshift-cluster-node-tuning-operator/tuned-5hvvb"
Oct 30 00:31:05.486406 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.484359    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a707a5d7-313f-4bef-8baf-a99aeae40d54-multus-cni-dir\") pod \"multus-h4hbh\" (UID: \"a707a5d7-313f-4bef-8baf-a99aeae40d54\") " pod="openshift-multus/multus-h4hbh"
Oct 30 00:31:05.486406 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.484409    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6af42f7e-20f7-4b52-9759-0bf1af4d85b4-host-run-ovn-kubernetes\") pod \"ovnkube-node-lxg8k\" (UID: \"6af42f7e-20f7-4b52-9759-0bf1af4d85b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxg8k"
Oct 30 00:31:05.486406 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.484444    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c5c3a55b-e259-4550-9b7e-f666a28d217e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zvcpp\" (UID: \"c5c3a55b-e259-4550-9b7e-f666a28d217e\") " pod="openshift-multus/multus-additional-cni-plugins-zvcpp"
Oct 30 00:31:05.486406 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.484487    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/88de9f3a-e7d0-400f-9091-46c0b89c1a3c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-v24xq\" (UID: \"88de9f3a-e7d0-400f-9091-46c0b89c1a3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v24xq"
Oct 30 00:31:05.486406 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.484522    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/88de9f3a-e7d0-400f-9091-46c0b89c1a3c-socket-dir\") pod \"aws-ebs-csi-driver-node-v24xq\" (UID: \"88de9f3a-e7d0-400f-9091-46c0b89c1a3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v24xq"
Oct 30 00:31:05.486406 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.484578    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4606f172-d949-471e-842c-b340b50e179b-etc-sysconfig\") pod \"tuned-5hvvb\" (UID: \"4606f172-d949-471e-842c-b340b50e179b\") " pod="openshift-cluster-node-tuning-operator/tuned-5hvvb"
Oct 30 00:31:05.487030 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.484630    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a707a5d7-313f-4bef-8baf-a99aeae40d54-cni-binary-copy\") pod \"multus-h4hbh\" (UID: \"a707a5d7-313f-4bef-8baf-a99aeae40d54\") " pod="openshift-multus/multus-h4hbh"
Oct 30 00:31:05.487030 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.484807    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/14285dfa-dd38-490e-a486-a1fcfc3d0214-serviceca\") pod \"node-ca-8b6d6\" (UID: \"14285dfa-dd38-490e-a486-a1fcfc3d0214\") " pod="openshift-image-registry/node-ca-8b6d6"
Oct 30 00:31:05.487030 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.485138    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c1e4fab8-6072-4fcd-9624-a60dee7aed86-iptables-alerter-script\") pod \"iptables-alerter-kph7f\" (UID: \"c1e4fab8-6072-4fcd-9624-a60dee7aed86\") " pod="openshift-network-operator/iptables-alerter-kph7f"
Oct 30 00:31:05.487030 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.485997    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4606f172-d949-471e-842c-b340b50e179b-etc-tuned\") pod \"tuned-5hvvb\" (UID: \"4606f172-d949-471e-842c-b340b50e179b\") " pod="openshift-cluster-node-tuning-operator/tuned-5hvvb"
Oct 30 00:31:05.487030 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.486315    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0d1b347f-c22e-489b-986d-d3b3e01fa36b-agent-certs\") pod \"konnectivity-agent-m5qgr\" (UID: \"0d1b347f-c22e-489b-986d-d3b3e01fa36b\") " pod="kube-system/konnectivity-agent-m5qgr"
Oct 30 00:31:05.487258 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.487103    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4606f172-d949-471e-842c-b340b50e179b-tmp\") pod \"tuned-5hvvb\" (UID: \"4606f172-d949-471e-842c-b340b50e179b\") " pod="openshift-cluster-node-tuning-operator/tuned-5hvvb"
Oct 30 00:31:05.487258 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.487211    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6af42f7e-20f7-4b52-9759-0bf1af4d85b4-ovn-node-metrics-cert\") pod \"ovnkube-node-lxg8k\" (UID: \"6af42f7e-20f7-4b52-9759-0bf1af4d85b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxg8k"
Oct 30 00:31:05.502641 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:05.502572    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 00:31:05.502641 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:05.502604    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 00:31:05.502641 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:05.502617    2579 projected.go:196] Error preparing data for projected volume kube-api-access-r998s for pod openshift-network-diagnostics/network-check-target-lngrr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 00:31:05.502868 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:05.502686    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d27ee5b1-d1aa-418c-9169-12cac37dcc2c-kube-api-access-r998s podName:d27ee5b1-d1aa-418c-9169-12cac37dcc2c nodeName:}" failed. No retries permitted until 2026-10-30 00:31:06.002665068 +0000 UTC m=+3.103971940 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-r998s" (UniqueName: "kubernetes.io/projected/d27ee5b1-d1aa-418c-9169-12cac37dcc2c-kube-api-access-r998s") pod "network-check-target-lngrr" (UID: "d27ee5b1-d1aa-418c-9169-12cac37dcc2c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 00:31:05.506276 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.505711    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxcbj\" (UniqueName: \"kubernetes.io/projected/88de9f3a-e7d0-400f-9091-46c0b89c1a3c-kube-api-access-nxcbj\") pod \"aws-ebs-csi-driver-node-v24xq\" (UID: \"88de9f3a-e7d0-400f-9091-46c0b89c1a3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v24xq"
Oct 30 00:31:05.506276 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.506217    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4d7j7\" (UniqueName: \"kubernetes.io/projected/af2d6f84-aea3-4698-be12-11bd6767caec-kube-api-access-4d7j7\") pod \"network-metrics-daemon-z58bb\" (UID: \"af2d6f84-aea3-4698-be12-11bd6767caec\") " pod="openshift-multus/network-metrics-daemon-z58bb"
Oct 30 00:31:05.507040 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.507009    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lbm4\" (UniqueName: \"kubernetes.io/projected/6af42f7e-20f7-4b52-9759-0bf1af4d85b4-kube-api-access-5lbm4\") pod \"ovnkube-node-lxg8k\" (UID: \"6af42f7e-20f7-4b52-9759-0bf1af4d85b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxg8k"
Oct 30 00:31:05.507302 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.507276    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-crppr\" (UniqueName: \"kubernetes.io/projected/078fd0e8-1393-45d0-baee-bbcd88095b2b-kube-api-access-crppr\") pod \"node-resolver-6gjtz\" (UID: \"078fd0e8-1393-45d0-baee-bbcd88095b2b\") " pod="openshift-dns/node-resolver-6gjtz"
Oct 30 00:31:05.508863 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.508783    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6llvc\" (UniqueName: \"kubernetes.io/projected/14285dfa-dd38-490e-a486-a1fcfc3d0214-kube-api-access-6llvc\") pod \"node-ca-8b6d6\" (UID: \"14285dfa-dd38-490e-a486-a1fcfc3d0214\") " pod="openshift-image-registry/node-ca-8b6d6"
Oct 30 00:31:05.521501 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.521477    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rl4f\" (UniqueName: \"kubernetes.io/projected/4606f172-d949-471e-842c-b340b50e179b-kube-api-access-6rl4f\") pod \"tuned-5hvvb\" (UID: \"4606f172-d949-471e-842c-b340b50e179b\") " pod="openshift-cluster-node-tuning-operator/tuned-5hvvb"
Oct 30 00:31:05.521889 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.521869    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqmpd\" (UniqueName: \"kubernetes.io/projected/c5c3a55b-e259-4550-9b7e-f666a28d217e-kube-api-access-mqmpd\") pod \"multus-additional-cni-plugins-zvcpp\" (UID: \"c5c3a55b-e259-4550-9b7e-f666a28d217e\") " pod="openshift-multus/multus-additional-cni-plugins-zvcpp"
Oct 30 00:31:05.521963 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.521939    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4cc6\" (UniqueName: \"kubernetes.io/projected/a707a5d7-313f-4bef-8baf-a99aeae40d54-kube-api-access-h4cc6\") pod \"multus-h4hbh\" (UID: \"a707a5d7-313f-4bef-8baf-a99aeae40d54\") " pod="openshift-multus/multus-h4hbh"
Oct 30 00:31:05.523535 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.523514    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jzc7\" (UniqueName: \"kubernetes.io/projected/c1e4fab8-6072-4fcd-9624-a60dee7aed86-kube-api-access-2jzc7\") pod \"iptables-alerter-kph7f\" (UID: \"c1e4fab8-6072-4fcd-9624-a60dee7aed86\") " pod="openshift-network-operator/iptables-alerter-kph7f"
Oct 30 00:31:05.601431 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.601398    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-kph7f"
Oct 30 00:31:05.601696 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.601669    2579 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 00:31:05.611662 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.611642    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-5hvvb"
Oct 30 00:31:05.612901 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:05.612863    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1e4fab8_6072_4fcd_9624_a60dee7aed86.slice/crio-fe1ce4b21e681d416ffba2a01a9f38924ef7afc592d8e253123303359bbdf086 WatchSource:0}: Error finding container fe1ce4b21e681d416ffba2a01a9f38924ef7afc592d8e253123303359bbdf086: Status 404 returned error can't find the container with id fe1ce4b21e681d416ffba2a01a9f38924ef7afc592d8e253123303359bbdf086
Oct 30 00:31:05.620902 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.620878    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-8b6d6"
Oct 30 00:31:05.621489 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:05.621464    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4606f172_d949_471e_842c_b340b50e179b.slice/crio-c7c9c16173cff184f1fd09e41dd1d8cb8e34d02ae447aa0ee2d901b05d22c048 WatchSource:0}: Error finding container c7c9c16173cff184f1fd09e41dd1d8cb8e34d02ae447aa0ee2d901b05d22c048: Status 404 returned error can't find the container with id c7c9c16173cff184f1fd09e41dd1d8cb8e34d02ae447aa0ee2d901b05d22c048
Oct 30 00:31:05.631570 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:05.631540    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14285dfa_dd38_490e_a486_a1fcfc3d0214.slice/crio-ea478e19d2f09e1d112adbc723346aa5cdea945578e9f53cf3b2cb2fda738476 WatchSource:0}: Error finding container ea478e19d2f09e1d112adbc723346aa5cdea945578e9f53cf3b2cb2fda738476: Status 404 returned error can't find the container with id ea478e19d2f09e1d112adbc723346aa5cdea945578e9f53cf3b2cb2fda738476
Oct 30 00:31:05.645948 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.645923    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-lxg8k"
Oct 30 00:31:05.648056 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.648035    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-h4hbh"
Oct 30 00:31:05.657314 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.657290    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-6gjtz"
Oct 30 00:31:05.658096 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:05.658062    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6af42f7e_20f7_4b52_9759_0bf1af4d85b4.slice/crio-b1739fe883553d159fc4a3b364b09e5ac408869eefc63b025dccf4c5273b7f8b WatchSource:0}: Error finding container b1739fe883553d159fc4a3b364b09e5ac408869eefc63b025dccf4c5273b7f8b: Status 404 returned error can't find the container with id b1739fe883553d159fc4a3b364b09e5ac408869eefc63b025dccf4c5273b7f8b
Oct 30 00:31:05.661849 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.661743    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-zvcpp"
Oct 30 00:31:05.661849 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:05.661826    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda707a5d7_313f_4bef_8baf_a99aeae40d54.slice/crio-650dac582e52282d5b674397229aa50ac75a19dc3768efb9eb967a5f252fc938 WatchSource:0}: Error finding container 650dac582e52282d5b674397229aa50ac75a19dc3768efb9eb967a5f252fc938: Status 404 returned error can't find the container with id 650dac582e52282d5b674397229aa50ac75a19dc3768efb9eb967a5f252fc938
Oct 30 00:31:05.663021 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.662998    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-m5qgr"
Oct 30 00:31:05.670567 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.670547    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v24xq"
Oct 30 00:31:05.676615 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:05.676590    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod078fd0e8_1393_45d0_baee_bbcd88095b2b.slice/crio-ba10707cdb72fd5b7212466e2efb1d9bf1fe0a9077139125e96f32d1d0295515 WatchSource:0}: Error finding container ba10707cdb72fd5b7212466e2efb1d9bf1fe0a9077139125e96f32d1d0295515: Status 404 returned error can't find the container with id ba10707cdb72fd5b7212466e2efb1d9bf1fe0a9077139125e96f32d1d0295515
Oct 30 00:31:05.680800 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:05.680776    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5c3a55b_e259_4550_9b7e_f666a28d217e.slice/crio-94e5870821441f01b75fd1499c09731d42f372db8af82d71e594850adef78e99 WatchSource:0}: Error finding container 94e5870821441f01b75fd1499c09731d42f372db8af82d71e594850adef78e99: Status 404 returned error can't find the container with id 94e5870821441f01b75fd1499c09731d42f372db8af82d71e594850adef78e99
Oct 30 00:31:05.684459 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:05.684433    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d1b347f_c22e_489b_986d_d3b3e01fa36b.slice/crio-bb5c2326fc6156453ddd2fd1052eb32784198ed45e9d6f91e0b3e693c64e57ef WatchSource:0}: Error finding container bb5c2326fc6156453ddd2fd1052eb32784198ed45e9d6f91e0b3e693c64e57ef: Status 404 returned error can't find the container with id bb5c2326fc6156453ddd2fd1052eb32784198ed45e9d6f91e0b3e693c64e57ef
Oct 30 00:31:05.685568 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:05.685034    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88de9f3a_e7d0_400f_9091_46c0b89c1a3c.slice/crio-83e0c503316f6412a51e1550577df70d9ffcbe1c49b03e2cbec69f56d9cbce9e WatchSource:0}: Error finding container 83e0c503316f6412a51e1550577df70d9ffcbe1c49b03e2cbec69f56d9cbce9e: Status 404 returned error can't find the container with id 83e0c503316f6412a51e1550577df70d9ffcbe1c49b03e2cbec69f56d9cbce9e
Oct 30 00:31:05.986143 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.986108    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af2d6f84-aea3-4698-be12-11bd6767caec-metrics-certs\") pod \"network-metrics-daemon-z58bb\" (UID: \"af2d6f84-aea3-4698-be12-11bd6767caec\") " pod="openshift-multus/network-metrics-daemon-z58bb"
Oct 30 00:31:05.986271 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:05.986159    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/80603987-09a1-4ca1-b8b4-3c9ac8d0efc6-original-pull-secret\") pod \"global-pull-secret-syncer-tz6g8\" (UID: \"80603987-09a1-4ca1-b8b4-3c9ac8d0efc6\") " pod="kube-system/global-pull-secret-syncer-tz6g8"
Oct 30 00:31:05.986343 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:05.986287    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 00:31:05.986442 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:05.986349    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/80603987-09a1-4ca1-b8b4-3c9ac8d0efc6-original-pull-secret podName:80603987-09a1-4ca1-b8b4-3c9ac8d0efc6 nodeName:}" failed. No retries permitted until 2026-10-30 00:31:06.986330753 +0000 UTC m=+4.087637626 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/80603987-09a1-4ca1-b8b4-3c9ac8d0efc6-original-pull-secret") pod "global-pull-secret-syncer-tz6g8" (UID: "80603987-09a1-4ca1-b8b4-3c9ac8d0efc6") : object "kube-system"/"original-pull-secret" not registered
Oct 30 00:31:05.986740 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:05.986720    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 00:31:05.986833 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:05.986774    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/af2d6f84-aea3-4698-be12-11bd6767caec-metrics-certs podName:af2d6f84-aea3-4698-be12-11bd6767caec nodeName:}" failed. No retries permitted until 2026-10-30 00:31:06.986759962 +0000 UTC m=+4.088066821 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/af2d6f84-aea3-4698-be12-11bd6767caec-metrics-certs") pod "network-metrics-daemon-z58bb" (UID: "af2d6f84-aea3-4698-be12-11bd6767caec") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 00:31:06.087328 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:06.087245    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r998s\" (UniqueName: \"kubernetes.io/projected/d27ee5b1-d1aa-418c-9169-12cac37dcc2c-kube-api-access-r998s\") pod \"network-check-target-lngrr\" (UID: \"d27ee5b1-d1aa-418c-9169-12cac37dcc2c\") " pod="openshift-network-diagnostics/network-check-target-lngrr"
Oct 30 00:31:06.087511 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:06.087414    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 00:31:06.087511 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:06.087437    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 00:31:06.087511 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:06.087448    2579 projected.go:196] Error preparing data for projected volume kube-api-access-r998s for pod openshift-network-diagnostics/network-check-target-lngrr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 00:31:06.087511 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:06.087506    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d27ee5b1-d1aa-418c-9169-12cac37dcc2c-kube-api-access-r998s podName:d27ee5b1-d1aa-418c-9169-12cac37dcc2c nodeName:}" failed. No retries permitted until 2026-10-30 00:31:07.087488736 +0000 UTC m=+4.188795609 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-r998s" (UniqueName: "kubernetes.io/projected/d27ee5b1-d1aa-418c-9169-12cac37dcc2c-kube-api-access-r998s") pod "network-check-target-lngrr" (UID: "d27ee5b1-d1aa-418c-9169-12cac37dcc2c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 00:31:06.323328 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:06.323238    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 00:26:05 +0000 UTC" deadline="2028-06-24 15:36:04.179442752 +0000 UTC"
Oct 30 00:31:06.323328 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:06.323281    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14487h4m57.856165118s"
Oct 30 00:31:06.398660 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:06.398075    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tz6g8"
Oct 30 00:31:06.398660 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:06.398206    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tz6g8" podUID="80603987-09a1-4ca1-b8b4-3c9ac8d0efc6"
Oct 30 00:31:06.421270 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:06.421174    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-5hvvb" event={"ID":"4606f172-d949-471e-842c-b340b50e179b","Type":"ContainerStarted","Data":"c7c9c16173cff184f1fd09e41dd1d8cb8e34d02ae447aa0ee2d901b05d22c048"}
Oct 30 00:31:06.437102 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:06.437062    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-7.ec2.internal" event={"ID":"74c81293f9236d86a7e367354c3aa34d","Type":"ContainerStarted","Data":"47de7b78b4a22546f051141196d2e5cb33bdc3753050239027f020d7c3b47306"}
Oct 30 00:31:06.446416 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:06.445255    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v24xq" event={"ID":"88de9f3a-e7d0-400f-9091-46c0b89c1a3c","Type":"ContainerStarted","Data":"83e0c503316f6412a51e1550577df70d9ffcbe1c49b03e2cbec69f56d9cbce9e"}
Oct 30 00:31:06.448963 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:06.448279    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-m5qgr" event={"ID":"0d1b347f-c22e-489b-986d-d3b3e01fa36b","Type":"ContainerStarted","Data":"bb5c2326fc6156453ddd2fd1052eb32784198ed45e9d6f91e0b3e693c64e57ef"}
Oct 30 00:31:06.451635 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:06.450527    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zvcpp" event={"ID":"c5c3a55b-e259-4550-9b7e-f666a28d217e","Type":"ContainerStarted","Data":"94e5870821441f01b75fd1499c09731d42f372db8af82d71e594850adef78e99"}
Oct 30 00:31:06.452738 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:06.452673    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-6gjtz" event={"ID":"078fd0e8-1393-45d0-baee-bbcd88095b2b","Type":"ContainerStarted","Data":"ba10707cdb72fd5b7212466e2efb1d9bf1fe0a9077139125e96f32d1d0295515"}
Oct 30 00:31:06.456165 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:06.456072    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lxg8k" event={"ID":"6af42f7e-20f7-4b52-9759-0bf1af4d85b4","Type":"ContainerStarted","Data":"b1739fe883553d159fc4a3b364b09e5ac408869eefc63b025dccf4c5273b7f8b"}
Oct 30 00:31:06.458924 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:06.458852    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-kph7f" event={"ID":"c1e4fab8-6072-4fcd-9624-a60dee7aed86","Type":"ContainerStarted","Data":"fe1ce4b21e681d416ffba2a01a9f38924ef7afc592d8e253123303359bbdf086"}
Oct 30 00:31:06.467917 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:06.467888    2579 generic.go:356] "Generic (PLEG): container finished" podID="70c88bf4a25fa0f049c2cc951e3cbb56" containerID="7375b6e950fa89b8e5c9eff221fc4e4c8532d97031eb86c570e03bfe63bb5b21" exitCode=0
Oct 30 00:31:06.468013 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:06.467967    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-7.ec2.internal" event={"ID":"70c88bf4a25fa0f049c2cc951e3cbb56","Type":"ContainerDied","Data":"7375b6e950fa89b8e5c9eff221fc4e4c8532d97031eb86c570e03bfe63bb5b21"}
Oct 30 00:31:06.479227 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:06.479161    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-h4hbh" event={"ID":"a707a5d7-313f-4bef-8baf-a99aeae40d54","Type":"ContainerStarted","Data":"650dac582e52282d5b674397229aa50ac75a19dc3768efb9eb967a5f252fc938"}
Oct 30 00:31:06.489515 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:06.489487    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-8b6d6" event={"ID":"14285dfa-dd38-490e-a486-a1fcfc3d0214","Type":"ContainerStarted","Data":"ea478e19d2f09e1d112adbc723346aa5cdea945578e9f53cf3b2cb2fda738476"}
Oct 30 00:31:06.516944 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:06.516542    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-7.ec2.internal" podStartSLOduration=2.516522604 podStartE2EDuration="2.516522604s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 00:31:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:31:06.475017261 +0000 UTC m=+3.576324143" watchObservedRunningTime="2026-10-30 00:31:06.516522604 +0000 UTC m=+3.617829485"
Oct 30 00:31:06.996198 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:06.996104    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af2d6f84-aea3-4698-be12-11bd6767caec-metrics-certs\") pod \"network-metrics-daemon-z58bb\" (UID: \"af2d6f84-aea3-4698-be12-11bd6767caec\") " pod="openshift-multus/network-metrics-daemon-z58bb"
Oct 30 00:31:06.996198 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:06.996169    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/80603987-09a1-4ca1-b8b4-3c9ac8d0efc6-original-pull-secret\") pod \"global-pull-secret-syncer-tz6g8\" (UID: \"80603987-09a1-4ca1-b8b4-3c9ac8d0efc6\") " pod="kube-system/global-pull-secret-syncer-tz6g8"
Oct 30 00:31:06.996439 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:06.996321    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 00:31:06.996439 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:06.996426    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/80603987-09a1-4ca1-b8b4-3c9ac8d0efc6-original-pull-secret podName:80603987-09a1-4ca1-b8b4-3c9ac8d0efc6 nodeName:}" failed. No retries permitted until 2026-10-30 00:31:08.996405033 +0000 UTC m=+6.097711910 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/80603987-09a1-4ca1-b8b4-3c9ac8d0efc6-original-pull-secret") pod "global-pull-secret-syncer-tz6g8" (UID: "80603987-09a1-4ca1-b8b4-3c9ac8d0efc6") : object "kube-system"/"original-pull-secret" not registered
Oct 30 00:31:06.996957 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:06.996886    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 00:31:06.996957 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:06.996946    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/af2d6f84-aea3-4698-be12-11bd6767caec-metrics-certs podName:af2d6f84-aea3-4698-be12-11bd6767caec nodeName:}" failed. No retries permitted until 2026-10-30 00:31:08.996929305 +0000 UTC m=+6.098236178 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/af2d6f84-aea3-4698-be12-11bd6767caec-metrics-certs") pod "network-metrics-daemon-z58bb" (UID: "af2d6f84-aea3-4698-be12-11bd6767caec") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 00:31:07.097650 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:07.096996    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r998s\" (UniqueName: \"kubernetes.io/projected/d27ee5b1-d1aa-418c-9169-12cac37dcc2c-kube-api-access-r998s\") pod \"network-check-target-lngrr\" (UID: \"d27ee5b1-d1aa-418c-9169-12cac37dcc2c\") " pod="openshift-network-diagnostics/network-check-target-lngrr"
Oct 30 00:31:07.097650 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:07.097223    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 00:31:07.097650 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:07.097246    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 00:31:07.097650 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:07.097257    2579 projected.go:196] Error preparing data for projected volume kube-api-access-r998s for pod openshift-network-diagnostics/network-check-target-lngrr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 00:31:07.097650 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:07.097317    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d27ee5b1-d1aa-418c-9169-12cac37dcc2c-kube-api-access-r998s podName:d27ee5b1-d1aa-418c-9169-12cac37dcc2c nodeName:}" failed. No retries permitted until 2026-10-30 00:31:09.097297237 +0000 UTC m=+6.198604111 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-r998s" (UniqueName: "kubernetes.io/projected/d27ee5b1-d1aa-418c-9169-12cac37dcc2c-kube-api-access-r998s") pod "network-check-target-lngrr" (UID: "d27ee5b1-d1aa-418c-9169-12cac37dcc2c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 00:31:07.324307 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:07.324192    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 00:26:05 +0000 UTC" deadline="2028-06-21 17:43:16.045183263 +0000 UTC"
Oct 30 00:31:07.324307 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:07.324258    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14417h12m8.720929254s"
Oct 30 00:31:07.398632 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:07.398540    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lngrr"
Oct 30 00:31:07.398632 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:07.398574    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z58bb"
Oct 30 00:31:07.398839 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:07.398667    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lngrr" podUID="d27ee5b1-d1aa-418c-9169-12cac37dcc2c"
Oct 30 00:31:07.398839 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:07.398806    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z58bb" podUID="af2d6f84-aea3-4698-be12-11bd6767caec"
Oct 30 00:31:07.501432 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:07.501351    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-7.ec2.internal" event={"ID":"70c88bf4a25fa0f049c2cc951e3cbb56","Type":"ContainerStarted","Data":"b7d66edb9a643c62c5f736c2f427a27e5f0213c1ef36ea340207f7db63f6399c"}
Oct 30 00:31:08.397989 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:08.397504    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tz6g8"
Oct 30 00:31:08.397989 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:08.397644    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tz6g8" podUID="80603987-09a1-4ca1-b8b4-3c9ac8d0efc6"
Oct 30 00:31:08.508158 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:08.508118    2579 generic.go:356] "Generic (PLEG): container finished" podID="c5c3a55b-e259-4550-9b7e-f666a28d217e" containerID="1a3e6d299f1bab1bdd66a93dc0a01da38b8a0fc3ac0dec8c3281e45a1cf57945" exitCode=0
Oct 30 00:31:08.509171 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:08.509145    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zvcpp" event={"ID":"c5c3a55b-e259-4550-9b7e-f666a28d217e","Type":"ContainerDied","Data":"1a3e6d299f1bab1bdd66a93dc0a01da38b8a0fc3ac0dec8c3281e45a1cf57945"}
Oct 30 00:31:08.574495 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:08.573373    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-7.ec2.internal" podStartSLOduration=4.573352363 podStartE2EDuration="4.573352363s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 00:31:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:31:07.540431565 +0000 UTC m=+4.641738447" watchObservedRunningTime="2026-10-30 00:31:08.573352363 +0000 UTC m=+5.674659243"
Oct 30 00:31:09.011369 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:09.010528    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af2d6f84-aea3-4698-be12-11bd6767caec-metrics-certs\") pod \"network-metrics-daemon-z58bb\" (UID: \"af2d6f84-aea3-4698-be12-11bd6767caec\") " pod="openshift-multus/network-metrics-daemon-z58bb"
Oct 30 00:31:09.011369 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:09.010595    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/80603987-09a1-4ca1-b8b4-3c9ac8d0efc6-original-pull-secret\") pod \"global-pull-secret-syncer-tz6g8\" (UID: \"80603987-09a1-4ca1-b8b4-3c9ac8d0efc6\") " pod="kube-system/global-pull-secret-syncer-tz6g8"
Oct 30 00:31:09.011369 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:09.010763    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 00:31:09.011369 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:09.010827    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/80603987-09a1-4ca1-b8b4-3c9ac8d0efc6-original-pull-secret podName:80603987-09a1-4ca1-b8b4-3c9ac8d0efc6 nodeName:}" failed. No retries permitted until 2026-10-30 00:31:13.010809738 +0000 UTC m=+10.112116616 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/80603987-09a1-4ca1-b8b4-3c9ac8d0efc6-original-pull-secret") pod "global-pull-secret-syncer-tz6g8" (UID: "80603987-09a1-4ca1-b8b4-3c9ac8d0efc6") : object "kube-system"/"original-pull-secret" not registered
Oct 30 00:31:09.011369 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:09.011225    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 00:31:09.011369 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:09.011279    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/af2d6f84-aea3-4698-be12-11bd6767caec-metrics-certs podName:af2d6f84-aea3-4698-be12-11bd6767caec nodeName:}" failed. No retries permitted until 2026-10-30 00:31:13.011263537 +0000 UTC m=+10.112570410 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/af2d6f84-aea3-4698-be12-11bd6767caec-metrics-certs") pod "network-metrics-daemon-z58bb" (UID: "af2d6f84-aea3-4698-be12-11bd6767caec") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 00:31:09.111685 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:09.111646    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r998s\" (UniqueName: \"kubernetes.io/projected/d27ee5b1-d1aa-418c-9169-12cac37dcc2c-kube-api-access-r998s\") pod \"network-check-target-lngrr\" (UID: \"d27ee5b1-d1aa-418c-9169-12cac37dcc2c\") " pod="openshift-network-diagnostics/network-check-target-lngrr"
Oct 30 00:31:09.111852 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:09.111830    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 00:31:09.111900 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:09.111865    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 00:31:09.111900 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:09.111879    2579 projected.go:196] Error preparing data for projected volume kube-api-access-r998s for pod openshift-network-diagnostics/network-check-target-lngrr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 00:31:09.111970 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:09.111943    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d27ee5b1-d1aa-418c-9169-12cac37dcc2c-kube-api-access-r998s podName:d27ee5b1-d1aa-418c-9169-12cac37dcc2c nodeName:}" failed. No retries permitted until 2026-10-30 00:31:13.111924574 +0000 UTC m=+10.213231449 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-r998s" (UniqueName: "kubernetes.io/projected/d27ee5b1-d1aa-418c-9169-12cac37dcc2c-kube-api-access-r998s") pod "network-check-target-lngrr" (UID: "d27ee5b1-d1aa-418c-9169-12cac37dcc2c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 00:31:09.397348 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:09.397180    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lngrr"
Oct 30 00:31:09.397348 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:09.397301    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lngrr" podUID="d27ee5b1-d1aa-418c-9169-12cac37dcc2c"
Oct 30 00:31:09.397802 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:09.397654    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z58bb"
Oct 30 00:31:09.397802 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:09.397769    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z58bb" podUID="af2d6f84-aea3-4698-be12-11bd6767caec"
Oct 30 00:31:09.523934 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:09.523239    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-m5qgr" event={"ID":"0d1b347f-c22e-489b-986d-d3b3e01fa36b","Type":"ContainerStarted","Data":"a2082b35fe1abd4da111a75c799393d50ab905422297dfbe5d4baac265abeef7"}
Oct 30 00:31:10.397948 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:10.397879    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tz6g8"
Oct 30 00:31:10.398315 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:10.398286    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tz6g8" podUID="80603987-09a1-4ca1-b8b4-3c9ac8d0efc6"
Oct 30 00:31:11.064691 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:11.064448    2579 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-m5qgr"
Oct 30 00:31:11.066252 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:11.066012    2579 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-m5qgr"
Oct 30 00:31:11.102623 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:11.102569    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-m5qgr" podStartSLOduration=4.627527847 podStartE2EDuration="8.102555684s" totalImagesPullingTime="3.475027837s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:31:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:31:09.560982744 +0000 UTC m=+6.662289626" watchObservedRunningTime="2026-10-30 00:31:11.102555684 +0000 UTC m=+8.203862613"
Oct 30 00:31:11.398576 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:11.398541    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lngrr"
Oct 30 00:31:11.398743 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:11.398588    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z58bb"
Oct 30 00:31:11.398743 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:11.398686    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lngrr" podUID="d27ee5b1-d1aa-418c-9169-12cac37dcc2c"
Oct 30 00:31:11.399070 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:11.398983    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z58bb" podUID="af2d6f84-aea3-4698-be12-11bd6767caec"
Oct 30 00:31:11.534014 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:11.533975    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-8b6d6" event={"ID":"14285dfa-dd38-490e-a486-a1fcfc3d0214","Type":"ContainerStarted","Data":"b6980ccd42244d9aa95d0d61f413b694c7c45f0276ceaa67a3d62ed30d8519d6"}
Oct 30 00:31:11.538608 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:11.538576    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v24xq" event={"ID":"88de9f3a-e7d0-400f-9091-46c0b89c1a3c","Type":"ContainerStarted","Data":"dcc34460742d67d8137be9101f56b409e0d7fed34d14bd4674121fc429ac37b0"}
Oct 30 00:31:11.571093 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:11.571032    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-8b6d6" podStartSLOduration=3.5420686630000002 podStartE2EDuration="8.571012965s" totalImagesPullingTime="5.028944302s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 00:31:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:31:11.570543469 +0000 UTC m=+8.671850349" watchObservedRunningTime="2026-10-30 00:31:11.571012965 +0000 UTC m=+8.672319847"
Oct 30 00:31:12.397577 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:12.397504    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tz6g8"
Oct 30 00:31:12.397923 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:12.397693    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tz6g8" podUID="80603987-09a1-4ca1-b8b4-3c9ac8d0efc6"
Oct 30 00:31:12.539652 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:12.539627    2579 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 00:31:12.705333 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:12.705306    2579 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 00:31:13.043607 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:13.043556    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af2d6f84-aea3-4698-be12-11bd6767caec-metrics-certs\") pod \"network-metrics-daemon-z58bb\" (UID: \"af2d6f84-aea3-4698-be12-11bd6767caec\") " pod="openshift-multus/network-metrics-daemon-z58bb"
Oct 30 00:31:13.043728 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:13.043639    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/80603987-09a1-4ca1-b8b4-3c9ac8d0efc6-original-pull-secret\") pod \"global-pull-secret-syncer-tz6g8\" (UID: \"80603987-09a1-4ca1-b8b4-3c9ac8d0efc6\") " pod="kube-system/global-pull-secret-syncer-tz6g8"
Oct 30 00:31:13.043830 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:13.043809    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 00:31:13.043902 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:13.043890    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/80603987-09a1-4ca1-b8b4-3c9ac8d0efc6-original-pull-secret podName:80603987-09a1-4ca1-b8b4-3c9ac8d0efc6 nodeName:}" failed. No retries permitted until 2026-10-30 00:31:21.043871638 +0000 UTC m=+18.145178502 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/80603987-09a1-4ca1-b8b4-3c9ac8d0efc6-original-pull-secret") pod "global-pull-secret-syncer-tz6g8" (UID: "80603987-09a1-4ca1-b8b4-3c9ac8d0efc6") : object "kube-system"/"original-pull-secret" not registered
Oct 30 00:31:13.043967 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:13.043897    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 00:31:13.044015 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:13.043973    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/af2d6f84-aea3-4698-be12-11bd6767caec-metrics-certs podName:af2d6f84-aea3-4698-be12-11bd6767caec nodeName:}" failed. No retries permitted until 2026-10-30 00:31:21.043955574 +0000 UTC m=+18.145262452 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/af2d6f84-aea3-4698-be12-11bd6767caec-metrics-certs") pod "network-metrics-daemon-z58bb" (UID: "af2d6f84-aea3-4698-be12-11bd6767caec") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 00:31:13.144651 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:13.144609    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r998s\" (UniqueName: \"kubernetes.io/projected/d27ee5b1-d1aa-418c-9169-12cac37dcc2c-kube-api-access-r998s\") pod \"network-check-target-lngrr\" (UID: \"d27ee5b1-d1aa-418c-9169-12cac37dcc2c\") " pod="openshift-network-diagnostics/network-check-target-lngrr"
Oct 30 00:31:13.144824 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:13.144802    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 00:31:13.144902 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:13.144834    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 00:31:13.144902 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:13.144848    2579 projected.go:196] Error preparing data for projected volume kube-api-access-r998s for pod openshift-network-diagnostics/network-check-target-lngrr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 00:31:13.145008 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:13.144912    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d27ee5b1-d1aa-418c-9169-12cac37dcc2c-kube-api-access-r998s podName:d27ee5b1-d1aa-418c-9169-12cac37dcc2c nodeName:}" failed. No retries permitted until 2026-10-30 00:31:21.144893552 +0000 UTC m=+18.246200413 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-r998s" (UniqueName: "kubernetes.io/projected/d27ee5b1-d1aa-418c-9169-12cac37dcc2c-kube-api-access-r998s") pod "network-check-target-lngrr" (UID: "d27ee5b1-d1aa-418c-9169-12cac37dcc2c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 00:31:13.326755 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:13.326667    2579 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T00:31:12.705330862Z","UUID":"1899431c-e56c-46ea-a517-4668e59ea82b","Handler":null,"Name":"","Endpoint":""}
Oct 30 00:31:13.330671 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:13.330102    2579 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 00:31:13.330671 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:13.330134    2579 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 00:31:13.399438 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:13.397740    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lngrr"
Oct 30 00:31:13.399438 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:13.397848    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lngrr" podUID="d27ee5b1-d1aa-418c-9169-12cac37dcc2c"
Oct 30 00:31:13.399438 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:13.397922    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z58bb"
Oct 30 00:31:13.399438 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:13.398022    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z58bb" podUID="af2d6f84-aea3-4698-be12-11bd6767caec"
Oct 30 00:31:13.544710 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:13.544635    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-h4hbh" event={"ID":"a707a5d7-313f-4bef-8baf-a99aeae40d54","Type":"ContainerStarted","Data":"f75fb99bb9c1447f54c09cd1532ec07f9a339d6791b88e303c270bee29c10153"}
Oct 30 00:31:13.551817 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:13.551783    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v24xq" event={"ID":"88de9f3a-e7d0-400f-9091-46c0b89c1a3c","Type":"ContainerStarted","Data":"059ad87cb2b94d4b11d2b3d1672d8f483be4864d3fc993117acdd466d2362ef0"}
Oct 30 00:31:13.554355 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:13.554326    2579 generic.go:356] "Generic (PLEG): container finished" podID="c5c3a55b-e259-4550-9b7e-f666a28d217e" containerID="7eecf80608ddf0d75051dd83a9c697ebb3371db9c26c54e64a6bdbb819d9bfd4" exitCode=0
Oct 30 00:31:13.554477 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:13.554365    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zvcpp" event={"ID":"c5c3a55b-e259-4550-9b7e-f666a28d217e","Type":"ContainerDied","Data":"7eecf80608ddf0d75051dd83a9c697ebb3371db9c26c54e64a6bdbb819d9bfd4"}
Oct 30 00:31:13.656890 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:13.656647    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-h4hbh" podStartSLOduration=3.295016047 podStartE2EDuration="10.656628s" totalImagesPullingTime="7.361611953s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 00:31:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:31:13.585126718 +0000 UTC m=+10.686433599" watchObservedRunningTime="2026-10-30 00:31:13.656628 +0000 UTC m=+10.757934882"
Oct 30 00:31:14.397566 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:14.397529    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tz6g8"
Oct 30 00:31:14.397740 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:14.397723    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tz6g8" podUID="80603987-09a1-4ca1-b8b4-3c9ac8d0efc6"
Oct 30 00:31:14.560600 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:14.559337    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v24xq" event={"ID":"88de9f3a-e7d0-400f-9091-46c0b89c1a3c","Type":"ContainerStarted","Data":"c7a65e6412684533a021e597ab4229c0adaf33b0cfe335ddc7c56bf929a2b1d7"}
Oct 30 00:31:14.562678 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:14.562642    2579 generic.go:356] "Generic (PLEG): container finished" podID="c5c3a55b-e259-4550-9b7e-f666a28d217e" containerID="ff59758a0d4edcf893f2a339ed6b9d1afe635bdb842b9d7bf4fac94408084cd7" exitCode=0
Oct 30 00:31:14.562833 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:14.562801    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zvcpp" event={"ID":"c5c3a55b-e259-4550-9b7e-f666a28d217e","Type":"ContainerDied","Data":"ff59758a0d4edcf893f2a339ed6b9d1afe635bdb842b9d7bf4fac94408084cd7"}
Oct 30 00:31:14.565533 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:14.565500    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-6gjtz" event={"ID":"078fd0e8-1393-45d0-baee-bbcd88095b2b","Type":"ContainerStarted","Data":"fa65b43bcb4ac62a933b305d1ba9b72e3b647b6d07976cdf52d2d56b97d9b24b"}
Oct 30 00:31:14.598504 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:14.598451    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v24xq" podStartSLOduration=3.8785975649999997 podStartE2EDuration="11.598438505s" totalImagesPullingTime="7.71984094s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 00:31:03 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:31:14.597755021 +0000 UTC m=+11.699061899" watchObservedRunningTime="2026-10-30 00:31:14.598438505 +0000 UTC m=+11.699745386"
Oct 30 00:31:14.633523 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:14.633460    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-6gjtz" podStartSLOduration=3.805092617 podStartE2EDuration="11.633443485s" totalImagesPullingTime="7.828350868s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 00:31:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:31:14.632768259 +0000 UTC m=+11.734075141" watchObservedRunningTime="2026-10-30 00:31:14.633443485 +0000 UTC m=+11.734750372"
Oct 30 00:31:15.397293 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:15.397251    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z58bb"
Oct 30 00:31:15.397470 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:15.397305    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lngrr"
Oct 30 00:31:15.397470 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:15.397408    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z58bb" podUID="af2d6f84-aea3-4698-be12-11bd6767caec"
Oct 30 00:31:15.397470 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:15.397461    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lngrr" podUID="d27ee5b1-d1aa-418c-9169-12cac37dcc2c"
Oct 30 00:31:15.570278 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:15.570243    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-kph7f" event={"ID":"c1e4fab8-6072-4fcd-9624-a60dee7aed86","Type":"ContainerStarted","Data":"6ab217781889693e3d21cdde318a6d28bbe1b0c98633128bf5da9b0a48000f31"}
Oct 30 00:31:15.606285 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:15.606228    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-kph7f" podStartSLOduration=5.036276899 podStartE2EDuration="12.60621091s" totalImagesPullingTime="7.569934011s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 00:31:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:31:15.606057164 +0000 UTC m=+12.707364046" watchObservedRunningTime="2026-10-30 00:31:15.60621091 +0000 UTC m=+12.707517791"
Oct 30 00:31:16.397631 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:16.397597    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tz6g8"
Oct 30 00:31:16.397780 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:16.397731    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tz6g8" podUID="80603987-09a1-4ca1-b8b4-3c9ac8d0efc6"
Oct 30 00:31:16.575116 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:16.575084    2579 generic.go:356] "Generic (PLEG): container finished" podID="c5c3a55b-e259-4550-9b7e-f666a28d217e" containerID="a86eb29daa94f829c573baa241891539cd0aff7b9834a1bac65d2b9267c79082" exitCode=0
Oct 30 00:31:16.575843 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:16.575208    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zvcpp" event={"ID":"c5c3a55b-e259-4550-9b7e-f666a28d217e","Type":"ContainerDied","Data":"a86eb29daa94f829c573baa241891539cd0aff7b9834a1bac65d2b9267c79082"}
Oct 30 00:31:16.577741 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:16.577717    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-5hvvb" event={"ID":"4606f172-d949-471e-842c-b340b50e179b","Type":"ContainerStarted","Data":"ad6b37ce9db47c99a3da5ea26f6ff33bb31493d87cc0e6c8388dab78857fa625"}
Oct 30 00:31:16.672145 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:16.672052    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-5hvvb" podStartSLOduration=3.761373919 podStartE2EDuration="13.672038792s" totalImagesPullingTime="9.910664873s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 00:31:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:31:16.671654397 +0000 UTC m=+13.772961294" watchObservedRunningTime="2026-10-30 00:31:16.672038792 +0000 UTC m=+13.773345679"
Oct 30 00:31:17.397215 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:17.397176    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lngrr"
Oct 30 00:31:17.397393 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:17.397223    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z58bb"
Oct 30 00:31:17.397393 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:17.397318    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lngrr" podUID="d27ee5b1-d1aa-418c-9169-12cac37dcc2c"
Oct 30 00:31:17.397505 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:17.397470    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z58bb" podUID="af2d6f84-aea3-4698-be12-11bd6767caec"
Oct 30 00:31:18.397770 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:18.397517    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tz6g8"
Oct 30 00:31:18.398226 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:18.397889    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tz6g8" podUID="80603987-09a1-4ca1-b8b4-3c9ac8d0efc6"
Oct 30 00:31:18.584953 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:18.584846    2579 generic.go:356] "Generic (PLEG): container finished" podID="c5c3a55b-e259-4550-9b7e-f666a28d217e" containerID="491a677a8fe80eed5b32874c6d29a09b4ced36528c56603427e807956d22e14a" exitCode=0
Oct 30 00:31:18.584953 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:18.584894    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zvcpp" event={"ID":"c5c3a55b-e259-4550-9b7e-f666a28d217e","Type":"ContainerDied","Data":"491a677a8fe80eed5b32874c6d29a09b4ced36528c56603427e807956d22e14a"}
Oct 30 00:31:19.398136 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:19.398109    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z58bb"
Oct 30 00:31:19.398544 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:19.398209    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z58bb" podUID="af2d6f84-aea3-4698-be12-11bd6767caec"
Oct 30 00:31:19.398544 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:19.398243    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lngrr"
Oct 30 00:31:19.398544 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:19.398288    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lngrr" podUID="d27ee5b1-d1aa-418c-9169-12cac37dcc2c"
Oct 30 00:31:19.590332 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:19.590287    2579 generic.go:356] "Generic (PLEG): container finished" podID="c5c3a55b-e259-4550-9b7e-f666a28d217e" containerID="0d6fce3030a94ab80f89f6dd8b2466fc094db4bc0f801aad3abe83f650b258f9" exitCode=0
Oct 30 00:31:19.590576 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:19.590555    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zvcpp" event={"ID":"c5c3a55b-e259-4550-9b7e-f666a28d217e","Type":"ContainerDied","Data":"0d6fce3030a94ab80f89f6dd8b2466fc094db4bc0f801aad3abe83f650b258f9"}
Oct 30 00:31:20.397732 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:20.397711    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tz6g8"
Oct 30 00:31:20.397857 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:20.397835    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tz6g8" podUID="80603987-09a1-4ca1-b8b4-3c9ac8d0efc6"
Oct 30 00:31:20.595322 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:20.595291    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lxg8k" event={"ID":"6af42f7e-20f7-4b52-9759-0bf1af4d85b4","Type":"ContainerStarted","Data":"651b31b2943965e29ad01f6125d0e62d902cdf70f2962ba8714df0dc15649bfe"}
Oct 30 00:31:20.595742 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:20.595393    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lxg8k" event={"ID":"6af42f7e-20f7-4b52-9759-0bf1af4d85b4","Type":"ContainerStarted","Data":"a13b39bd6aec1fef8e202030e422a9bd87dddea7d72a8c900599d4f918820142"}
Oct 30 00:31:20.595742 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:20.595431    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lxg8k" event={"ID":"6af42f7e-20f7-4b52-9759-0bf1af4d85b4","Type":"ContainerStarted","Data":"44eb951a602f9fbdfd2fecfda907373edd9f9a8e0b95fedc89683f234aad1e22"}
Oct 30 00:31:20.595742 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:20.595460    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lxg8k" event={"ID":"6af42f7e-20f7-4b52-9759-0bf1af4d85b4","Type":"ContainerStarted","Data":"282a61c2213470ef39c47c5428b44bc85ef682c13609b4f01f218863239b1813"}
Oct 30 00:31:20.595742 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:20.595492    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lxg8k" event={"ID":"6af42f7e-20f7-4b52-9759-0bf1af4d85b4","Type":"ContainerStarted","Data":"7d0bdd3d6bf4dc5783ffc2eebdf58e86c14bc4b40a99cb56fc1d6de7341532e4"}
Oct 30 00:31:20.595742 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:20.595523    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lxg8k" event={"ID":"6af42f7e-20f7-4b52-9759-0bf1af4d85b4","Type":"ContainerStarted","Data":"da189fca77dbf2c27aed66af29c887ca216f59291823a996ab14b5545243b87f"}
Oct 30 00:31:20.598594 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:20.598569    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zvcpp" event={"ID":"c5c3a55b-e259-4550-9b7e-f666a28d217e","Type":"ContainerStarted","Data":"c8640e58b16257ad0d48fe111e9048267cb78d973d0bb16003f1cf2db018ef3c"}
Oct 30 00:31:20.662685 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:20.662635    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-zvcpp" podStartSLOduration=7.417916145 podStartE2EDuration="17.662621712s" totalImagesPullingTime="10.244705567s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 00:31:03 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:31:20.650428043 +0000 UTC m=+17.751734923" watchObservedRunningTime="2026-10-30 00:31:20.662621712 +0000 UTC m=+17.763928589"
Oct 30 00:31:21.100750 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:21.100702    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af2d6f84-aea3-4698-be12-11bd6767caec-metrics-certs\") pod \"network-metrics-daemon-z58bb\" (UID: \"af2d6f84-aea3-4698-be12-11bd6767caec\") " pod="openshift-multus/network-metrics-daemon-z58bb"
Oct 30 00:31:21.100750 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:21.100754    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/80603987-09a1-4ca1-b8b4-3c9ac8d0efc6-original-pull-secret\") pod \"global-pull-secret-syncer-tz6g8\" (UID: \"80603987-09a1-4ca1-b8b4-3c9ac8d0efc6\") " pod="kube-system/global-pull-secret-syncer-tz6g8"
Oct 30 00:31:21.101003 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:21.100864    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 00:31:21.101003 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:21.100921    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/80603987-09a1-4ca1-b8b4-3c9ac8d0efc6-original-pull-secret podName:80603987-09a1-4ca1-b8b4-3c9ac8d0efc6 nodeName:}" failed. No retries permitted until 2026-10-30 00:31:37.100906169 +0000 UTC m=+34.202213027 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/80603987-09a1-4ca1-b8b4-3c9ac8d0efc6-original-pull-secret") pod "global-pull-secret-syncer-tz6g8" (UID: "80603987-09a1-4ca1-b8b4-3c9ac8d0efc6") : object "kube-system"/"original-pull-secret" not registered
Oct 30 00:31:21.101003 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:21.100864    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 00:31:21.101003 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:21.100990    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/af2d6f84-aea3-4698-be12-11bd6767caec-metrics-certs podName:af2d6f84-aea3-4698-be12-11bd6767caec nodeName:}" failed. No retries permitted until 2026-10-30 00:31:37.100978324 +0000 UTC m=+34.202285197 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/af2d6f84-aea3-4698-be12-11bd6767caec-metrics-certs") pod "network-metrics-daemon-z58bb" (UID: "af2d6f84-aea3-4698-be12-11bd6767caec") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 00:31:21.201067 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:21.201029    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r998s\" (UniqueName: \"kubernetes.io/projected/d27ee5b1-d1aa-418c-9169-12cac37dcc2c-kube-api-access-r998s\") pod \"network-check-target-lngrr\" (UID: \"d27ee5b1-d1aa-418c-9169-12cac37dcc2c\") " pod="openshift-network-diagnostics/network-check-target-lngrr"
Oct 30 00:31:21.201233 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:21.201154    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 00:31:21.201233 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:21.201170    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 00:31:21.201233 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:21.201179    2579 projected.go:196] Error preparing data for projected volume kube-api-access-r998s for pod openshift-network-diagnostics/network-check-target-lngrr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 00:31:21.201233 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:21.201221    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d27ee5b1-d1aa-418c-9169-12cac37dcc2c-kube-api-access-r998s podName:d27ee5b1-d1aa-418c-9169-12cac37dcc2c nodeName:}" failed. No retries permitted until 2026-10-30 00:31:37.201210298 +0000 UTC m=+34.302517156 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-r998s" (UniqueName: "kubernetes.io/projected/d27ee5b1-d1aa-418c-9169-12cac37dcc2c-kube-api-access-r998s") pod "network-check-target-lngrr" (UID: "d27ee5b1-d1aa-418c-9169-12cac37dcc2c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 00:31:21.398202 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:21.398123    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z58bb"
Oct 30 00:31:21.398322 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:21.398123    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lngrr"
Oct 30 00:31:21.398322 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:21.398235    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z58bb" podUID="af2d6f84-aea3-4698-be12-11bd6767caec"
Oct 30 00:31:21.398322 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:21.398300    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lngrr" podUID="d27ee5b1-d1aa-418c-9169-12cac37dcc2c"
Oct 30 00:31:22.397276 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:22.397238    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tz6g8"
Oct 30 00:31:22.397664 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:22.397341    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tz6g8" podUID="80603987-09a1-4ca1-b8b4-3c9ac8d0efc6"
Oct 30 00:31:22.606998 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:22.606970    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lxg8k" event={"ID":"6af42f7e-20f7-4b52-9759-0bf1af4d85b4","Type":"ContainerStarted","Data":"ad1a9423271198a6ed1f5eea43ca29337d34c98af192065d6213e91e63ec39e8"}
Oct 30 00:31:23.398695 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:23.398661    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lngrr"
Oct 30 00:31:23.399219 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:23.398781    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lngrr" podUID="d27ee5b1-d1aa-418c-9169-12cac37dcc2c"
Oct 30 00:31:23.399219 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:23.398804    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z58bb"
Oct 30 00:31:23.399219 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:23.398928    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z58bb" podUID="af2d6f84-aea3-4698-be12-11bd6767caec"
Oct 30 00:31:24.397521 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:24.397478    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tz6g8"
Oct 30 00:31:24.397690 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:24.397603    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tz6g8" podUID="80603987-09a1-4ca1-b8b4-3c9ac8d0efc6"
Oct 30 00:31:25.400396 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:25.400354    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z58bb"
Oct 30 00:31:25.401017 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:25.400468    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z58bb" podUID="af2d6f84-aea3-4698-be12-11bd6767caec"
Oct 30 00:31:25.401017 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:25.400521    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lngrr"
Oct 30 00:31:25.401017 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:25.400598    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lngrr" podUID="d27ee5b1-d1aa-418c-9169-12cac37dcc2c"
Oct 30 00:31:25.616822 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:25.616788    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lxg8k" event={"ID":"6af42f7e-20f7-4b52-9759-0bf1af4d85b4","Type":"ContainerStarted","Data":"a57fb595c88bde3142030ae9431ef1aa18db64002d712f01f6f84e4871f0d510"}
Oct 30 00:31:25.617070 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:25.617055    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-lxg8k"
Oct 30 00:31:25.617148 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:25.617113    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-lxg8k"
Oct 30 00:31:25.632276 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:25.632255    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lxg8k"
Oct 30 00:31:25.696722 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:25.696629    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-lxg8k" podStartSLOduration=8.167869363 podStartE2EDuration="22.696599212s" totalImagesPullingTime="14.528729849s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 00:31:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:31:25.696500967 +0000 UTC m=+22.797807847" watchObservedRunningTime="2026-10-30 00:31:25.696599212 +0000 UTC m=+22.797906093"
Oct 30 00:31:26.397841 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:26.397679    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tz6g8"
Oct 30 00:31:26.397980 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:26.397910    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tz6g8" podUID="80603987-09a1-4ca1-b8b4-3c9ac8d0efc6"
Oct 30 00:31:26.620521 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:26.620493    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-lxg8k"
Oct 30 00:31:26.636763 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:26.636738    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lxg8k"
Oct 30 00:31:26.746859 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:26.746754    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-lngrr"]
Oct 30 00:31:26.747009 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:26.746920    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lngrr"
Oct 30 00:31:26.747058 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:26.747039    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lngrr" podUID="d27ee5b1-d1aa-418c-9169-12cac37dcc2c"
Oct 30 00:31:26.747315 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:26.747294    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-tz6g8"]
Oct 30 00:31:26.747415 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:26.747404    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tz6g8"
Oct 30 00:31:26.747498 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:26.747481    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tz6g8" podUID="80603987-09a1-4ca1-b8b4-3c9ac8d0efc6"
Oct 30 00:31:26.749363 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:26.749343    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-z58bb"]
Oct 30 00:31:26.749466 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:26.749449    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z58bb"
Oct 30 00:31:26.749529 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:26.749516    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z58bb" podUID="af2d6f84-aea3-4698-be12-11bd6767caec"
Oct 30 00:31:28.397268 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:28.397240    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tz6g8"
Oct 30 00:31:28.397757 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:28.397240    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z58bb"
Oct 30 00:31:28.397757 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:28.397344    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tz6g8" podUID="80603987-09a1-4ca1-b8b4-3c9ac8d0efc6"
Oct 30 00:31:28.397757 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:28.397240    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lngrr"
Oct 30 00:31:28.397757 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:28.397429    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z58bb" podUID="af2d6f84-aea3-4698-be12-11bd6767caec"
Oct 30 00:31:28.397757 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:28.397511    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lngrr" podUID="d27ee5b1-d1aa-418c-9169-12cac37dcc2c"
Oct 30 00:31:30.397392 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:30.397352    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z58bb"
Oct 30 00:31:30.397809 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:30.397468    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tz6g8"
Oct 30 00:31:30.397809 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:30.397477    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z58bb" podUID="af2d6f84-aea3-4698-be12-11bd6767caec"
Oct 30 00:31:30.397809 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:30.397498    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lngrr"
Oct 30 00:31:30.397809 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:30.397571    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tz6g8" podUID="80603987-09a1-4ca1-b8b4-3c9ac8d0efc6"
Oct 30 00:31:30.397809 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:30.397634    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lngrr" podUID="d27ee5b1-d1aa-418c-9169-12cac37dcc2c"
Oct 30 00:31:32.397274 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:32.397246    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z58bb"
Oct 30 00:31:32.397662 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:32.397347    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lngrr"
Oct 30 00:31:32.397662 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:32.397486    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tz6g8"
Oct 30 00:31:32.397662 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:32.397502    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z58bb" podUID="af2d6f84-aea3-4698-be12-11bd6767caec"
Oct 30 00:31:32.397662 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:32.397592    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tz6g8" podUID="80603987-09a1-4ca1-b8b4-3c9ac8d0efc6"
Oct 30 00:31:32.397810 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:32.397675    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lngrr" podUID="d27ee5b1-d1aa-418c-9169-12cac37dcc2c"
Oct 30 00:31:32.685264 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:32.685186    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-7.ec2.internal" event="NodeReady"
Oct 30 00:31:32.685447 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:32.685337    2579 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Oct 30 00:31:32.742659 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:32.742628    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-fbdvl"]
Oct 30 00:31:32.749653 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:32.749630    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-cqjlz"]
Oct 30 00:31:32.749794 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:32.749771    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-fbdvl"
Oct 30 00:31:32.753649 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:32.753631    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-f2b4d"]
Oct 30 00:31:32.753777 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:32.753760    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-cqjlz"
Oct 30 00:31:32.766707 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:32.766685    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-zqdbk"]
Oct 30 00:31:32.766850 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:32.766781    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-f2b4d"
Oct 30 00:31:32.771988 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:32.771956    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\""
Oct 30 00:31:32.771988 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:32.771974    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\""
Oct 30 00:31:32.775781 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:32.775544    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-btj6v"]
Oct 30 00:31:32.777108 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:32.777083    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-zqdbk"
Oct 30 00:31:32.782704 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:32.782683    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-85f7b79cf7-h2tpp"]
Oct 30 00:31:32.782908 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:32.782880    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-btj6v"
Oct 30 00:31:32.798019 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:32.797966    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Oct 30 00:31:32.798254 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:32.798209    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-vk9xp\""
Oct 30 00:31:32.804696 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:32.804667    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-kjsj6"]
Oct 30 00:31:32.804805 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:32.804789    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-85f7b79cf7-h2tpp"
Oct 30 00:31:32.820539 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:32.818338    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-246g2"]
Oct 30 00:31:32.820539 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:32.818471    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-kjsj6"
Oct 30 00:31:32.840475 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:32.840447    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-sklth"]
Oct 30 00:31:32.840582 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:32.840560    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-246g2"
Oct 30 00:31:32.854301 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:32.854276    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-sxknf"]
Oct 30 00:31:32.854423 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:32.854412    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-sklth"
Oct 30 00:31:32.869529 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:32.869504    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-bjzgs"]
Oct 30 00:31:32.869624 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:32.869615    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-sxknf"
Oct 30 00:31:32.871462 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:32.871440    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Oct 30 00:31:32.871740 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:32.871717    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Oct 30 00:31:32.871841 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:32.871764    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Oct 30 00:31:32.871841 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:32.871776    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Oct 30 00:31:32.871841 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:32.871786    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Oct 30 00:31:32.871841 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:32.871777    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 30 00:31:32.871841 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:32.871812    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 30 00:31:32.872401 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:32.871990    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 00:31:32.872401 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:32.872195    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 30 00:31:32.872401 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:32.872258    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 30 00:31:32.872401 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:32.872270    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Oct 30 00:31:32.872401 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:32.872346    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Oct 30 00:31:32.875137 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:32.875050    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Oct 30 00:31:32.880192 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:32.880174    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/a3d234b3-76d4-47d5-98a4-dafd83477b17-stats-auth\") pod \"router-default-85f7b79cf7-h2tpp\" (UID: \"a3d234b3-76d4-47d5-98a4-dafd83477b17\") " pod="openshift-ingress/router-default-85f7b79cf7-h2tpp"
Oct 30 00:31:32.880288 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:32.880198    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvgs7\" (UniqueName: \"kubernetes.io/projected/783c7a98-5ebf-44dd-8da9-797e71c28169-kube-api-access-vvgs7\") pod \"kas-connection-checker-685895c7c9-fbdvl\" (UID: \"783c7a98-5ebf-44dd-8da9-797e71c28169\") " pod="kube-system/kas-connection-checker-685895c7c9-fbdvl"
Oct 30 00:31:32.880288 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:32.880213    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/a3d234b3-76d4-47d5-98a4-dafd83477b17-default-certificate\") pod \"router-default-85f7b79cf7-h2tpp\" (UID: \"a3d234b3-76d4-47d5-98a4-dafd83477b17\") " pod="openshift-ingress/router-default-85f7b79cf7-h2tpp"
Oct 30 00:31:32.880288 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:32.880228    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a3d234b3-76d4-47d5-98a4-dafd83477b17-service-ca-bundle\") pod \"router-default-85f7b79cf7-h2tpp\" (UID: \"a3d234b3-76d4-47d5-98a4-dafd83477b17\") " pod="openshift-ingress/router-default-85f7b79cf7-h2tpp"
Oct 30 00:31:32.880475 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:32.880322    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/9aba3099-fe4c-4846-9a34-25db2c2fe593-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-kjsj6\" (UID: \"9aba3099-fe4c-4846-9a34-25db2c2fe593\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-kjsj6"
Oct 30 00:31:32.880475 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:32.880367    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/251cf359-f9f5-44ae-8ba9-42bf0cabb1ab-config-volume\") pod \"dns-default-btj6v\" (UID: \"251cf359-f9f5-44ae-8ba9-42bf0cabb1ab\") " pod="openshift-dns/dns-default-btj6v"
Oct 30 00:31:32.880566 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:32.880470    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjmbq\" (UniqueName: \"kubernetes.io/projected/a3d234b3-76d4-47d5-98a4-dafd83477b17-kube-api-access-bjmbq\") pod \"router-default-85f7b79cf7-h2tpp\" (UID: \"a3d234b3-76d4-47d5-98a4-dafd83477b17\") " pod="openshift-ingress/router-default-85f7b79cf7-h2tpp"
Oct 30 00:31:32.880566 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:32.880504    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d7c5e976-596e-4cb5-81a9-b16262b3c85b-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-zqdbk\" (UID: \"d7c5e976-596e-4cb5-81a9-b16262b3c85b\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-zqdbk"
Oct 30 00:31:32.880641 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:32.880598    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xds6d\" (UniqueName: \"kubernetes.io/projected/139583a1-f622-49ab-8c1e-af168fcdf6c4-kube-api-access-xds6d\") pod \"kas-connection-checker-685895c7c9-f2b4d\" (UID: \"139583a1-f622-49ab-8c1e-af168fcdf6c4\") " pod="kube-system/kas-connection-checker-685895c7c9-f2b4d"
Oct 30 00:31:32.880641 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:32.880620    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbj4z\" (UniqueName: \"kubernetes.io/projected/9822c484-fd06-4e5e-94b4-ef0f5ade9a13-kube-api-access-xbj4z\") pod \"kas-connection-checker-685895c7c9-cqjlz\" (UID: \"9822c484-fd06-4e5e-94b4-ef0f5ade9a13\") " pod="kube-system/kas-connection-checker-685895c7c9-cqjlz"
Oct 30 00:31:32.880641 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:32.880638    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/251cf359-f9f5-44ae-8ba9-42bf0cabb1ab-metrics-tls\") pod \"dns-default-btj6v\" (UID: \"251cf359-f9f5-44ae-8ba9-42bf0cabb1ab\") " pod="openshift-dns/dns-default-btj6v"
Oct 30 00:31:32.880737 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:32.880654    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kpm2\" (UniqueName: \"kubernetes.io/projected/251cf359-f9f5-44ae-8ba9-42bf0cabb1ab-kube-api-access-6kpm2\") pod \"dns-default-btj6v\" (UID: \"251cf359-f9f5-44ae-8ba9-42bf0cabb1ab\") " pod="openshift-dns/dns-default-btj6v"
Oct 30 00:31:32.880771 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:32.880752    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9aba3099-fe4c-4846-9a34-25db2c2fe593-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-kjsj6\" (UID: \"9aba3099-fe4c-4846-9a34-25db2c2fe593\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-kjsj6"
Oct 30 00:31:32.880806 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:32.880783    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d7c5e976-596e-4cb5-81a9-b16262b3c85b-nginx-conf\") pod \"networking-console-plugin-86c478bb88-zqdbk\" (UID: \"d7c5e976-596e-4cb5-81a9-b16262b3c85b\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-zqdbk"
Oct 30 00:31:32.880806 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:32.880800    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jjgr\" (UniqueName: \"kubernetes.io/projected/9aba3099-fe4c-4846-9a34-25db2c2fe593-kube-api-access-2jjgr\") pod \"cluster-monitoring-operator-884576c57-kjsj6\" (UID: \"9aba3099-fe4c-4846-9a34-25db2c2fe593\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-kjsj6"
Oct 30 00:31:32.880903 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:32.880815    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/251cf359-f9f5-44ae-8ba9-42bf0cabb1ab-tmp-dir\") pod \"dns-default-btj6v\" (UID: \"251cf359-f9f5-44ae-8ba9-42bf0cabb1ab\") " pod="openshift-dns/dns-default-btj6v"
Oct 30 00:31:32.880903 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:32.880860    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3d234b3-76d4-47d5-98a4-dafd83477b17-metrics-certs\") pod \"router-default-85f7b79cf7-h2tpp\" (UID: \"a3d234b3-76d4-47d5-98a4-dafd83477b17\") " pod="openshift-ingress/router-default-85f7b79cf7-h2tpp"
Oct 30 00:31:32.884472 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:32.884448    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Oct 30 00:31:32.884472 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:32.884453    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Oct 30 00:31:32.884472 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:32.884469    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Oct 30 00:31:32.884665 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:32.884649    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-2nnk8"]
Oct 30 00:31:32.884722 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:32.884663    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Oct 30 00:31:32.884772 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:32.884749    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-bjzgs"
Oct 30 00:31:32.896223 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:32.896196    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Oct 30 00:31:32.896223 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:32.896199    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Oct 30 00:31:32.896223 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:32.896222    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Oct 30 00:31:32.896411 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:32.896236    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Oct 30 00:31:32.896411 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:32.896265    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Oct 30 00:31:32.896411 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:32.896312    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-fch7k\""
Oct 30 00:31:32.896613 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:32.896599    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 30 00:31:32.896648 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:32.896631    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Oct 30 00:31:32.899764 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:32.899747    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-g7c2w"]
Oct 30 00:31:32.900003 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:32.899874    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-2nnk8"
Oct 30 00:31:32.912176 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:32.912158    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6d4f74cbd-njpvm"]
Oct 30 00:31:32.912285 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:32.912272    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-g7c2w"
Oct 30 00:31:32.928015 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:32.927994    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-jg5h6"]
Oct 30 00:31:32.928127 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:32.928111    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6d4f74cbd-njpvm"
Oct 30 00:31:32.940810 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:32.940756    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6c69f88f85-88bkg"]
Oct 30 00:31:32.940901 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:32.940870    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-jg5h6"
Oct 30 00:31:32.948568 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:32.948546    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-fbdvl"]
Oct 30 00:31:32.948659 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:32.948576    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-wqf5x"]
Oct 30 00:31:32.948659 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:32.948644    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6c69f88f85-88bkg"
Oct 30 00:31:32.952903 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:32.952883    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-cqjlz"]
Oct 30 00:31:32.952992 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:32.952914    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-f2b4d"]
Oct 30 00:31:32.952992 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:32.952936    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-kjsj6"]
Oct 30 00:31:32.952992 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:32.952978    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-bjzgs"]
Oct 30 00:31:32.953122 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:32.952994    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-sxknf"]
Oct 30 00:31:32.953122 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:32.953008    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-sklth"]
Oct 30 00:31:32.953122 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:32.953023    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-85f7b79cf7-h2tpp"]
Oct 30 00:31:32.953122 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:32.953036    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-g7c2w"]
Oct 30 00:31:32.953122 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:32.953050    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-wqf5x"]
Oct 30 00:31:32.953122 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:32.953062    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-btj6v"]
Oct 30 00:31:32.953122 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:32.953078    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-zqdbk"]
Oct 30 00:31:32.953122 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:32.953094    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6d4f74cbd-njpvm"]
Oct 30 00:31:32.953122 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:32.953106    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6c69f88f85-88bkg"]
Oct 30 00:31:32.953122 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:32.953118    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-246g2"]
Oct 30 00:31:32.953621 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:32.953131    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-2nnk8"]
Oct 30 00:31:32.953621 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:32.953157    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-jg5h6"]
Oct 30 00:31:32.953621 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:32.953232    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-wqf5x"
Oct 30 00:31:32.971952 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:32.971881    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Oct 30 00:31:32.971952 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:32.971918    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 30 00:31:32.971952 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:32.971908    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Oct 30 00:31:32.971952 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:32.971908    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\""
Oct 30 00:31:32.972219 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:32.971996    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Oct 30 00:31:32.972219 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:32.972001    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Oct 30 00:31:32.972219 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:32.972034    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 30 00:31:32.972219 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:32.972041    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 00:31:32.972219 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:32.972090    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 00:31:32.972219 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:32.972124    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Oct 30 00:31:32.972219 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:32.972127    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Oct 30 00:31:32.972219 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:32.972157    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Oct 30 00:31:32.972526 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:32.972255    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Oct 30 00:31:32.981118 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:32.981101    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Oct 30 00:31:32.981257 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:32.981239    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e4ca0ea9-b57d-4d4c-be3a-2da5fc26a65d-serving-cert\") pod \"insights-operator-7c4986d449-sklth\" (UID: \"e4ca0ea9-b57d-4d4c-be3a-2da5fc26a65d\") " pod="openshift-insights/insights-operator-7c4986d449-sklth"
Oct 30 00:31:32.981257 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:32.981243    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Oct 30 00:31:32.981398 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:32.981308    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/61903cf2-2b92-4f40-ab52-25509de587ef-config\") pod \"service-ca-operator-544d749c8c-bjzgs\" (UID: \"61903cf2-2b92-4f40-ab52-25509de587ef\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-bjzgs"
Oct 30 00:31:32.981398 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:32.981348    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b874991e-9562-4848-b8f1-4b5a2b7cddc4-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-sxknf\" (UID: \"b874991e-9562-4848-b8f1-4b5a2b7cddc4\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-sxknf"
Oct 30 00:31:32.981458 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:32.981432    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvtmg\" (UniqueName: \"kubernetes.io/projected/b874991e-9562-4848-b8f1-4b5a2b7cddc4-kube-api-access-zvtmg\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-sxknf\" (UID: \"b874991e-9562-4848-b8f1-4b5a2b7cddc4\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-sxknf"
Oct 30 00:31:32.981559 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:32.981540    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kpm2\" (UniqueName: \"kubernetes.io/projected/251cf359-f9f5-44ae-8ba9-42bf0cabb1ab-kube-api-access-6kpm2\") pod \"dns-default-btj6v\" (UID: \"251cf359-f9f5-44ae-8ba9-42bf0cabb1ab\") " pod="openshift-dns/dns-default-btj6v"
Oct 30 00:31:32.981595 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:32.981576    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjrlm\" (UniqueName: \"kubernetes.io/projected/5dc56e7e-f3c1-43eb-a21b-8be6bb4706cf-kube-api-access-pjrlm\") pod \"console-operator-7cd56b8dbc-g7c2w\" (UID: \"5dc56e7e-f3c1-43eb-a21b-8be6bb4706cf\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-g7c2w"
Oct 30 00:31:32.981626 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:32.981616    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9aba3099-fe4c-4846-9a34-25db2c2fe593-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-kjsj6\" (UID: \"9aba3099-fe4c-4846-9a34-25db2c2fe593\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-kjsj6"
Oct 30 00:31:32.981664 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:32.981649    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5dc56e7e-f3c1-43eb-a21b-8be6bb4706cf-serving-cert\") pod \"console-operator-7cd56b8dbc-g7c2w\" (UID: \"5dc56e7e-f3c1-43eb-a21b-8be6bb4706cf\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-g7c2w"
Oct 30 00:31:32.981713 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:32.981680    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jjgr\" (UniqueName: \"kubernetes.io/projected/9aba3099-fe4c-4846-9a34-25db2c2fe593-kube-api-access-2jjgr\") pod \"cluster-monitoring-operator-884576c57-kjsj6\" (UID: \"9aba3099-fe4c-4846-9a34-25db2c2fe593\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-kjsj6"
Oct 30 00:31:32.981713 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:32.981699    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/251cf359-f9f5-44ae-8ba9-42bf0cabb1ab-tmp-dir\") pod \"dns-default-btj6v\" (UID: \"251cf359-f9f5-44ae-8ba9-42bf0cabb1ab\") " pod="openshift-dns/dns-default-btj6v"
Oct 30 00:31:32.981812 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:32.981722    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/a3d234b3-76d4-47d5-98a4-dafd83477b17-stats-auth\") pod \"router-default-85f7b79cf7-h2tpp\" (UID: \"a3d234b3-76d4-47d5-98a4-dafd83477b17\") " pod="openshift-ingress/router-default-85f7b79cf7-h2tpp"
Oct 30 00:31:32.981812 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:32.981738    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvgs7\" (UniqueName: \"kubernetes.io/projected/783c7a98-5ebf-44dd-8da9-797e71c28169-kube-api-access-vvgs7\") pod \"kas-connection-checker-685895c7c9-fbdvl\" (UID: \"783c7a98-5ebf-44dd-8da9-797e71c28169\") " pod="kube-system/kas-connection-checker-685895c7c9-fbdvl"
Oct 30 00:31:32.981812 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:32.981740    2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 00:31:32.981812 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:32.981754    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/a3d234b3-76d4-47d5-98a4-dafd83477b17-default-certificate\") pod \"router-default-85f7b79cf7-h2tpp\" (UID: \"a3d234b3-76d4-47d5-98a4-dafd83477b17\") " pod="openshift-ingress/router-default-85f7b79cf7-h2tpp"
Oct 30 00:31:32.981812 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:32.981808    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9aba3099-fe4c-4846-9a34-25db2c2fe593-cluster-monitoring-operator-tls podName:9aba3099-fe4c-4846-9a34-25db2c2fe593 nodeName:}" failed. No retries permitted until 2026-10-30 00:31:33.48178916 +0000 UTC m=+30.583096034 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/9aba3099-fe4c-4846-9a34-25db2c2fe593-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-kjsj6" (UID: "9aba3099-fe4c-4846-9a34-25db2c2fe593") : secret "cluster-monitoring-operator-tls" not found
Oct 30 00:31:32.982046 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:32.981948    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/e4ca0ea9-b57d-4d4c-be3a-2da5fc26a65d-snapshots\") pod \"insights-operator-7c4986d449-sklth\" (UID: \"e4ca0ea9-b57d-4d4c-be3a-2da5fc26a65d\") " pod="openshift-insights/insights-operator-7c4986d449-sklth"
Oct 30 00:31:32.982046 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:32.982019    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/251cf359-f9f5-44ae-8ba9-42bf0cabb1ab-config-volume\") pod \"dns-default-btj6v\" (UID: \"251cf359-f9f5-44ae-8ba9-42bf0cabb1ab\") " pod="openshift-dns/dns-default-btj6v"
Oct 30 00:31:32.982147 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:32.982050    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/9aba3099-fe4c-4846-9a34-25db2c2fe593-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-kjsj6\" (UID: \"9aba3099-fe4c-4846-9a34-25db2c2fe593\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-kjsj6"
Oct 30 00:31:32.982147 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:32.982077    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjmbq\" (UniqueName: \"kubernetes.io/projected/a3d234b3-76d4-47d5-98a4-dafd83477b17-kube-api-access-bjmbq\") pod \"router-default-85f7b79cf7-h2tpp\" (UID: \"a3d234b3-76d4-47d5-98a4-dafd83477b17\") " pod="openshift-ingress/router-default-85f7b79cf7-h2tpp"
Oct 30 00:31:32.982147 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:32.982105    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e4ca0ea9-b57d-4d4c-be3a-2da5fc26a65d-tmp\") pod \"insights-operator-7c4986d449-sklth\" (UID: \"e4ca0ea9-b57d-4d4c-be3a-2da5fc26a65d\") " pod="openshift-insights/insights-operator-7c4986d449-sklth"
Oct 30 00:31:32.982147 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:32.982132    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-566pb\" (UniqueName: \"kubernetes.io/projected/1881dcfe-c2da-4253-bca5-93f3526e14fa-kube-api-access-566pb\") pod \"cluster-samples-operator-6ddb6c8b94-2nnk8\" (UID: \"1881dcfe-c2da-4253-bca5-93f3526e14fa\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-2nnk8"
Oct 30 00:31:32.982330 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:32.982161    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1881dcfe-c2da-4253-bca5-93f3526e14fa-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-2nnk8\" (UID: \"1881dcfe-c2da-4253-bca5-93f3526e14fa\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-2nnk8"
Oct 30 00:31:32.982330 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:32.982187    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5dc56e7e-f3c1-43eb-a21b-8be6bb4706cf-config\") pod \"console-operator-7cd56b8dbc-g7c2w\" (UID: \"5dc56e7e-f3c1-43eb-a21b-8be6bb4706cf\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-g7c2w"
Oct 30 00:31:32.982330 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:32.982220    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xds6d\" (UniqueName: \"kubernetes.io/projected/139583a1-f622-49ab-8c1e-af168fcdf6c4-kube-api-access-xds6d\") pod \"kas-connection-checker-685895c7c9-f2b4d\" (UID: \"139583a1-f622-49ab-8c1e-af168fcdf6c4\") " pod="kube-system/kas-connection-checker-685895c7c9-f2b4d"
Oct 30 00:31:32.982330 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:32.982245    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbj4z\" (UniqueName: \"kubernetes.io/projected/9822c484-fd06-4e5e-94b4-ef0f5ade9a13-kube-api-access-xbj4z\") pod \"kas-connection-checker-685895c7c9-cqjlz\" (UID: \"9822c484-fd06-4e5e-94b4-ef0f5ade9a13\") " pod="kube-system/kas-connection-checker-685895c7c9-cqjlz"
Oct 30 00:31:32.982330 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:32.982273    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/251cf359-f9f5-44ae-8ba9-42bf0cabb1ab-metrics-tls\") pod \"dns-default-btj6v\" (UID: \"251cf359-f9f5-44ae-8ba9-42bf0cabb1ab\") " pod="openshift-dns/dns-default-btj6v"
Oct 30 00:31:32.982330 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:32.982313    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5dc56e7e-f3c1-43eb-a21b-8be6bb4706cf-trusted-ca\") pod \"console-operator-7cd56b8dbc-g7c2w\" (UID: \"5dc56e7e-f3c1-43eb-a21b-8be6bb4706cf\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-g7c2w"
Oct 30 00:31:32.982670 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:32.982340    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzg5q\" (UniqueName: \"kubernetes.io/projected/e4ca0ea9-b57d-4d4c-be3a-2da5fc26a65d-kube-api-access-qzg5q\") pod \"insights-operator-7c4986d449-sklth\" (UID: \"e4ca0ea9-b57d-4d4c-be3a-2da5fc26a65d\") " pod="openshift-insights/insights-operator-7c4986d449-sklth"
Oct 30 00:31:32.982670 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:32.982370    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ngcm\" (UniqueName: \"kubernetes.io/projected/cea1b649-9c31-4434-9448-7f6652496830-kube-api-access-2ngcm\") pod \"volume-data-source-validator-58c9f5f46d-246g2\" (UID: \"cea1b649-9c31-4434-9448-7f6652496830\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-246g2"
Oct 30 00:31:32.982670 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:32.982430    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d7c5e976-596e-4cb5-81a9-b16262b3c85b-nginx-conf\") pod \"networking-console-plugin-86c478bb88-zqdbk\" (UID: \"d7c5e976-596e-4cb5-81a9-b16262b3c85b\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-zqdbk"
Oct 30 00:31:32.982670 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:32.982455    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/1881dcfe-c2da-4253-bca5-93f3526e14fa-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-2nnk8\" (UID: \"1881dcfe-c2da-4253-bca5-93f3526e14fa\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-2nnk8"
Oct 30 00:31:32.982670 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:32.982484    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3d234b3-76d4-47d5-98a4-dafd83477b17-metrics-certs\") pod \"router-default-85f7b79cf7-h2tpp\" (UID: \"a3d234b3-76d4-47d5-98a4-dafd83477b17\") " pod="openshift-ingress/router-default-85f7b79cf7-h2tpp"
Oct 30 00:31:32.982670 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:32.982508    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e4ca0ea9-b57d-4d4c-be3a-2da5fc26a65d-service-ca-bundle\") pod \"insights-operator-7c4986d449-sklth\" (UID: \"e4ca0ea9-b57d-4d4c-be3a-2da5fc26a65d\") " pod="openshift-insights/insights-operator-7c4986d449-sklth"
Oct 30 00:31:32.982670 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:32.982531    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/61903cf2-2b92-4f40-ab52-25509de587ef-serving-cert\") pod \"service-ca-operator-544d749c8c-bjzgs\" (UID: \"61903cf2-2b92-4f40-ab52-25509de587ef\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-bjzgs"
Oct 30 00:31:32.982670 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:32.982563    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b874991e-9562-4848-b8f1-4b5a2b7cddc4-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-sxknf\" (UID: \"b874991e-9562-4848-b8f1-4b5a2b7cddc4\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-sxknf"
Oct 30 00:31:32.982670 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:32.982584    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/251cf359-f9f5-44ae-8ba9-42bf0cabb1ab-config-volume\") pod \"dns-default-btj6v\" (UID: \"251cf359-f9f5-44ae-8ba9-42bf0cabb1ab\") " pod="openshift-dns/dns-default-btj6v"
Oct 30 00:31:32.982670 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:32.982589    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a3d234b3-76d4-47d5-98a4-dafd83477b17-service-ca-bundle\") pod \"router-default-85f7b79cf7-h2tpp\" (UID: \"a3d234b3-76d4-47d5-98a4-dafd83477b17\") " pod="openshift-ingress/router-default-85f7b79cf7-h2tpp"
Oct 30 00:31:32.982670 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:32.982642    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d7c5e976-596e-4cb5-81a9-b16262b3c85b-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-zqdbk\" (UID: \"d7c5e976-596e-4cb5-81a9-b16262b3c85b\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-zqdbk"
Oct 30 00:31:32.982670 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:32.982670    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgb9w\" (UniqueName: \"kubernetes.io/projected/61903cf2-2b92-4f40-ab52-25509de587ef-kube-api-access-qgb9w\") pod \"service-ca-operator-544d749c8c-bjzgs\" (UID: \"61903cf2-2b92-4f40-ab52-25509de587ef\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-bjzgs"
Oct 30 00:31:32.983235 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:32.982678    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a3d234b3-76d4-47d5-98a4-dafd83477b17-service-ca-bundle podName:a3d234b3-76d4-47d5-98a4-dafd83477b17 nodeName:}" failed. No retries permitted until 2026-10-30 00:31:33.48266417 +0000 UTC m=+30.583971044 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/a3d234b3-76d4-47d5-98a4-dafd83477b17-service-ca-bundle") pod "router-default-85f7b79cf7-h2tpp" (UID: "a3d234b3-76d4-47d5-98a4-dafd83477b17") : configmap references non-existent config key: service-ca.crt
Oct 30 00:31:32.983235 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:32.982789    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e4ca0ea9-b57d-4d4c-be3a-2da5fc26a65d-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-sklth\" (UID: \"e4ca0ea9-b57d-4d4c-be3a-2da5fc26a65d\") " pod="openshift-insights/insights-operator-7c4986d449-sklth"
Oct 30 00:31:32.983235 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:32.982890    2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 00:31:32.983235 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:32.982933    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7c5e976-596e-4cb5-81a9-b16262b3c85b-networking-console-plugin-cert podName:d7c5e976-596e-4cb5-81a9-b16262b3c85b nodeName:}" failed. No retries permitted until 2026-10-30 00:31:33.482918643 +0000 UTC m=+30.584225518 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/d7c5e976-596e-4cb5-81a9-b16262b3c85b-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-zqdbk" (UID: "d7c5e976-596e-4cb5-81a9-b16262b3c85b") : secret "networking-console-plugin-cert" not found
Oct 30 00:31:32.983235 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:32.983070    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/9aba3099-fe4c-4846-9a34-25db2c2fe593-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-kjsj6\" (UID: \"9aba3099-fe4c-4846-9a34-25db2c2fe593\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-kjsj6"
Oct 30 00:31:32.983511 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:32.983346    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d7c5e976-596e-4cb5-81a9-b16262b3c85b-nginx-conf\") pod \"networking-console-plugin-86c478bb88-zqdbk\" (UID: \"d7c5e976-596e-4cb5-81a9-b16262b3c85b\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-zqdbk"
Oct 30 00:31:32.983565 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:32.983524    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 00:31:32.983614 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:32.983603    2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 00:31:32.983668 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:32.983647    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a3d234b3-76d4-47d5-98a4-dafd83477b17-metrics-certs podName:a3d234b3-76d4-47d5-98a4-dafd83477b17 nodeName:}" failed. No retries permitted until 2026-10-30 00:31:33.483631463 +0000 UTC m=+30.584938341 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a3d234b3-76d4-47d5-98a4-dafd83477b17-metrics-certs") pod "router-default-85f7b79cf7-h2tpp" (UID: "a3d234b3-76d4-47d5-98a4-dafd83477b17") : secret "router-metrics-certs-default" not found
Oct 30 00:31:32.983727 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:32.983700    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/251cf359-f9f5-44ae-8ba9-42bf0cabb1ab-metrics-tls podName:251cf359-f9f5-44ae-8ba9-42bf0cabb1ab nodeName:}" failed. No retries permitted until 2026-10-30 00:31:33.483689306 +0000 UTC m=+30.584996190 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/251cf359-f9f5-44ae-8ba9-42bf0cabb1ab-metrics-tls") pod "dns-default-btj6v" (UID: "251cf359-f9f5-44ae-8ba9-42bf0cabb1ab") : secret "dns-default-metrics-tls" not found
Oct 30 00:31:32.984045 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:32.984024    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/251cf359-f9f5-44ae-8ba9-42bf0cabb1ab-tmp-dir\") pod \"dns-default-btj6v\" (UID: \"251cf359-f9f5-44ae-8ba9-42bf0cabb1ab\") " pod="openshift-dns/dns-default-btj6v"
Oct 30 00:31:32.985507 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:32.985489    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/a3d234b3-76d4-47d5-98a4-dafd83477b17-default-certificate\") pod \"router-default-85f7b79cf7-h2tpp\" (UID: \"a3d234b3-76d4-47d5-98a4-dafd83477b17\") " pod="openshift-ingress/router-default-85f7b79cf7-h2tpp"
Oct 30 00:31:32.985588 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:32.985490    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/a3d234b3-76d4-47d5-98a4-dafd83477b17-stats-auth\") pod \"router-default-85f7b79cf7-h2tpp\" (UID: \"a3d234b3-76d4-47d5-98a4-dafd83477b17\") " pod="openshift-ingress/router-default-85f7b79cf7-h2tpp"
Oct 30 00:31:32.996488 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:32.996466    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Oct 30 00:31:32.996577 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:32.996560    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 30 00:31:32.996577 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:32.996571    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Oct 30 00:31:32.996695 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:32.996679    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Oct 30 00:31:32.996989 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:32.996971    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-nrfgz\""
Oct 30 00:31:32.997037 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:32.997024    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Oct 30 00:31:32.997094 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:32.997080    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Oct 30 00:31:32.997131 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:32.997083    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-j8zc4\""
Oct 30 00:31:32.997482 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:32.997467    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Oct 30 00:31:33.005211 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:33.005191    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jjgr\" (UniqueName: \"kubernetes.io/projected/9aba3099-fe4c-4846-9a34-25db2c2fe593-kube-api-access-2jjgr\") pod \"cluster-monitoring-operator-884576c57-kjsj6\" (UID: \"9aba3099-fe4c-4846-9a34-25db2c2fe593\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-kjsj6"
Oct 30 00:31:33.026632 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:33.026610    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvgs7\" (UniqueName: \"kubernetes.io/projected/783c7a98-5ebf-44dd-8da9-797e71c28169-kube-api-access-vvgs7\") pod \"kas-connection-checker-685895c7c9-fbdvl\" (UID: \"783c7a98-5ebf-44dd-8da9-797e71c28169\") " pod="kube-system/kas-connection-checker-685895c7c9-fbdvl"
Oct 30 00:31:33.026825 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:33.026809    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kpm2\" (UniqueName: \"kubernetes.io/projected/251cf359-f9f5-44ae-8ba9-42bf0cabb1ab-kube-api-access-6kpm2\") pod \"dns-default-btj6v\" (UID: \"251cf359-f9f5-44ae-8ba9-42bf0cabb1ab\") " pod="openshift-dns/dns-default-btj6v"
Oct 30 00:31:33.047323 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:33.047298    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjmbq\" (UniqueName: \"kubernetes.io/projected/a3d234b3-76d4-47d5-98a4-dafd83477b17-kube-api-access-bjmbq\") pod \"router-default-85f7b79cf7-h2tpp\" (UID: \"a3d234b3-76d4-47d5-98a4-dafd83477b17\") " pod="openshift-ingress/router-default-85f7b79cf7-h2tpp"
Oct 30 00:31:33.048046 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:33.048028    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xds6d\" (UniqueName: \"kubernetes.io/projected/139583a1-f622-49ab-8c1e-af168fcdf6c4-kube-api-access-xds6d\") pod \"kas-connection-checker-685895c7c9-f2b4d\" (UID: \"139583a1-f622-49ab-8c1e-af168fcdf6c4\") " pod="kube-system/kas-connection-checker-685895c7c9-f2b4d"
Oct 30 00:31:33.053120 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:33.053101    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbj4z\" (UniqueName: \"kubernetes.io/projected/9822c484-fd06-4e5e-94b4-ef0f5ade9a13-kube-api-access-xbj4z\") pod \"kas-connection-checker-685895c7c9-cqjlz\" (UID: \"9822c484-fd06-4e5e-94b4-ef0f5ade9a13\") " pod="kube-system/kas-connection-checker-685895c7c9-cqjlz"
Oct 30 00:31:33.067697 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:33.067678    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-fbdvl"
Oct 30 00:31:33.075682 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:33.075655    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-cqjlz"
Oct 30 00:31:33.083745 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:33.083727    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ngcm\" (UniqueName: \"kubernetes.io/projected/cea1b649-9c31-4434-9448-7f6652496830-kube-api-access-2ngcm\") pod \"volume-data-source-validator-58c9f5f46d-246g2\" (UID: \"cea1b649-9c31-4434-9448-7f6652496830\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-246g2"
Oct 30 00:31:33.083828 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:33.083763    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/1881dcfe-c2da-4253-bca5-93f3526e14fa-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-2nnk8\" (UID: \"1881dcfe-c2da-4253-bca5-93f3526e14fa\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-2nnk8"
Oct 30 00:31:33.083828 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:33.083788    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e4ca0ea9-b57d-4d4c-be3a-2da5fc26a65d-service-ca-bundle\") pod \"insights-operator-7c4986d449-sklth\" (UID: \"e4ca0ea9-b57d-4d4c-be3a-2da5fc26a65d\") " pod="openshift-insights/insights-operator-7c4986d449-sklth"
Oct 30 00:31:33.083828 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:33.083804    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/61903cf2-2b92-4f40-ab52-25509de587ef-serving-cert\") pod \"service-ca-operator-544d749c8c-bjzgs\" (UID: \"61903cf2-2b92-4f40-ab52-25509de587ef\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-bjzgs"
Oct 30 00:31:33.083828 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:33.083824    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed-registry-tls\") pod \"image-registry-6d4f74cbd-njpvm\" (UID: \"2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed\") " pod="openshift-image-registry/image-registry-6d4f74cbd-njpvm"
Oct 30 00:31:33.083997 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:33.083941    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b874991e-9562-4848-b8f1-4b5a2b7cddc4-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-sxknf\" (UID: \"b874991e-9562-4848-b8f1-4b5a2b7cddc4\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-sxknf"
Oct 30 00:31:33.084056 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:33.084009    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgb9w\" (UniqueName: \"kubernetes.io/projected/61903cf2-2b92-4f40-ab52-25509de587ef-kube-api-access-qgb9w\") pod \"service-ca-operator-544d749c8c-bjzgs\" (UID: \"61903cf2-2b92-4f40-ab52-25509de587ef\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-bjzgs"
Oct 30 00:31:33.084056 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:33.084042    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e4ca0ea9-b57d-4d4c-be3a-2da5fc26a65d-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-sklth\" (UID: \"e4ca0ea9-b57d-4d4c-be3a-2da5fc26a65d\") " pod="openshift-insights/insights-operator-7c4986d449-sklth"
Oct 30 00:31:33.085942 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:33.084072    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5ed36f74-5c9e-408e-a7fb-2364144f054e-bound-sa-token\") pod \"image-registry-6c69f88f85-88bkg\" (UID: \"5ed36f74-5c9e-408e-a7fb-2364144f054e\") " pod="openshift-image-registry/image-registry-6c69f88f85-88bkg"
Oct 30 00:31:33.085942 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:33.084103    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e4ca0ea9-b57d-4d4c-be3a-2da5fc26a65d-serving-cert\") pod \"insights-operator-7c4986d449-sklth\" (UID: \"e4ca0ea9-b57d-4d4c-be3a-2da5fc26a65d\") " pod="openshift-insights/insights-operator-7c4986d449-sklth"
Oct 30 00:31:33.085942 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:33.084127    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/61903cf2-2b92-4f40-ab52-25509de587ef-config\") pod \"service-ca-operator-544d749c8c-bjzgs\" (UID: \"61903cf2-2b92-4f40-ab52-25509de587ef\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-bjzgs"
Oct 30 00:31:33.085942 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:33.084150    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5ed36f74-5c9e-408e-a7fb-2364144f054e-installation-pull-secrets\") pod \"image-registry-6c69f88f85-88bkg\" (UID: \"5ed36f74-5c9e-408e-a7fb-2364144f054e\") " pod="openshift-image-registry/image-registry-6c69f88f85-88bkg"
Oct 30 00:31:33.085942 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:33.084178    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b874991e-9562-4848-b8f1-4b5a2b7cddc4-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-sxknf\" (UID: \"b874991e-9562-4848-b8f1-4b5a2b7cddc4\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-sxknf"
Oct 30 00:31:33.085942 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:33.084205    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvtmg\" (UniqueName: \"kubernetes.io/projected/b874991e-9562-4848-b8f1-4b5a2b7cddc4-kube-api-access-zvtmg\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-sxknf\" (UID: \"b874991e-9562-4848-b8f1-4b5a2b7cddc4\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-sxknf"
Oct 30 00:31:33.085942 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:33.084242    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjrlm\" (UniqueName: \"kubernetes.io/projected/5dc56e7e-f3c1-43eb-a21b-8be6bb4706cf-kube-api-access-pjrlm\") pod \"console-operator-7cd56b8dbc-g7c2w\" (UID: \"5dc56e7e-f3c1-43eb-a21b-8be6bb4706cf\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-g7c2w"
Oct 30 00:31:33.085942 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:33.084269    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhs2g\" (UniqueName: \"kubernetes.io/projected/9b2b3b14-8027-489a-88f0-d056226c2f67-kube-api-access-bhs2g\") pod \"network-check-source-6f78d59f99-jg5h6\" (UID: \"9b2b3b14-8027-489a-88f0-d056226c2f67\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-jg5h6"
Oct 30 00:31:33.085942 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:33.084317    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5dc56e7e-f3c1-43eb-a21b-8be6bb4706cf-serving-cert\") pod \"console-operator-7cd56b8dbc-g7c2w\" (UID: \"5dc56e7e-f3c1-43eb-a21b-8be6bb4706cf\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-g7c2w"
Oct 30 00:31:33.085942 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:33.084346    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed-registry-certificates\") pod \"image-registry-6d4f74cbd-njpvm\" (UID: \"2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed\") " pod="openshift-image-registry/image-registry-6d4f74cbd-njpvm"
Oct 30 00:31:33.085942 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:33.084365    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e4ca0ea9-b57d-4d4c-be3a-2da5fc26a65d-service-ca-bundle\") pod \"insights-operator-7c4986d449-sklth\" (UID: \"e4ca0ea9-b57d-4d4c-be3a-2da5fc26a65d\") " pod="openshift-insights/insights-operator-7c4986d449-sklth"
Oct 30 00:31:33.085942 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:33.084391    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84rlf\" (UniqueName: \"kubernetes.io/projected/2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed-kube-api-access-84rlf\") pod \"image-registry-6d4f74cbd-njpvm\" (UID: \"2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed\") " pod="openshift-image-registry/image-registry-6d4f74cbd-njpvm"
Oct 30 00:31:33.085942 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:33.084557    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b874991e-9562-4848-b8f1-4b5a2b7cddc4-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-sxknf\" (UID: \"b874991e-9562-4848-b8f1-4b5a2b7cddc4\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-sxknf"
Oct 30 00:31:33.085942 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:33.084656    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/559377b2-e3fe-4889-995e-55127771d05e-cert\") pod \"ingress-canary-wqf5x\" (UID: \"559377b2-e3fe-4889-995e-55127771d05e\") " pod="openshift-ingress-canary/ingress-canary-wqf5x"
Oct 30 00:31:33.085942 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:33.084665    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/1881dcfe-c2da-4253-bca5-93f3526e14fa-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-2nnk8\" (UID: \"1881dcfe-c2da-4253-bca5-93f3526e14fa\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-2nnk8"
Oct 30 00:31:33.086790 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:33.084692    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/e4ca0ea9-b57d-4d4c-be3a-2da5fc26a65d-snapshots\") pod \"insights-operator-7c4986d449-sklth\" (UID: \"e4ca0ea9-b57d-4d4c-be3a-2da5fc26a65d\") " pod="openshift-insights/insights-operator-7c4986d449-sklth"
Oct 30 00:31:33.086790 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:33.084850    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e4ca0ea9-b57d-4d4c-be3a-2da5fc26a65d-tmp\") pod \"insights-operator-7c4986d449-sklth\" (UID: \"e4ca0ea9-b57d-4d4c-be3a-2da5fc26a65d\") " pod="openshift-insights/insights-operator-7c4986d449-sklth"
Oct 30 00:31:33.086790 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:33.084892    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-566pb\" (UniqueName: \"kubernetes.io/projected/1881dcfe-c2da-4253-bca5-93f3526e14fa-kube-api-access-566pb\") pod \"cluster-samples-operator-6ddb6c8b94-2nnk8\" (UID: \"1881dcfe-c2da-4253-bca5-93f3526e14fa\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-2nnk8"
Oct 30 00:31:33.086790 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:33.084922    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed-trusted-ca\") pod \"image-registry-6d4f74cbd-njpvm\" (UID: \"2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed\") " pod="openshift-image-registry/image-registry-6d4f74cbd-njpvm"
Oct 30 00:31:33.086790 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:33.084965    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1881dcfe-c2da-4253-bca5-93f3526e14fa-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-2nnk8\" (UID: \"1881dcfe-c2da-4253-bca5-93f3526e14fa\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-2nnk8"
Oct 30 00:31:33.086790 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:33.084991    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5dc56e7e-f3c1-43eb-a21b-8be6bb4706cf-config\") pod \"console-operator-7cd56b8dbc-g7c2w\" (UID: \"5dc56e7e-f3c1-43eb-a21b-8be6bb4706cf\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-g7c2w"
Oct 30 00:31:33.086790 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:33.085019    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed-installation-pull-secrets\") pod \"image-registry-6d4f74cbd-njpvm\" (UID: \"2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed\") " pod="openshift-image-registry/image-registry-6d4f74cbd-njpvm"
Oct 30 00:31:33.086790 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:33.085047    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed-bound-sa-token\") pod \"image-registry-6d4f74cbd-njpvm\" (UID: \"2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed\") " pod="openshift-image-registry/image-registry-6d4f74cbd-njpvm"
Oct 30 00:31:33.086790 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:33.085118    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5ed36f74-5c9e-408e-a7fb-2364144f054e-ca-trust-extracted\") pod \"image-registry-6c69f88f85-88bkg\" (UID: \"5ed36f74-5c9e-408e-a7fb-2364144f054e\") " pod="openshift-image-registry/image-registry-6c69f88f85-88bkg"
Oct 30 00:31:33.086790 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:33.085149    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5ed36f74-5c9e-408e-a7fb-2364144f054e-registry-certificates\") pod \"image-registry-6c69f88f85-88bkg\" (UID: \"5ed36f74-5c9e-408e-a7fb-2364144f054e\") " pod="openshift-image-registry/image-registry-6c69f88f85-88bkg"
Oct 30 00:31:33.086790 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:33.085240    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/e4ca0ea9-b57d-4d4c-be3a-2da5fc26a65d-snapshots\") pod \"insights-operator-7c4986d449-sklth\" (UID: \"e4ca0ea9-b57d-4d4c-be3a-2da5fc26a65d\") " pod="openshift-insights/insights-operator-7c4986d449-sklth"
Oct 30 00:31:33.086790 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:33.085250    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e4ca0ea9-b57d-4d4c-be3a-2da5fc26a65d-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-sklth\" (UID: \"e4ca0ea9-b57d-4d4c-be3a-2da5fc26a65d\") " pod="openshift-insights/insights-operator-7c4986d449-sklth"
Oct 30 00:31:33.086790 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:33.085404    2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 00:31:33.086790 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:33.085410    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed-image-registry-private-configuration\") pod \"image-registry-6d4f74cbd-njpvm\" (UID: \"2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed\") " pod="openshift-image-registry/image-registry-6d4f74cbd-njpvm"
Oct 30 00:31:33.086790 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:33.085413    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e4ca0ea9-b57d-4d4c-be3a-2da5fc26a65d-tmp\") pod \"insights-operator-7c4986d449-sklth\" (UID: \"e4ca0ea9-b57d-4d4c-be3a-2da5fc26a65d\") " pod="openshift-insights/insights-operator-7c4986d449-sklth"
Oct 30 00:31:33.086790 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:33.085521    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5ed36f74-5c9e-408e-a7fb-2364144f054e-trusted-ca\") pod \"image-registry-6c69f88f85-88bkg\" (UID: \"5ed36f74-5c9e-408e-a7fb-2364144f054e\") " pod="openshift-image-registry/image-registry-6c69f88f85-88bkg"
Oct 30 00:31:33.087216 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:33.085542    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1881dcfe-c2da-4253-bca5-93f3526e14fa-samples-operator-tls podName:1881dcfe-c2da-4253-bca5-93f3526e14fa nodeName:}" failed. No retries permitted until 2026-10-30 00:31:33.58552251 +0000 UTC m=+30.686829383 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/1881dcfe-c2da-4253-bca5-93f3526e14fa-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-2nnk8" (UID: "1881dcfe-c2da-4253-bca5-93f3526e14fa") : secret "samples-operator-tls" not found
Oct 30 00:31:33.087216 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:33.085617    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmzz7\" (UniqueName: \"kubernetes.io/projected/559377b2-e3fe-4889-995e-55127771d05e-kube-api-access-zmzz7\") pod \"ingress-canary-wqf5x\" (UID: \"559377b2-e3fe-4889-995e-55127771d05e\") " pod="openshift-ingress-canary/ingress-canary-wqf5x"
Oct 30 00:31:33.087216 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:33.085651    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5ed36f74-5c9e-408e-a7fb-2364144f054e-registry-tls\") pod \"image-registry-6c69f88f85-88bkg\" (UID: \"5ed36f74-5c9e-408e-a7fb-2364144f054e\") " pod="openshift-image-registry/image-registry-6c69f88f85-88bkg"
Oct 30 00:31:33.087216 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:33.085715    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed-ca-trust-extracted\") pod \"image-registry-6d4f74cbd-njpvm\" (UID: \"2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed\") " pod="openshift-image-registry/image-registry-6d4f74cbd-njpvm"
Oct 30 00:31:33.087216 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:33.085756    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7l585\" (UniqueName: \"kubernetes.io/projected/5ed36f74-5c9e-408e-a7fb-2364144f054e-kube-api-access-7l585\") pod \"image-registry-6c69f88f85-88bkg\" (UID: \"5ed36f74-5c9e-408e-a7fb-2364144f054e\") " pod="openshift-image-registry/image-registry-6c69f88f85-88bkg"
Oct 30 00:31:33.087216 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:33.085851    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5dc56e7e-f3c1-43eb-a21b-8be6bb4706cf-trusted-ca\") pod \"console-operator-7cd56b8dbc-g7c2w\" (UID: \"5dc56e7e-f3c1-43eb-a21b-8be6bb4706cf\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-g7c2w"
Oct 30 00:31:33.087216 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:33.085905    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5ed36f74-5c9e-408e-a7fb-2364144f054e-image-registry-private-configuration\") pod \"image-registry-6c69f88f85-88bkg\" (UID: \"5ed36f74-5c9e-408e-a7fb-2364144f054e\") " pod="openshift-image-registry/image-registry-6c69f88f85-88bkg"
Oct 30 00:31:33.087216 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:33.085919    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5dc56e7e-f3c1-43eb-a21b-8be6bb4706cf-config\") pod \"console-operator-7cd56b8dbc-g7c2w\" (UID: \"5dc56e7e-f3c1-43eb-a21b-8be6bb4706cf\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-g7c2w"
Oct 30 00:31:33.087216 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:33.085930    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/61903cf2-2b92-4f40-ab52-25509de587ef-config\") pod \"service-ca-operator-544d749c8c-bjzgs\" (UID: \"61903cf2-2b92-4f40-ab52-25509de587ef\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-bjzgs"
Oct 30 00:31:33.087216 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:33.085942    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzg5q\" (UniqueName: \"kubernetes.io/projected/e4ca0ea9-b57d-4d4c-be3a-2da5fc26a65d-kube-api-access-qzg5q\") pod \"insights-operator-7c4986d449-sklth\" (UID: \"e4ca0ea9-b57d-4d4c-be3a-2da5fc26a65d\") " pod="openshift-insights/insights-operator-7c4986d449-sklth"
Oct 30 00:31:33.087216 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:33.086202    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/61903cf2-2b92-4f40-ab52-25509de587ef-serving-cert\") pod \"service-ca-operator-544d749c8c-bjzgs\" (UID: \"61903cf2-2b92-4f40-ab52-25509de587ef\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-bjzgs"
Oct 30 00:31:33.087216 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:33.086907    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e4ca0ea9-b57d-4d4c-be3a-2da5fc26a65d-serving-cert\") pod \"insights-operator-7c4986d449-sklth\" (UID: \"e4ca0ea9-b57d-4d4c-be3a-2da5fc26a65d\") " pod="openshift-insights/insights-operator-7c4986d449-sklth"
Oct 30 00:31:33.087216 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:33.086928    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b874991e-9562-4848-b8f1-4b5a2b7cddc4-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-sxknf\" (UID: \"b874991e-9562-4848-b8f1-4b5a2b7cddc4\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-sxknf"
Oct 30 00:31:33.087216 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:33.086978    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-f2b4d"
Oct 30 00:31:33.088087 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:33.088070    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5dc56e7e-f3c1-43eb-a21b-8be6bb4706cf-serving-cert\") pod \"console-operator-7cd56b8dbc-g7c2w\" (UID: \"5dc56e7e-f3c1-43eb-a21b-8be6bb4706cf\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-g7c2w"
Oct 30 00:31:33.088087 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:33.088083    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5dc56e7e-f3c1-43eb-a21b-8be6bb4706cf-trusted-ca\") pod \"console-operator-7cd56b8dbc-g7c2w\" (UID: \"5dc56e7e-f3c1-43eb-a21b-8be6bb4706cf\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-g7c2w"
Oct 30 00:31:33.106865 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:33.106838    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvtmg\" (UniqueName: \"kubernetes.io/projected/b874991e-9562-4848-b8f1-4b5a2b7cddc4-kube-api-access-zvtmg\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-sxknf\" (UID: \"b874991e-9562-4848-b8f1-4b5a2b7cddc4\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-sxknf"
Oct 30 00:31:33.107450 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:33.107424    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgb9w\" (UniqueName: \"kubernetes.io/projected/61903cf2-2b92-4f40-ab52-25509de587ef-kube-api-access-qgb9w\") pod \"service-ca-operator-544d749c8c-bjzgs\" (UID: \"61903cf2-2b92-4f40-ab52-25509de587ef\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-bjzgs"
Oct 30 00:31:33.107547 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:33.107457    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ngcm\" (UniqueName: \"kubernetes.io/projected/cea1b649-9c31-4434-9448-7f6652496830-kube-api-access-2ngcm\") pod \"volume-data-source-validator-58c9f5f46d-246g2\" (UID: \"cea1b649-9c31-4434-9448-7f6652496830\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-246g2"
Oct 30 00:31:33.107547 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:33.107464    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjrlm\" (UniqueName: \"kubernetes.io/projected/5dc56e7e-f3c1-43eb-a21b-8be6bb4706cf-kube-api-access-pjrlm\") pod \"console-operator-7cd56b8dbc-g7c2w\" (UID: \"5dc56e7e-f3c1-43eb-a21b-8be6bb4706cf\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-g7c2w"
Oct 30 00:31:33.107547 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:33.107523    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-566pb\" (UniqueName: \"kubernetes.io/projected/1881dcfe-c2da-4253-bca5-93f3526e14fa-kube-api-access-566pb\") pod \"cluster-samples-operator-6ddb6c8b94-2nnk8\" (UID: \"1881dcfe-c2da-4253-bca5-93f3526e14fa\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-2nnk8"
Oct 30 00:31:33.108849 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:33.108826    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzg5q\" (UniqueName: \"kubernetes.io/projected/e4ca0ea9-b57d-4d4c-be3a-2da5fc26a65d-kube-api-access-qzg5q\") pod \"insights-operator-7c4986d449-sklth\" (UID: \"e4ca0ea9-b57d-4d4c-be3a-2da5fc26a65d\") " pod="openshift-insights/insights-operator-7c4986d449-sklth"
Oct 30 00:31:33.177767 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:33.177735    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-246g2"
Oct 30 00:31:33.186442 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:33.186411    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5ed36f74-5c9e-408e-a7fb-2364144f054e-ca-trust-extracted\") pod \"image-registry-6c69f88f85-88bkg\" (UID: \"5ed36f74-5c9e-408e-a7fb-2364144f054e\") " pod="openshift-image-registry/image-registry-6c69f88f85-88bkg"
Oct 30 00:31:33.186533 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:33.186459    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5ed36f74-5c9e-408e-a7fb-2364144f054e-registry-certificates\") pod \"image-registry-6c69f88f85-88bkg\" (UID: \"5ed36f74-5c9e-408e-a7fb-2364144f054e\") " pod="openshift-image-registry/image-registry-6c69f88f85-88bkg"
Oct 30 00:31:33.186533 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:33.186489    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed-image-registry-private-configuration\") pod \"image-registry-6d4f74cbd-njpvm\" (UID: \"2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed\") " pod="openshift-image-registry/image-registry-6d4f74cbd-njpvm"
Oct 30 00:31:33.186533 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:33.186514    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5ed36f74-5c9e-408e-a7fb-2364144f054e-trusted-ca\") pod \"image-registry-6c69f88f85-88bkg\" (UID: \"5ed36f74-5c9e-408e-a7fb-2364144f054e\") " pod="openshift-image-registry/image-registry-6c69f88f85-88bkg"
Oct 30 00:31:33.186642 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:33.186542    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmzz7\" (UniqueName: \"kubernetes.io/projected/559377b2-e3fe-4889-995e-55127771d05e-kube-api-access-zmzz7\") pod \"ingress-canary-wqf5x\" (UID: \"559377b2-e3fe-4889-995e-55127771d05e\") " pod="openshift-ingress-canary/ingress-canary-wqf5x"
Oct 30 00:31:33.186642 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:33.186566    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5ed36f74-5c9e-408e-a7fb-2364144f054e-registry-tls\") pod \"image-registry-6c69f88f85-88bkg\" (UID: \"5ed36f74-5c9e-408e-a7fb-2364144f054e\") " pod="openshift-image-registry/image-registry-6c69f88f85-88bkg"
Oct 30 00:31:33.186642 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:33.186605    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed-ca-trust-extracted\") pod \"image-registry-6d4f74cbd-njpvm\" (UID: \"2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed\") " pod="openshift-image-registry/image-registry-6d4f74cbd-njpvm"
Oct 30 00:31:33.186642 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:33.186629    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7l585\" (UniqueName: \"kubernetes.io/projected/5ed36f74-5c9e-408e-a7fb-2364144f054e-kube-api-access-7l585\") pod \"image-registry-6c69f88f85-88bkg\" (UID: \"5ed36f74-5c9e-408e-a7fb-2364144f054e\") " pod="openshift-image-registry/image-registry-6c69f88f85-88bkg"
Oct 30 00:31:33.187086 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:33.187064    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5ed36f74-5c9e-408e-a7fb-2364144f054e-image-registry-private-configuration\") pod \"image-registry-6c69f88f85-88bkg\" (UID: \"5ed36f74-5c9e-408e-a7fb-2364144f054e\") " pod="openshift-image-registry/image-registry-6c69f88f85-88bkg"
Oct 30 00:31:33.187149 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:33.187116    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 00:31:33.187149 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:33.187137    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6c69f88f85-88bkg: secret "image-registry-tls" not found
Oct 30 00:31:33.187234 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:33.187154    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed-registry-tls\") pod \"image-registry-6d4f74cbd-njpvm\" (UID: \"2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed\") " pod="openshift-image-registry/image-registry-6d4f74cbd-njpvm"
Oct 30 00:31:33.187234 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:33.187202    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5ed36f74-5c9e-408e-a7fb-2364144f054e-registry-tls podName:5ed36f74-5c9e-408e-a7fb-2364144f054e nodeName:}" failed. No retries permitted until 2026-10-30 00:31:33.687183175 +0000 UTC m=+30.788490041 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5ed36f74-5c9e-408e-a7fb-2364144f054e-registry-tls") pod "image-registry-6c69f88f85-88bkg" (UID: "5ed36f74-5c9e-408e-a7fb-2364144f054e") : secret "image-registry-tls" not found
Oct 30 00:31:33.187343 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:33.187244    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5ed36f74-5c9e-408e-a7fb-2364144f054e-bound-sa-token\") pod \"image-registry-6c69f88f85-88bkg\" (UID: \"5ed36f74-5c9e-408e-a7fb-2364144f054e\") " pod="openshift-image-registry/image-registry-6c69f88f85-88bkg"
Oct 30 00:31:33.187343 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:33.187277    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5ed36f74-5c9e-408e-a7fb-2364144f054e-installation-pull-secrets\") pod \"image-registry-6c69f88f85-88bkg\" (UID: \"5ed36f74-5c9e-408e-a7fb-2364144f054e\") " pod="openshift-image-registry/image-registry-6c69f88f85-88bkg"
Oct 30 00:31:33.187343 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:33.187318    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhs2g\" (UniqueName: \"kubernetes.io/projected/9b2b3b14-8027-489a-88f0-d056226c2f67-kube-api-access-bhs2g\") pod \"network-check-source-6f78d59f99-jg5h6\" (UID: \"9b2b3b14-8027-489a-88f0-d056226c2f67\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-jg5h6"
Oct 30 00:31:33.187511 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:33.187376    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed-registry-certificates\") pod \"image-registry-6d4f74cbd-njpvm\" (UID: \"2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed\") " pod="openshift-image-registry/image-registry-6d4f74cbd-njpvm"
Oct 30 00:31:33.187511 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:33.187422    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84rlf\" (UniqueName: \"kubernetes.io/projected/2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed-kube-api-access-84rlf\") pod \"image-registry-6d4f74cbd-njpvm\" (UID: \"2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed\") " pod="openshift-image-registry/image-registry-6d4f74cbd-njpvm"
Oct 30 00:31:33.187511 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:33.187454    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/559377b2-e3fe-4889-995e-55127771d05e-cert\") pod \"ingress-canary-wqf5x\" (UID: \"559377b2-e3fe-4889-995e-55127771d05e\") " pod="openshift-ingress-canary/ingress-canary-wqf5x"
Oct 30 00:31:33.187511 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:33.187479    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed-ca-trust-extracted\") pod \"image-registry-6d4f74cbd-njpvm\" (UID: \"2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed\") " pod="openshift-image-registry/image-registry-6d4f74cbd-njpvm"
Oct 30 00:31:33.187700 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:33.187520    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed-trusted-ca\") pod \"image-registry-6d4f74cbd-njpvm\" (UID: \"2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed\") " pod="openshift-image-registry/image-registry-6d4f74cbd-njpvm"
Oct 30 00:31:33.187700 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:33.187551    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed-installation-pull-secrets\") pod \"image-registry-6d4f74cbd-njpvm\" (UID: \"2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed\") " pod="openshift-image-registry/image-registry-6d4f74cbd-njpvm"
Oct 30 00:31:33.187700 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:33.187567    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed-bound-sa-token\") pod \"image-registry-6d4f74cbd-njpvm\" (UID: \"2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed\") " pod="openshift-image-registry/image-registry-6d4f74cbd-njpvm"
Oct 30 00:31:33.188071 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:33.188053    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5ed36f74-5c9e-408e-a7fb-2364144f054e-ca-trust-extracted\") pod \"image-registry-6c69f88f85-88bkg\" (UID: \"5ed36f74-5c9e-408e-a7fb-2364144f054e\") " pod="openshift-image-registry/image-registry-6c69f88f85-88bkg"
Oct 30 00:31:33.188811 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:33.188786    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5ed36f74-5c9e-408e-a7fb-2364144f054e-trusted-ca\") pod \"image-registry-6c69f88f85-88bkg\" (UID: \"5ed36f74-5c9e-408e-a7fb-2364144f054e\") " pod="openshift-image-registry/image-registry-6c69f88f85-88bkg"
Oct 30 00:31:33.189622 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:33.189597    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed-image-registry-private-configuration\") pod \"image-registry-6d4f74cbd-njpvm\" (UID: \"2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed\") " pod="openshift-image-registry/image-registry-6d4f74cbd-njpvm"
Oct 30 00:31:33.190159 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:33.190135    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed-registry-certificates\") pod \"image-registry-6d4f74cbd-njpvm\" (UID: \"2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed\") " pod="openshift-image-registry/image-registry-6d4f74cbd-njpvm"
Oct 30 00:31:33.190559 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:33.190539    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 00:31:33.190559 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:33.187242    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5ed36f74-5c9e-408e-a7fb-2364144f054e-registry-certificates\") pod \"image-registry-6c69f88f85-88bkg\" (UID: \"5ed36f74-5c9e-408e-a7fb-2364144f054e\") " pod="openshift-image-registry/image-registry-6c69f88f85-88bkg"
Oct 30 00:31:33.190559 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:33.190561    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6d4f74cbd-njpvm: secret "image-registry-tls" not found
Oct 30 00:31:33.190738 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:33.190547    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed-trusted-ca\") pod \"image-registry-6d4f74cbd-njpvm\" (UID: \"2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed\") " pod="openshift-image-registry/image-registry-6d4f74cbd-njpvm"
Oct 30 00:31:33.190738 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:33.190613    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed-registry-tls podName:2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed nodeName:}" failed. No retries permitted until 2026-10-30 00:31:33.69059697 +0000 UTC m=+30.791903843 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed-registry-tls") pod "image-registry-6d4f74cbd-njpvm" (UID: "2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed") : secret "image-registry-tls" not found
Oct 30 00:31:33.190738 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:33.190626    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/559377b2-e3fe-4889-995e-55127771d05e-cert\") pod \"ingress-canary-wqf5x\" (UID: \"559377b2-e3fe-4889-995e-55127771d05e\") " pod="openshift-ingress-canary/ingress-canary-wqf5x"
Oct 30 00:31:33.192156 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:33.192101    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5ed36f74-5c9e-408e-a7fb-2364144f054e-image-registry-private-configuration\") pod \"image-registry-6c69f88f85-88bkg\" (UID: \"5ed36f74-5c9e-408e-a7fb-2364144f054e\") " pod="openshift-image-registry/image-registry-6c69f88f85-88bkg"
Oct 30 00:31:33.192341 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:33.192324    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed-installation-pull-secrets\") pod \"image-registry-6d4f74cbd-njpvm\" (UID: \"2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed\") " pod="openshift-image-registry/image-registry-6d4f74cbd-njpvm"
Oct 30 00:31:33.192466 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:33.192441    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-sxknf"
Oct 30 00:31:33.193401 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:33.193314    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5ed36f74-5c9e-408e-a7fb-2364144f054e-installation-pull-secrets\") pod \"image-registry-6c69f88f85-88bkg\" (UID: \"5ed36f74-5c9e-408e-a7fb-2364144f054e\") " pod="openshift-image-registry/image-registry-6c69f88f85-88bkg"
Oct 30 00:31:33.207346 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:33.207324    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-sklth"
Oct 30 00:31:33.217704 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:33.217658    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmzz7\" (UniqueName: \"kubernetes.io/projected/559377b2-e3fe-4889-995e-55127771d05e-kube-api-access-zmzz7\") pod \"ingress-canary-wqf5x\" (UID: \"559377b2-e3fe-4889-995e-55127771d05e\") " pod="openshift-ingress-canary/ingress-canary-wqf5x"
Oct 30 00:31:33.221522 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:33.221495    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-bjzgs"
Oct 30 00:31:33.232764 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:33.232518    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-84rlf\" (UniqueName: \"kubernetes.io/projected/2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed-kube-api-access-84rlf\") pod \"image-registry-6d4f74cbd-njpvm\" (UID: \"2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed\") " pod="openshift-image-registry/image-registry-6d4f74cbd-njpvm"
Oct 30 00:31:33.232764 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:33.232708    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7l585\" (UniqueName: \"kubernetes.io/projected/5ed36f74-5c9e-408e-a7fb-2364144f054e-kube-api-access-7l585\") pod \"image-registry-6c69f88f85-88bkg\" (UID: \"5ed36f74-5c9e-408e-a7fb-2364144f054e\") " pod="openshift-image-registry/image-registry-6c69f88f85-88bkg"
Oct 30 00:31:33.232934 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:33.232898    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed-bound-sa-token\") pod \"image-registry-6d4f74cbd-njpvm\" (UID: \"2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed\") " pod="openshift-image-registry/image-registry-6d4f74cbd-njpvm"
Oct 30 00:31:33.233097 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:33.233078    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhs2g\" (UniqueName: \"kubernetes.io/projected/9b2b3b14-8027-489a-88f0-d056226c2f67-kube-api-access-bhs2g\") pod \"network-check-source-6f78d59f99-jg5h6\" (UID: \"9b2b3b14-8027-489a-88f0-d056226c2f67\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-jg5h6"
Oct 30 00:31:33.234490 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:33.234470    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5ed36f74-5c9e-408e-a7fb-2364144f054e-bound-sa-token\") pod \"image-registry-6c69f88f85-88bkg\" (UID: \"5ed36f74-5c9e-408e-a7fb-2364144f054e\") " pod="openshift-image-registry/image-registry-6c69f88f85-88bkg"
Oct 30 00:31:33.258627 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:33.256976    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-g7c2w"
Oct 30 00:31:33.261553 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:33.261365    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-jg5h6"
Oct 30 00:31:33.275088 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:33.273903    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-wqf5x"
Oct 30 00:31:33.414485 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:33.413296    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-fbdvl"]
Oct 30 00:31:33.416946 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:33.415582    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod783c7a98_5ebf_44dd_8da9_797e71c28169.slice/crio-c401bfa9817c86ba3e9537e9932491b450d77a68e1d6d855d93aed8ae46c3445 WatchSource:0}: Error finding container c401bfa9817c86ba3e9537e9932491b450d77a68e1d6d855d93aed8ae46c3445: Status 404 returned error can't find the container with id c401bfa9817c86ba3e9537e9932491b450d77a68e1d6d855d93aed8ae46c3445
Oct 30 00:31:33.435800 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:33.435778    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-cqjlz"]
Oct 30 00:31:33.437068 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:33.437040    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9822c484_fd06_4e5e_94b4_ef0f5ade9a13.slice/crio-0a1b48da841c11cc03ad3c6dd62932dd0b7de30c4e34fc221f366ec25b3a9ee5 WatchSource:0}: Error finding container 0a1b48da841c11cc03ad3c6dd62932dd0b7de30c4e34fc221f366ec25b3a9ee5: Status 404 returned error can't find the container with id 0a1b48da841c11cc03ad3c6dd62932dd0b7de30c4e34fc221f366ec25b3a9ee5
Oct 30 00:31:33.454211 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:33.454148    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-f2b4d"]
Oct 30 00:31:33.458608 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:33.458581    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod139583a1_f622_49ab_8c1e_af168fcdf6c4.slice/crio-43e781b035175dffab13f8dbbfd8ba443ef3114ebef4867715bd8514bcd1217e WatchSource:0}: Error finding container 43e781b035175dffab13f8dbbfd8ba443ef3114ebef4867715bd8514bcd1217e: Status 404 returned error can't find the container with id 43e781b035175dffab13f8dbbfd8ba443ef3114ebef4867715bd8514bcd1217e
Oct 30 00:31:33.493401 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:33.492611    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3d234b3-76d4-47d5-98a4-dafd83477b17-metrics-certs\") pod \"router-default-85f7b79cf7-h2tpp\" (UID: \"a3d234b3-76d4-47d5-98a4-dafd83477b17\") " pod="openshift-ingress/router-default-85f7b79cf7-h2tpp"
Oct 30 00:31:33.493401 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:33.492671    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a3d234b3-76d4-47d5-98a4-dafd83477b17-service-ca-bundle\") pod \"router-default-85f7b79cf7-h2tpp\" (UID: \"a3d234b3-76d4-47d5-98a4-dafd83477b17\") " pod="openshift-ingress/router-default-85f7b79cf7-h2tpp"
Oct 30 00:31:33.493401 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:33.492705    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d7c5e976-596e-4cb5-81a9-b16262b3c85b-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-zqdbk\" (UID: \"d7c5e976-596e-4cb5-81a9-b16262b3c85b\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-zqdbk"
Oct 30 00:31:33.493401 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:33.492763    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9aba3099-fe4c-4846-9a34-25db2c2fe593-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-kjsj6\" (UID: \"9aba3099-fe4c-4846-9a34-25db2c2fe593\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-kjsj6"
Oct 30 00:31:33.493401 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:33.492869    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/251cf359-f9f5-44ae-8ba9-42bf0cabb1ab-metrics-tls\") pod \"dns-default-btj6v\" (UID: \"251cf359-f9f5-44ae-8ba9-42bf0cabb1ab\") " pod="openshift-dns/dns-default-btj6v"
Oct 30 00:31:33.493401 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:33.493021    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 00:31:33.493401 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:33.493078    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/251cf359-f9f5-44ae-8ba9-42bf0cabb1ab-metrics-tls podName:251cf359-f9f5-44ae-8ba9-42bf0cabb1ab nodeName:}" failed. No retries permitted until 2026-10-30 00:31:34.493061556 +0000 UTC m=+31.594368432 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/251cf359-f9f5-44ae-8ba9-42bf0cabb1ab-metrics-tls") pod "dns-default-btj6v" (UID: "251cf359-f9f5-44ae-8ba9-42bf0cabb1ab") : secret "dns-default-metrics-tls" not found
Oct 30 00:31:33.497409 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:33.495796    2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 00:31:33.497409 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:33.495858    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a3d234b3-76d4-47d5-98a4-dafd83477b17-metrics-certs podName:a3d234b3-76d4-47d5-98a4-dafd83477b17 nodeName:}" failed. No retries permitted until 2026-10-30 00:31:34.495841537 +0000 UTC m=+31.597148411 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a3d234b3-76d4-47d5-98a4-dafd83477b17-metrics-certs") pod "router-default-85f7b79cf7-h2tpp" (UID: "a3d234b3-76d4-47d5-98a4-dafd83477b17") : secret "router-metrics-certs-default" not found
Oct 30 00:31:33.497409 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:33.495935    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a3d234b3-76d4-47d5-98a4-dafd83477b17-service-ca-bundle podName:a3d234b3-76d4-47d5-98a4-dafd83477b17 nodeName:}" failed. No retries permitted until 2026-10-30 00:31:34.495924135 +0000 UTC m=+31.597231007 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/a3d234b3-76d4-47d5-98a4-dafd83477b17-service-ca-bundle") pod "router-default-85f7b79cf7-h2tpp" (UID: "a3d234b3-76d4-47d5-98a4-dafd83477b17") : configmap references non-existent config key: service-ca.crt
Oct 30 00:31:33.497409 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:33.495990    2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 00:31:33.497409 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:33.496026    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7c5e976-596e-4cb5-81a9-b16262b3c85b-networking-console-plugin-cert podName:d7c5e976-596e-4cb5-81a9-b16262b3c85b nodeName:}" failed. No retries permitted until 2026-10-30 00:31:34.49601415 +0000 UTC m=+31.597321022 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/d7c5e976-596e-4cb5-81a9-b16262b3c85b-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-zqdbk" (UID: "d7c5e976-596e-4cb5-81a9-b16262b3c85b") : secret "networking-console-plugin-cert" not found
Oct 30 00:31:33.497409 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:33.496078    2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 00:31:33.497409 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:33.496108    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9aba3099-fe4c-4846-9a34-25db2c2fe593-cluster-monitoring-operator-tls podName:9aba3099-fe4c-4846-9a34-25db2c2fe593 nodeName:}" failed. No retries permitted until 2026-10-30 00:31:34.496098012 +0000 UTC m=+31.597404884 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/9aba3099-fe4c-4846-9a34-25db2c2fe593-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-kjsj6" (UID: "9aba3099-fe4c-4846-9a34-25db2c2fe593") : secret "cluster-monitoring-operator-tls" not found
Oct 30 00:31:33.509282 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:33.509262    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-246g2"]
Oct 30 00:31:33.511044 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:33.511012    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcea1b649_9c31_4434_9448_7f6652496830.slice/crio-a342adc28b63c1d46dd71a11bcf85d3d52a05101220c43ef41724801c48c9fa4 WatchSource:0}: Error finding container a342adc28b63c1d46dd71a11bcf85d3d52a05101220c43ef41724801c48c9fa4: Status 404 returned error can't find the container with id a342adc28b63c1d46dd71a11bcf85d3d52a05101220c43ef41724801c48c9fa4
Oct 30 00:31:33.523577 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:33.523552    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-sxknf"]
Oct 30 00:31:33.541090 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:33.541066    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb874991e_9562_4848_b8f1_4b5a2b7cddc4.slice/crio-c42e0453a3ed59a2dec4c08c3a2acc4b6c3a846e1abb8dc1b4db077d05741d0c WatchSource:0}: Error finding container c42e0453a3ed59a2dec4c08c3a2acc4b6c3a846e1abb8dc1b4db077d05741d0c: Status 404 returned error can't find the container with id c42e0453a3ed59a2dec4c08c3a2acc4b6c3a846e1abb8dc1b4db077d05741d0c
Oct 30 00:31:33.548907 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:33.548882    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-sklth"]
Oct 30 00:31:33.558333 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:33.558309    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4ca0ea9_b57d_4d4c_be3a_2da5fc26a65d.slice/crio-c0ca8a0286da3206a1984da7ca07bcd8f48346aeb32382aa1ac127f0b959599f WatchSource:0}: Error finding container c0ca8a0286da3206a1984da7ca07bcd8f48346aeb32382aa1ac127f0b959599f: Status 404 returned error can't find the container with id c0ca8a0286da3206a1984da7ca07bcd8f48346aeb32382aa1ac127f0b959599f
Oct 30 00:31:33.582992 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:33.582969    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-bjzgs"]
Oct 30 00:31:33.583601 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:33.583579    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61903cf2_2b92_4f40_ab52_25509de587ef.slice/crio-f43de3eccd69933959e622c93a59fc85020fb8a8b6088a8cfc2cad3cccb1eff0 WatchSource:0}: Error finding container f43de3eccd69933959e622c93a59fc85020fb8a8b6088a8cfc2cad3cccb1eff0: Status 404 returned error can't find the container with id f43de3eccd69933959e622c93a59fc85020fb8a8b6088a8cfc2cad3cccb1eff0
Oct 30 00:31:33.594234 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:33.594212    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1881dcfe-c2da-4253-bca5-93f3526e14fa-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-2nnk8\" (UID: \"1881dcfe-c2da-4253-bca5-93f3526e14fa\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-2nnk8"
Oct 30 00:31:33.594366 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:33.594348    2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 00:31:33.594427 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:33.594421    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1881dcfe-c2da-4253-bca5-93f3526e14fa-samples-operator-tls podName:1881dcfe-c2da-4253-bca5-93f3526e14fa nodeName:}" failed. No retries permitted until 2026-10-30 00:31:34.594406482 +0000 UTC m=+31.695713353 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/1881dcfe-c2da-4253-bca5-93f3526e14fa-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-2nnk8" (UID: "1881dcfe-c2da-4253-bca5-93f3526e14fa") : secret "samples-operator-tls" not found
Oct 30 00:31:33.615216 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:33.615190    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-jg5h6"]
Oct 30 00:31:33.616160 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:33.616138    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b2b3b14_8027_489a_88f0_d056226c2f67.slice/crio-cc9244c17a2ca2d72c4199cc0781aaa7e22b7859adfb3b8c52ee0fb4348e35ef WatchSource:0}: Error finding container cc9244c17a2ca2d72c4199cc0781aaa7e22b7859adfb3b8c52ee0fb4348e35ef: Status 404 returned error can't find the container with id cc9244c17a2ca2d72c4199cc0781aaa7e22b7859adfb3b8c52ee0fb4348e35ef
Oct 30 00:31:33.628002 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:33.627983    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-g7c2w"]
Oct 30 00:31:33.629067 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:33.629037    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5dc56e7e_f3c1_43eb_a21b_8be6bb4706cf.slice/crio-7e631fdaf2b86d82764221a535cde6bc365d8253137f5469f9370b342ecc6c6d WatchSource:0}: Error finding container 7e631fdaf2b86d82764221a535cde6bc365d8253137f5469f9370b342ecc6c6d: Status 404 returned error can't find the container with id 7e631fdaf2b86d82764221a535cde6bc365d8253137f5469f9370b342ecc6c6d
Oct 30 00:31:33.631825 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:33.631797    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-wqf5x"]
Oct 30 00:31:33.632692 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:33.632663    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod559377b2_e3fe_4889_995e_55127771d05e.slice/crio-70a7f9743975c1002c0feb8db36a717b4dc312d0b220be0a3b8dfca5e7122ebd WatchSource:0}: Error finding container 70a7f9743975c1002c0feb8db36a717b4dc312d0b220be0a3b8dfca5e7122ebd: Status 404 returned error can't find the container with id 70a7f9743975c1002c0feb8db36a717b4dc312d0b220be0a3b8dfca5e7122ebd
Oct 30 00:31:33.635823 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:33.635791    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-jg5h6" event={"ID":"9b2b3b14-8027-489a-88f0-d056226c2f67","Type":"ContainerStarted","Data":"cc9244c17a2ca2d72c4199cc0781aaa7e22b7859adfb3b8c52ee0fb4348e35ef"}
Oct 30 00:31:33.637440 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:33.637414    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-cqjlz" event={"ID":"9822c484-fd06-4e5e-94b4-ef0f5ade9a13","Type":"ContainerStarted","Data":"af94ecdc759722f8c2ff4e375826fad0d8f10370eb19f945a6319d32b19301fc"}
Oct 30 00:31:33.637506 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:33.637458    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-cqjlz" event={"ID":"9822c484-fd06-4e5e-94b4-ef0f5ade9a13","Type":"ContainerStarted","Data":"0a1b48da841c11cc03ad3c6dd62932dd0b7de30c4e34fc221f366ec25b3a9ee5"}
Oct 30 00:31:33.638923 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:33.638904    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-bjzgs" event={"ID":"61903cf2-2b92-4f40-ab52-25509de587ef","Type":"ContainerStarted","Data":"f43de3eccd69933959e622c93a59fc85020fb8a8b6088a8cfc2cad3cccb1eff0"}
Oct 30 00:31:33.639920 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:33.639891    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-g7c2w" event={"ID":"5dc56e7e-f3c1-43eb-a21b-8be6bb4706cf","Type":"ContainerStarted","Data":"7e631fdaf2b86d82764221a535cde6bc365d8253137f5469f9370b342ecc6c6d"}
Oct 30 00:31:33.641485 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:33.641459    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-sklth" event={"ID":"e4ca0ea9-b57d-4d4c-be3a-2da5fc26a65d","Type":"ContainerStarted","Data":"c0ca8a0286da3206a1984da7ca07bcd8f48346aeb32382aa1ac127f0b959599f"}
Oct 30 00:31:33.642484 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:33.642468    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-sxknf" event={"ID":"b874991e-9562-4848-b8f1-4b5a2b7cddc4","Type":"ContainerStarted","Data":"c42e0453a3ed59a2dec4c08c3a2acc4b6c3a846e1abb8dc1b4db077d05741d0c"}
Oct 30 00:31:33.643710 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:33.643692    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-fbdvl" event={"ID":"783c7a98-5ebf-44dd-8da9-797e71c28169","Type":"ContainerStarted","Data":"7287f0460bde53418d8cd2363e2d87dd3f33c05709371900a56c2a7c5616c0a3"}
Oct 30 00:31:33.643781 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:33.643730    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-fbdvl" event={"ID":"783c7a98-5ebf-44dd-8da9-797e71c28169","Type":"ContainerStarted","Data":"c401bfa9817c86ba3e9537e9932491b450d77a68e1d6d855d93aed8ae46c3445"}
Oct 30 00:31:33.645337 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:33.645319    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-246g2" event={"ID":"cea1b649-9c31-4434-9448-7f6652496830","Type":"ContainerStarted","Data":"a342adc28b63c1d46dd71a11bcf85d3d52a05101220c43ef41724801c48c9fa4"}
Oct 30 00:31:33.646494 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:33.646476    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-f2b4d" event={"ID":"139583a1-f622-49ab-8c1e-af168fcdf6c4","Type":"ContainerStarted","Data":"ca4d87595d107ba42451d71da5d4af617fc5d3085c41a97162fcb3a902949d8c"}
Oct 30 00:31:33.646582 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:33.646504    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-f2b4d" event={"ID":"139583a1-f622-49ab-8c1e-af168fcdf6c4","Type":"ContainerStarted","Data":"43e781b035175dffab13f8dbbfd8ba443ef3114ebef4867715bd8514bcd1217e"}
Oct 30 00:31:33.670596 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:33.670574    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-m5qgr"
Oct 30 00:31:33.670694 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:33.670671    2579 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 00:31:33.671683 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:33.671574    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-m5qgr"
Oct 30 00:31:33.675825 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:33.675750    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-cqjlz" podStartSLOduration=246.675736186 podStartE2EDuration="4m6.675736186s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:27:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:31:33.675316969 +0000 UTC m=+30.776623851" watchObservedRunningTime="2026-10-30 00:31:33.675736186 +0000 UTC m=+30.777043067"
Oct 30 00:31:33.694661 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:33.694638    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5ed36f74-5c9e-408e-a7fb-2364144f054e-registry-tls\") pod \"image-registry-6c69f88f85-88bkg\" (UID: \"5ed36f74-5c9e-408e-a7fb-2364144f054e\") " pod="openshift-image-registry/image-registry-6c69f88f85-88bkg"
Oct 30 00:31:33.694808 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:33.694790    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 00:31:33.694865 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:33.694809    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6c69f88f85-88bkg: secret "image-registry-tls" not found
Oct 30 00:31:33.694920 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:33.694804    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed-registry-tls\") pod \"image-registry-6d4f74cbd-njpvm\" (UID: \"2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed\") " pod="openshift-image-registry/image-registry-6d4f74cbd-njpvm"
Oct 30 00:31:33.694920 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:33.694877    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5ed36f74-5c9e-408e-a7fb-2364144f054e-registry-tls podName:5ed36f74-5c9e-408e-a7fb-2364144f054e nodeName:}" failed. No retries permitted until 2026-10-30 00:31:34.694862882 +0000 UTC m=+31.796169740 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5ed36f74-5c9e-408e-a7fb-2364144f054e-registry-tls") pod "image-registry-6c69f88f85-88bkg" (UID: "5ed36f74-5c9e-408e-a7fb-2364144f054e") : secret "image-registry-tls" not found
Oct 30 00:31:33.694920 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:33.694890    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 00:31:33.694920 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:33.694905    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6d4f74cbd-njpvm: secret "image-registry-tls" not found
Oct 30 00:31:33.695099 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:33.694962    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed-registry-tls podName:2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed nodeName:}" failed. No retries permitted until 2026-10-30 00:31:34.694946911 +0000 UTC m=+31.796253785 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed-registry-tls") pod "image-registry-6d4f74cbd-njpvm" (UID: "2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed") : secret "image-registry-tls" not found
Oct 30 00:31:33.714514 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:33.714434    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-f2b4d" podStartSLOduration=246.714423387 podStartE2EDuration="4m6.714423387s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:27:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:31:33.714184311 +0000 UTC m=+30.815491191" watchObservedRunningTime="2026-10-30 00:31:33.714423387 +0000 UTC m=+30.815730274"
Oct 30 00:31:33.756144 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:33.756093    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-fbdvl" podStartSLOduration=246.756078964 podStartE2EDuration="4m6.756078964s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:27:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:31:33.755488966 +0000 UTC m=+30.856795846" watchObservedRunningTime="2026-10-30 00:31:33.756078964 +0000 UTC m=+30.857385843"
Oct 30 00:31:34.397611 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:34.397578    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lngrr"
Oct 30 00:31:34.398085 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:34.398067    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tz6g8"
Oct 30 00:31:34.401956 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:34.398502    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z58bb"
Oct 30 00:31:34.413340 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:34.412877    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 00:31:34.413340 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:34.413107    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 00:31:34.413340 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:34.413182    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-rw2kd\""
Oct 30 00:31:34.413340 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:34.413246    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-xrsjq\""
Oct 30 00:31:34.413340 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:34.413313    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-6gpzw\""
Oct 30 00:31:34.505090 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:34.503839    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3d234b3-76d4-47d5-98a4-dafd83477b17-metrics-certs\") pod \"router-default-85f7b79cf7-h2tpp\" (UID: \"a3d234b3-76d4-47d5-98a4-dafd83477b17\") " pod="openshift-ingress/router-default-85f7b79cf7-h2tpp"
Oct 30 00:31:34.505090 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:34.503896    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a3d234b3-76d4-47d5-98a4-dafd83477b17-service-ca-bundle\") pod \"router-default-85f7b79cf7-h2tpp\" (UID: \"a3d234b3-76d4-47d5-98a4-dafd83477b17\") " pod="openshift-ingress/router-default-85f7b79cf7-h2tpp"
Oct 30 00:31:34.505090 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:34.503920    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d7c5e976-596e-4cb5-81a9-b16262b3c85b-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-zqdbk\" (UID: \"d7c5e976-596e-4cb5-81a9-b16262b3c85b\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-zqdbk"
Oct 30 00:31:34.505090 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:34.503982    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9aba3099-fe4c-4846-9a34-25db2c2fe593-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-kjsj6\" (UID: \"9aba3099-fe4c-4846-9a34-25db2c2fe593\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-kjsj6"
Oct 30 00:31:34.505090 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:34.504087    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/251cf359-f9f5-44ae-8ba9-42bf0cabb1ab-metrics-tls\") pod \"dns-default-btj6v\" (UID: \"251cf359-f9f5-44ae-8ba9-42bf0cabb1ab\") " pod="openshift-dns/dns-default-btj6v"
Oct 30 00:31:34.505090 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:34.504245    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 00:31:34.505090 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:34.504332    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/251cf359-f9f5-44ae-8ba9-42bf0cabb1ab-metrics-tls podName:251cf359-f9f5-44ae-8ba9-42bf0cabb1ab nodeName:}" failed. No retries permitted until 2026-10-30 00:31:36.50431476 +0000 UTC m=+33.605621632 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/251cf359-f9f5-44ae-8ba9-42bf0cabb1ab-metrics-tls") pod "dns-default-btj6v" (UID: "251cf359-f9f5-44ae-8ba9-42bf0cabb1ab") : secret "dns-default-metrics-tls" not found
Oct 30 00:31:34.505090 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:34.504796    2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 00:31:34.505090 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:34.504839    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a3d234b3-76d4-47d5-98a4-dafd83477b17-metrics-certs podName:a3d234b3-76d4-47d5-98a4-dafd83477b17 nodeName:}" failed. No retries permitted until 2026-10-30 00:31:36.504826618 +0000 UTC m=+33.606133489 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a3d234b3-76d4-47d5-98a4-dafd83477b17-metrics-certs") pod "router-default-85f7b79cf7-h2tpp" (UID: "a3d234b3-76d4-47d5-98a4-dafd83477b17") : secret "router-metrics-certs-default" not found
Oct 30 00:31:34.505090 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:34.504889    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a3d234b3-76d4-47d5-98a4-dafd83477b17-service-ca-bundle podName:a3d234b3-76d4-47d5-98a4-dafd83477b17 nodeName:}" failed. No retries permitted until 2026-10-30 00:31:36.504883122 +0000 UTC m=+33.606189993 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/a3d234b3-76d4-47d5-98a4-dafd83477b17-service-ca-bundle") pod "router-default-85f7b79cf7-h2tpp" (UID: "a3d234b3-76d4-47d5-98a4-dafd83477b17") : configmap references non-existent config key: service-ca.crt
Oct 30 00:31:34.505090 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:34.504925    2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 00:31:34.505090 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:34.504944    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7c5e976-596e-4cb5-81a9-b16262b3c85b-networking-console-plugin-cert podName:d7c5e976-596e-4cb5-81a9-b16262b3c85b nodeName:}" failed. No retries permitted until 2026-10-30 00:31:36.504938235 +0000 UTC m=+33.606245092 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/d7c5e976-596e-4cb5-81a9-b16262b3c85b-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-zqdbk" (UID: "d7c5e976-596e-4cb5-81a9-b16262b3c85b") : secret "networking-console-plugin-cert" not found
Oct 30 00:31:34.505090 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:34.504975    2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 00:31:34.505090 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:34.504995    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9aba3099-fe4c-4846-9a34-25db2c2fe593-cluster-monitoring-operator-tls podName:9aba3099-fe4c-4846-9a34-25db2c2fe593 nodeName:}" failed. No retries permitted until 2026-10-30 00:31:36.504988889 +0000 UTC m=+33.606295759 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/9aba3099-fe4c-4846-9a34-25db2c2fe593-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-kjsj6" (UID: "9aba3099-fe4c-4846-9a34-25db2c2fe593") : secret "cluster-monitoring-operator-tls" not found
Oct 30 00:31:34.606248 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:34.605228    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1881dcfe-c2da-4253-bca5-93f3526e14fa-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-2nnk8\" (UID: \"1881dcfe-c2da-4253-bca5-93f3526e14fa\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-2nnk8"
Oct 30 00:31:34.606248 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:34.605619    2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 00:31:34.606248 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:34.605680    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1881dcfe-c2da-4253-bca5-93f3526e14fa-samples-operator-tls podName:1881dcfe-c2da-4253-bca5-93f3526e14fa nodeName:}" failed. No retries permitted until 2026-10-30 00:31:36.605663043 +0000 UTC m=+33.706969905 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/1881dcfe-c2da-4253-bca5-93f3526e14fa-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-2nnk8" (UID: "1881dcfe-c2da-4253-bca5-93f3526e14fa") : secret "samples-operator-tls" not found
Oct 30 00:31:34.656358 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:34.656273    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-wqf5x" event={"ID":"559377b2-e3fe-4889-995e-55127771d05e","Type":"ContainerStarted","Data":"70a7f9743975c1002c0feb8db36a717b4dc312d0b220be0a3b8dfca5e7122ebd"}
Oct 30 00:31:34.706227 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:34.706190    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5ed36f74-5c9e-408e-a7fb-2364144f054e-registry-tls\") pod \"image-registry-6c69f88f85-88bkg\" (UID: \"5ed36f74-5c9e-408e-a7fb-2364144f054e\") " pod="openshift-image-registry/image-registry-6c69f88f85-88bkg"
Oct 30 00:31:34.706409 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:34.706284    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed-registry-tls\") pod \"image-registry-6d4f74cbd-njpvm\" (UID: \"2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed\") " pod="openshift-image-registry/image-registry-6d4f74cbd-njpvm"
Oct 30 00:31:34.706489 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:34.706449    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 00:31:34.706489 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:34.706466    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6d4f74cbd-njpvm: secret "image-registry-tls" not found
Oct 30 00:31:34.706584 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:34.706525    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed-registry-tls podName:2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed nodeName:}" failed. No retries permitted until 2026-10-30 00:31:36.706506795 +0000 UTC m=+33.807813876 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed-registry-tls") pod "image-registry-6d4f74cbd-njpvm" (UID: "2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed") : secret "image-registry-tls" not found
Oct 30 00:31:34.706908 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:34.706877    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 00:31:34.706908 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:34.706907    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6c69f88f85-88bkg: secret "image-registry-tls" not found
Oct 30 00:31:34.707053 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:34.706949    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5ed36f74-5c9e-408e-a7fb-2364144f054e-registry-tls podName:5ed36f74-5c9e-408e-a7fb-2364144f054e nodeName:}" failed. No retries permitted until 2026-10-30 00:31:36.706936397 +0000 UTC m=+33.808243268 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5ed36f74-5c9e-408e-a7fb-2364144f054e-registry-tls") pod "image-registry-6c69f88f85-88bkg" (UID: "5ed36f74-5c9e-408e-a7fb-2364144f054e") : secret "image-registry-tls" not found
Oct 30 00:31:36.527771 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:36.527734    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/251cf359-f9f5-44ae-8ba9-42bf0cabb1ab-metrics-tls\") pod \"dns-default-btj6v\" (UID: \"251cf359-f9f5-44ae-8ba9-42bf0cabb1ab\") " pod="openshift-dns/dns-default-btj6v"
Oct 30 00:31:36.528333 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:36.527815    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3d234b3-76d4-47d5-98a4-dafd83477b17-metrics-certs\") pod \"router-default-85f7b79cf7-h2tpp\" (UID: \"a3d234b3-76d4-47d5-98a4-dafd83477b17\") " pod="openshift-ingress/router-default-85f7b79cf7-h2tpp"
Oct 30 00:31:36.528333 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:36.527856    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a3d234b3-76d4-47d5-98a4-dafd83477b17-service-ca-bundle\") pod \"router-default-85f7b79cf7-h2tpp\" (UID: \"a3d234b3-76d4-47d5-98a4-dafd83477b17\") " pod="openshift-ingress/router-default-85f7b79cf7-h2tpp"
Oct 30 00:31:36.528333 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:36.527886    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d7c5e976-596e-4cb5-81a9-b16262b3c85b-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-zqdbk\" (UID: \"d7c5e976-596e-4cb5-81a9-b16262b3c85b\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-zqdbk"
Oct 30 00:31:36.528333 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:36.527910    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 00:31:36.528333 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:36.527940    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9aba3099-fe4c-4846-9a34-25db2c2fe593-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-kjsj6\" (UID: \"9aba3099-fe4c-4846-9a34-25db2c2fe593\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-kjsj6"
Oct 30 00:31:36.528333 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:36.527973    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/251cf359-f9f5-44ae-8ba9-42bf0cabb1ab-metrics-tls podName:251cf359-f9f5-44ae-8ba9-42bf0cabb1ab nodeName:}" failed. No retries permitted until 2026-10-30 00:31:40.527954783 +0000 UTC m=+37.629261658 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/251cf359-f9f5-44ae-8ba9-42bf0cabb1ab-metrics-tls") pod "dns-default-btj6v" (UID: "251cf359-f9f5-44ae-8ba9-42bf0cabb1ab") : secret "dns-default-metrics-tls" not found
Oct 30 00:31:36.528333 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:36.528031    2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 00:31:36.528333 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:36.528067    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9aba3099-fe4c-4846-9a34-25db2c2fe593-cluster-monitoring-operator-tls podName:9aba3099-fe4c-4846-9a34-25db2c2fe593 nodeName:}" failed. No retries permitted until 2026-10-30 00:31:40.528055841 +0000 UTC m=+37.629362717 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/9aba3099-fe4c-4846-9a34-25db2c2fe593-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-kjsj6" (UID: "9aba3099-fe4c-4846-9a34-25db2c2fe593") : secret "cluster-monitoring-operator-tls" not found
Oct 30 00:31:36.528333 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:36.528119    2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 00:31:36.528333 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:36.528152    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a3d234b3-76d4-47d5-98a4-dafd83477b17-metrics-certs podName:a3d234b3-76d4-47d5-98a4-dafd83477b17 nodeName:}" failed. No retries permitted until 2026-10-30 00:31:40.528141209 +0000 UTC m=+37.629448083 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a3d234b3-76d4-47d5-98a4-dafd83477b17-metrics-certs") pod "router-default-85f7b79cf7-h2tpp" (UID: "a3d234b3-76d4-47d5-98a4-dafd83477b17") : secret "router-metrics-certs-default" not found
Oct 30 00:31:36.528333 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:36.528208    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a3d234b3-76d4-47d5-98a4-dafd83477b17-service-ca-bundle podName:a3d234b3-76d4-47d5-98a4-dafd83477b17 nodeName:}" failed. No retries permitted until 2026-10-30 00:31:40.528200287 +0000 UTC m=+37.629507145 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/a3d234b3-76d4-47d5-98a4-dafd83477b17-service-ca-bundle") pod "router-default-85f7b79cf7-h2tpp" (UID: "a3d234b3-76d4-47d5-98a4-dafd83477b17") : configmap references non-existent config key: service-ca.crt
Oct 30 00:31:36.528333 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:36.528280    2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 00:31:36.528333 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:36.528339    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7c5e976-596e-4cb5-81a9-b16262b3c85b-networking-console-plugin-cert podName:d7c5e976-596e-4cb5-81a9-b16262b3c85b nodeName:}" failed. No retries permitted until 2026-10-30 00:31:40.528326765 +0000 UTC m=+37.629633626 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/d7c5e976-596e-4cb5-81a9-b16262b3c85b-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-zqdbk" (UID: "d7c5e976-596e-4cb5-81a9-b16262b3c85b") : secret "networking-console-plugin-cert" not found
Oct 30 00:31:36.628792 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:36.628756    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1881dcfe-c2da-4253-bca5-93f3526e14fa-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-2nnk8\" (UID: \"1881dcfe-c2da-4253-bca5-93f3526e14fa\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-2nnk8"
Oct 30 00:31:36.628973 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:36.628886    2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 00:31:36.628973 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:36.628955    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1881dcfe-c2da-4253-bca5-93f3526e14fa-samples-operator-tls podName:1881dcfe-c2da-4253-bca5-93f3526e14fa nodeName:}" failed. No retries permitted until 2026-10-30 00:31:40.628935742 +0000 UTC m=+37.730242614 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/1881dcfe-c2da-4253-bca5-93f3526e14fa-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-2nnk8" (UID: "1881dcfe-c2da-4253-bca5-93f3526e14fa") : secret "samples-operator-tls" not found
Oct 30 00:31:36.674061 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:36.673966    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-246g2" event={"ID":"cea1b649-9c31-4434-9448-7f6652496830","Type":"ContainerStarted","Data":"feb3c14b250e426f2431f014d6171a04dd1951c6378739e560dbc88b94146889"}
Oct 30 00:31:36.714794 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:36.713793    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-246g2" podStartSLOduration=201.317729394 podStartE2EDuration="3m23.713773813s" totalImagesPullingTime="2.396044419s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:28:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:31:36.713287848 +0000 UTC m=+33.814594730" watchObservedRunningTime="2026-10-30 00:31:36.713773813 +0000 UTC m=+33.815080694"
Oct 30 00:31:36.730523 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:36.730484    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5ed36f74-5c9e-408e-a7fb-2364144f054e-registry-tls\") pod \"image-registry-6c69f88f85-88bkg\" (UID: \"5ed36f74-5c9e-408e-a7fb-2364144f054e\") " pod="openshift-image-registry/image-registry-6c69f88f85-88bkg"
Oct 30 00:31:36.730698 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:36.730578    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 00:31:36.730698 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:36.730595    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6c69f88f85-88bkg: secret "image-registry-tls" not found
Oct 30 00:31:36.730698 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:36.730601    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed-registry-tls\") pod \"image-registry-6d4f74cbd-njpvm\" (UID: \"2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed\") " pod="openshift-image-registry/image-registry-6d4f74cbd-njpvm"
Oct 30 00:31:36.730698 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:36.730654    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5ed36f74-5c9e-408e-a7fb-2364144f054e-registry-tls podName:5ed36f74-5c9e-408e-a7fb-2364144f054e nodeName:}" failed. No retries permitted until 2026-10-30 00:31:40.730636757 +0000 UTC m=+37.831943637 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5ed36f74-5c9e-408e-a7fb-2364144f054e-registry-tls") pod "image-registry-6c69f88f85-88bkg" (UID: "5ed36f74-5c9e-408e-a7fb-2364144f054e") : secret "image-registry-tls" not found
Oct 30 00:31:36.730698 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:36.730692    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 00:31:36.730698 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:36.730702    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6d4f74cbd-njpvm: secret "image-registry-tls" not found
Oct 30 00:31:36.731049 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:36.730735    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed-registry-tls podName:2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed nodeName:}" failed. No retries permitted until 2026-10-30 00:31:40.730724549 +0000 UTC m=+37.832031424 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed-registry-tls") pod "image-registry-6d4f74cbd-njpvm" (UID: "2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed") : secret "image-registry-tls" not found
Oct 30 00:31:37.135311 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:37.135224    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af2d6f84-aea3-4698-be12-11bd6767caec-metrics-certs\") pod \"network-metrics-daemon-z58bb\" (UID: \"af2d6f84-aea3-4698-be12-11bd6767caec\") " pod="openshift-multus/network-metrics-daemon-z58bb"
Oct 30 00:31:37.135311 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:37.135284    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/80603987-09a1-4ca1-b8b4-3c9ac8d0efc6-original-pull-secret\") pod \"global-pull-secret-syncer-tz6g8\" (UID: \"80603987-09a1-4ca1-b8b4-3c9ac8d0efc6\") " pod="kube-system/global-pull-secret-syncer-tz6g8"
Oct 30 00:31:37.135564 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:37.135448    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 00:31:37.135564 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:37.135532    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/af2d6f84-aea3-4698-be12-11bd6767caec-metrics-certs podName:af2d6f84-aea3-4698-be12-11bd6767caec nodeName:}" failed. No retries permitted until 2026-10-30 00:32:09.135514233 +0000 UTC m=+66.236821095 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/af2d6f84-aea3-4698-be12-11bd6767caec-metrics-certs") pod "network-metrics-daemon-z58bb" (UID: "af2d6f84-aea3-4698-be12-11bd6767caec") : secret "metrics-daemon-secret" not found
Oct 30 00:31:37.138395 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:37.138356    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/80603987-09a1-4ca1-b8b4-3c9ac8d0efc6-original-pull-secret\") pod \"global-pull-secret-syncer-tz6g8\" (UID: \"80603987-09a1-4ca1-b8b4-3c9ac8d0efc6\") " pod="kube-system/global-pull-secret-syncer-tz6g8"
Oct 30 00:31:37.160597 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:37.160392    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tz6g8"
Oct 30 00:31:37.236620 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:37.236050    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r998s\" (UniqueName: \"kubernetes.io/projected/d27ee5b1-d1aa-418c-9169-12cac37dcc2c-kube-api-access-r998s\") pod \"network-check-target-lngrr\" (UID: \"d27ee5b1-d1aa-418c-9169-12cac37dcc2c\") " pod="openshift-network-diagnostics/network-check-target-lngrr"
Oct 30 00:31:37.239617 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:37.239518    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r998s\" (UniqueName: \"kubernetes.io/projected/d27ee5b1-d1aa-418c-9169-12cac37dcc2c-kube-api-access-r998s\") pod \"network-check-target-lngrr\" (UID: \"d27ee5b1-d1aa-418c-9169-12cac37dcc2c\") " pod="openshift-network-diagnostics/network-check-target-lngrr"
Oct 30 00:31:37.420296 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:37.420220    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lngrr"
Oct 30 00:31:37.541312 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:37.541282    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-tz6g8"]
Oct 30 00:31:37.650338 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:37.650306    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80603987_09a1_4ca1_b8b4_3c9ac8d0efc6.slice/crio-d4b5066e0524802f2cab4f8c5119b9785e3cd5bbdac5d0b5fd0b350f47f5baaa WatchSource:0}: Error finding container d4b5066e0524802f2cab4f8c5119b9785e3cd5bbdac5d0b5fd0b350f47f5baaa: Status 404 returned error can't find the container with id d4b5066e0524802f2cab4f8c5119b9785e3cd5bbdac5d0b5fd0b350f47f5baaa
Oct 30 00:31:37.681311 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:37.681277    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-tz6g8" event={"ID":"80603987-09a1-4ca1-b8b4-3c9ac8d0efc6","Type":"ContainerStarted","Data":"d4b5066e0524802f2cab4f8c5119b9785e3cd5bbdac5d0b5fd0b350f47f5baaa"}
Oct 30 00:31:38.003069 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:38.003037    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-lngrr"]
Oct 30 00:31:38.163493 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:38.163453    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd27ee5b1_d1aa_418c_9169_12cac37dcc2c.slice/crio-57c013576400090006215c678b7fcc105a57fbc3a4f73eb482fe1d1da5fcf302 WatchSource:0}: Error finding container 57c013576400090006215c678b7fcc105a57fbc3a4f73eb482fe1d1da5fcf302: Status 404 returned error can't find the container with id 57c013576400090006215c678b7fcc105a57fbc3a4f73eb482fe1d1da5fcf302
Oct 30 00:31:38.687149 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:38.687116    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-lngrr" event={"ID":"d27ee5b1-d1aa-418c-9169-12cac37dcc2c","Type":"ContainerStarted","Data":"57c013576400090006215c678b7fcc105a57fbc3a4f73eb482fe1d1da5fcf302"}
Oct 30 00:31:38.690071 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:38.690050    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-g7c2w_5dc56e7e-f3c1-43eb-a21b-8be6bb4706cf/console-operator/0.log"
Oct 30 00:31:38.690173 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:38.690088    2579 generic.go:356] "Generic (PLEG): container finished" podID="5dc56e7e-f3c1-43eb-a21b-8be6bb4706cf" containerID="7a55d5c177963901f74cf2107d8e9e068aa2b7228bcbc9d6ef8002b680110da0" exitCode=255
Oct 30 00:31:38.690173 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:38.690120    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-g7c2w" event={"ID":"5dc56e7e-f3c1-43eb-a21b-8be6bb4706cf","Type":"ContainerDied","Data":"7a55d5c177963901f74cf2107d8e9e068aa2b7228bcbc9d6ef8002b680110da0"}
Oct 30 00:31:38.690488 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:38.690470    2579 scope.go:122] "RemoveContainer" containerID="7a55d5c177963901f74cf2107d8e9e068aa2b7228bcbc9d6ef8002b680110da0"
Oct 30 00:31:39.511252 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:39.511165    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-g2dlc"]
Oct 30 00:31:39.532906 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:39.532873    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-khqcq"]
Oct 30 00:31:39.533837 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:39.533813    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-g2dlc"
Oct 30 00:31:39.548607 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:39.548584    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Oct 30 00:31:39.555654 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:39.555620    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-g2dlc"]
Oct 30 00:31:39.555764 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:39.555661    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-khqcq"]
Oct 30 00:31:39.555764 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:39.555754    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-khqcq"
Oct 30 00:31:39.555880 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:39.555822    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Oct 30 00:31:39.555932 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:39.555921    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-f65k2\""
Oct 30 00:31:39.591526 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:39.591498    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-tbjdq"]
Oct 30 00:31:39.618749 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:39.618726    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-zzlf6-kvglg"]
Oct 30 00:31:39.620420 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:39.618961    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-tbjdq"
Oct 30 00:31:39.644853 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:39.644827    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-tbjdq"]
Oct 30 00:31:39.644979 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:39.644877    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-zzlf6-kvglg"]
Oct 30 00:31:39.644979 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:39.644927    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-zzlf6-kvglg"
Oct 30 00:31:39.648813 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:39.648773    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 30 00:31:39.648922 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:39.648822    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-ngd4v\""
Oct 30 00:31:39.649232 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:39.649175    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 30 00:31:39.659003 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:39.658982    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-9hmc7\""
Oct 30 00:31:39.662417 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:39.662340    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dj9wz\" (UniqueName: \"kubernetes.io/projected/c4d423f3-2077-47fb-bf4d-d9991dd5eac1-kube-api-access-dj9wz\") pod \"migrator-66df5d496b-g2dlc\" (UID: \"c4d423f3-2077-47fb-bf4d-d9991dd5eac1\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-g2dlc"
Oct 30 00:31:39.662508 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:39.662476    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkng8\" (UniqueName: \"kubernetes.io/projected/928e22b3-c3fe-453b-bbc8-a67accd2fcb4-kube-api-access-vkng8\") pod \"migrator-66df5d496b-khqcq\" (UID: \"928e22b3-c3fe-453b-bbc8-a67accd2fcb4\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-khqcq"
Oct 30 00:31:39.719161 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:39.719130    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-g7c2w_5dc56e7e-f3c1-43eb-a21b-8be6bb4706cf/console-operator/1.log"
Oct 30 00:31:39.719982 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:39.719964    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-g7c2w_5dc56e7e-f3c1-43eb-a21b-8be6bb4706cf/console-operator/0.log"
Oct 30 00:31:39.720048 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:39.720001    2579 generic.go:356] "Generic (PLEG): container finished" podID="5dc56e7e-f3c1-43eb-a21b-8be6bb4706cf" containerID="b9a214f6a9f96ff4dcd1880bfe7e3bbf691313502185f22d37d7989d8110023a" exitCode=255
Oct 30 00:31:39.720098 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:39.720082    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-g7c2w" event={"ID":"5dc56e7e-f3c1-43eb-a21b-8be6bb4706cf","Type":"ContainerDied","Data":"b9a214f6a9f96ff4dcd1880bfe7e3bbf691313502185f22d37d7989d8110023a"}
Oct 30 00:31:39.720147 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:39.720125    2579 scope.go:122] "RemoveContainer" containerID="7a55d5c177963901f74cf2107d8e9e068aa2b7228bcbc9d6ef8002b680110da0"
Oct 30 00:31:39.720370 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:39.720352    2579 scope.go:122] "RemoveContainer" containerID="b9a214f6a9f96ff4dcd1880bfe7e3bbf691313502185f22d37d7989d8110023a"
Oct 30 00:31:39.720736 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:39.720662    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-g7c2w_openshift-console-operator(5dc56e7e-f3c1-43eb-a21b-8be6bb4706cf)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-g7c2w" podUID="5dc56e7e-f3c1-43eb-a21b-8be6bb4706cf"
Oct 30 00:31:39.723171 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:39.723139    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-sklth" event={"ID":"e4ca0ea9-b57d-4d4c-be3a-2da5fc26a65d","Type":"ContainerStarted","Data":"696239c269899c960abd90dacba42f6bd67abbd33c8406dc29a4900955093ae4"}
Oct 30 00:31:39.725674 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:39.725648    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-sxknf" event={"ID":"b874991e-9562-4848-b8f1-4b5a2b7cddc4","Type":"ContainerStarted","Data":"6480afc5a4cd6b87539908cc8f161f96c3724a81bfd3e829a511c95f9e005bf8"}
Oct 30 00:31:39.727618 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:39.727598    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-jg5h6" event={"ID":"9b2b3b14-8027-489a-88f0-d056226c2f67","Type":"ContainerStarted","Data":"ae837303f806c38e9b85bf0b349ab989b7febb64907ec7e2e45276fe3cd611a8"}
Oct 30 00:31:39.729769 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:39.729740    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-wqf5x" event={"ID":"559377b2-e3fe-4889-995e-55127771d05e","Type":"ContainerStarted","Data":"bd1437e46065c7d2e9f06b1e73d55cc0debaab8019918b26d829687d15b40eaf"}
Oct 30 00:31:39.731485 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:39.731463    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-bjzgs" event={"ID":"61903cf2-2b92-4f40-ab52-25509de587ef","Type":"ContainerStarted","Data":"e60031b407f1559d4973b6c9c56cd518f0aa584a7f73932563c0129f202e36f4"}
Oct 30 00:31:39.763053 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:39.762987    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkng8\" (UniqueName: \"kubernetes.io/projected/928e22b3-c3fe-453b-bbc8-a67accd2fcb4-kube-api-access-vkng8\") pod \"migrator-66df5d496b-khqcq\" (UID: \"928e22b3-c3fe-453b-bbc8-a67accd2fcb4\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-khqcq"
Oct 30 00:31:39.763155 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:39.763068    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/bba787ce-4e61-4a5a-a728-af73c68dfdd3-archives-path\") pod \"periodic-gathering-zzlf6-kvglg\" (UID: \"bba787ce-4e61-4a5a-a728-af73c68dfdd3\") " pod="openshift-insights/periodic-gathering-zzlf6-kvglg"
Oct 30 00:31:39.763155 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:39.763098    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzc86\" (UniqueName: \"kubernetes.io/projected/bba787ce-4e61-4a5a-a728-af73c68dfdd3-kube-api-access-fzc86\") pod \"periodic-gathering-zzlf6-kvglg\" (UID: \"bba787ce-4e61-4a5a-a728-af73c68dfdd3\") " pod="openshift-insights/periodic-gathering-zzlf6-kvglg"
Oct 30 00:31:39.763236 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:39.763154    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/94a6f5e3-d553-487e-a6f1-8c2b0fc5c695-crio-socket\") pod \"insights-runtime-extractor-tbjdq\" (UID: \"94a6f5e3-d553-487e-a6f1-8c2b0fc5c695\") " pod="openshift-insights/insights-runtime-extractor-tbjdq"
Oct 30 00:31:39.763236 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:39.763189    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/94a6f5e3-d553-487e-a6f1-8c2b0fc5c695-data-volume\") pod \"insights-runtime-extractor-tbjdq\" (UID: \"94a6f5e3-d553-487e-a6f1-8c2b0fc5c695\") " pod="openshift-insights/insights-runtime-extractor-tbjdq"
Oct 30 00:31:39.763302 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:39.763244    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bba787ce-4e61-4a5a-a728-af73c68dfdd3-trusted-ca-bundle\") pod \"periodic-gathering-zzlf6-kvglg\" (UID: \"bba787ce-4e61-4a5a-a728-af73c68dfdd3\") " pod="openshift-insights/periodic-gathering-zzlf6-kvglg"
Oct 30 00:31:39.763302 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:39.763273    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bba787ce-4e61-4a5a-a728-af73c68dfdd3-service-ca-bundle\") pod \"periodic-gathering-zzlf6-kvglg\" (UID: \"bba787ce-4e61-4a5a-a728-af73c68dfdd3\") " pod="openshift-insights/periodic-gathering-zzlf6-kvglg"
Oct 30 00:31:39.763404 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:39.763324    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjms2\" (UniqueName: \"kubernetes.io/projected/94a6f5e3-d553-487e-a6f1-8c2b0fc5c695-kube-api-access-wjms2\") pod \"insights-runtime-extractor-tbjdq\" (UID: \"94a6f5e3-d553-487e-a6f1-8c2b0fc5c695\") " pod="openshift-insights/insights-runtime-extractor-tbjdq"
Oct 30 00:31:39.763404 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:39.763344    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/94a6f5e3-d553-487e-a6f1-8c2b0fc5c695-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tbjdq\" (UID: \"94a6f5e3-d553-487e-a6f1-8c2b0fc5c695\") " pod="openshift-insights/insights-runtime-extractor-tbjdq"
Oct 30 00:31:39.763404 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:39.763372    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dj9wz\" (UniqueName: \"kubernetes.io/projected/c4d423f3-2077-47fb-bf4d-d9991dd5eac1-kube-api-access-dj9wz\") pod \"migrator-66df5d496b-g2dlc\" (UID: \"c4d423f3-2077-47fb-bf4d-d9991dd5eac1\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-g2dlc"
Oct 30 00:31:39.763505 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:39.763453    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/94a6f5e3-d553-487e-a6f1-8c2b0fc5c695-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tbjdq\" (UID: \"94a6f5e3-d553-487e-a6f1-8c2b0fc5c695\") " pod="openshift-insights/insights-runtime-extractor-tbjdq"
Oct 30 00:31:39.801011 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:39.800976    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkng8\" (UniqueName: \"kubernetes.io/projected/928e22b3-c3fe-453b-bbc8-a67accd2fcb4-kube-api-access-vkng8\") pod \"migrator-66df5d496b-khqcq\" (UID: \"928e22b3-c3fe-453b-bbc8-a67accd2fcb4\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-khqcq"
Oct 30 00:31:39.806198 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:39.806168    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dj9wz\" (UniqueName: \"kubernetes.io/projected/c4d423f3-2077-47fb-bf4d-d9991dd5eac1-kube-api-access-dj9wz\") pod \"migrator-66df5d496b-g2dlc\" (UID: \"c4d423f3-2077-47fb-bf4d-d9991dd5eac1\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-g2dlc"
Oct 30 00:31:39.814440 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:39.814373    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-jg5h6" podStartSLOduration=200.869071347 podStartE2EDuration="3m26.814355397s" totalImagesPullingTime="5.94528405s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:28:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:31:39.813075775 +0000 UTC m=+36.914382658" watchObservedRunningTime="2026-10-30 00:31:39.814355397 +0000 UTC m=+36.915662274"
Oct 30 00:31:39.857480 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:39.857403    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-operator-7c4986d449-sklth" podStartSLOduration=201.775481443 podStartE2EDuration="3m26.857370039s" totalImagesPullingTime="5.081888596s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:28:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:31:39.856170683 +0000 UTC m=+36.957477564" watchObservedRunningTime="2026-10-30 00:31:39.857370039 +0000 UTC m=+36.958676922"
Oct 30 00:31:39.865576 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:39.865539    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/94a6f5e3-d553-487e-a6f1-8c2b0fc5c695-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tbjdq\" (UID: \"94a6f5e3-d553-487e-a6f1-8c2b0fc5c695\") " pod="openshift-insights/insights-runtime-extractor-tbjdq"
Oct 30 00:31:39.865740 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:39.865647    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/bba787ce-4e61-4a5a-a728-af73c68dfdd3-archives-path\") pod \"periodic-gathering-zzlf6-kvglg\" (UID: \"bba787ce-4e61-4a5a-a728-af73c68dfdd3\") " pod="openshift-insights/periodic-gathering-zzlf6-kvglg"
Oct 30 00:31:39.865740 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:39.865664    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzc86\" (UniqueName: \"kubernetes.io/projected/bba787ce-4e61-4a5a-a728-af73c68dfdd3-kube-api-access-fzc86\") pod \"periodic-gathering-zzlf6-kvglg\" (UID: \"bba787ce-4e61-4a5a-a728-af73c68dfdd3\") " pod="openshift-insights/periodic-gathering-zzlf6-kvglg"
Oct 30 00:31:39.865740 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:39.865718    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/94a6f5e3-d553-487e-a6f1-8c2b0fc5c695-crio-socket\") pod \"insights-runtime-extractor-tbjdq\" (UID: \"94a6f5e3-d553-487e-a6f1-8c2b0fc5c695\") " pod="openshift-insights/insights-runtime-extractor-tbjdq"
Oct 30 00:31:39.865888 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:39.865745    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/94a6f5e3-d553-487e-a6f1-8c2b0fc5c695-data-volume\") pod \"insights-runtime-extractor-tbjdq\" (UID: \"94a6f5e3-d553-487e-a6f1-8c2b0fc5c695\") " pod="openshift-insights/insights-runtime-extractor-tbjdq"
Oct 30 00:31:39.865888 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:39.865816    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bba787ce-4e61-4a5a-a728-af73c68dfdd3-trusted-ca-bundle\") pod \"periodic-gathering-zzlf6-kvglg\" (UID: \"bba787ce-4e61-4a5a-a728-af73c68dfdd3\") " pod="openshift-insights/periodic-gathering-zzlf6-kvglg"
Oct 30 00:31:39.865888 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:39.865861    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bba787ce-4e61-4a5a-a728-af73c68dfdd3-service-ca-bundle\") pod \"periodic-gathering-zzlf6-kvglg\" (UID: \"bba787ce-4e61-4a5a-a728-af73c68dfdd3\") " pod="openshift-insights/periodic-gathering-zzlf6-kvglg"
Oct 30 00:31:39.866039 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:39.865922    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjms2\" (UniqueName: \"kubernetes.io/projected/94a6f5e3-d553-487e-a6f1-8c2b0fc5c695-kube-api-access-wjms2\") pod \"insights-runtime-extractor-tbjdq\" (UID: \"94a6f5e3-d553-487e-a6f1-8c2b0fc5c695\") " pod="openshift-insights/insights-runtime-extractor-tbjdq"
Oct 30 00:31:39.866039 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:39.865943    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/94a6f5e3-d553-487e-a6f1-8c2b0fc5c695-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tbjdq\" (UID: \"94a6f5e3-d553-487e-a6f1-8c2b0fc5c695\") " pod="openshift-insights/insights-runtime-extractor-tbjdq"
Oct 30 00:31:39.866975 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:39.866946    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 00:31:39.867101 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:39.867018    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/94a6f5e3-d553-487e-a6f1-8c2b0fc5c695-kube-rbac-proxy-cm podName:94a6f5e3-d553-487e-a6f1-8c2b0fc5c695 nodeName:}" failed. No retries permitted until 2026-10-30 00:31:40.366998099 +0000 UTC m=+37.468304957 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/94a6f5e3-d553-487e-a6f1-8c2b0fc5c695-kube-rbac-proxy-cm") pod "insights-runtime-extractor-tbjdq" (UID: "94a6f5e3-d553-487e-a6f1-8c2b0fc5c695") : configmap "kube-rbac-proxy" not found
Oct 30 00:31:39.867801 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:39.867771    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bba787ce-4e61-4a5a-a728-af73c68dfdd3-trusted-ca-bundle\") pod \"periodic-gathering-zzlf6-kvglg\" (UID: \"bba787ce-4e61-4a5a-a728-af73c68dfdd3\") " pod="openshift-insights/periodic-gathering-zzlf6-kvglg"
Oct 30 00:31:39.868869 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:39.868844    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/bba787ce-4e61-4a5a-a728-af73c68dfdd3-archives-path\") pod \"periodic-gathering-zzlf6-kvglg\" (UID: \"bba787ce-4e61-4a5a-a728-af73c68dfdd3\") " pod="openshift-insights/periodic-gathering-zzlf6-kvglg"
Oct 30 00:31:39.869478 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:39.869454    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/94a6f5e3-d553-487e-a6f1-8c2b0fc5c695-crio-socket\") pod \"insights-runtime-extractor-tbjdq\" (UID: \"94a6f5e3-d553-487e-a6f1-8c2b0fc5c695\") " pod="openshift-insights/insights-runtime-extractor-tbjdq"
Oct 30 00:31:39.872843 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:39.869889    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/94a6f5e3-d553-487e-a6f1-8c2b0fc5c695-data-volume\") pod \"insights-runtime-extractor-tbjdq\" (UID: \"94a6f5e3-d553-487e-a6f1-8c2b0fc5c695\") " pod="openshift-insights/insights-runtime-extractor-tbjdq"
Oct 30 00:31:39.872843 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:39.872075    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 00:31:39.872843 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:39.872131    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94a6f5e3-d553-487e-a6f1-8c2b0fc5c695-insights-runtime-extractor-tls podName:94a6f5e3-d553-487e-a6f1-8c2b0fc5c695 nodeName:}" failed. No retries permitted until 2026-10-30 00:31:40.372113042 +0000 UTC m=+37.473419902 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/94a6f5e3-d553-487e-a6f1-8c2b0fc5c695-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tbjdq" (UID: "94a6f5e3-d553-487e-a6f1-8c2b0fc5c695") : secret "insights-runtime-extractor-tls" not found
Oct 30 00:31:39.872843 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:39.872670    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-g2dlc"
Oct 30 00:31:39.873926 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:39.873902    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bba787ce-4e61-4a5a-a728-af73c68dfdd3-service-ca-bundle\") pod \"periodic-gathering-zzlf6-kvglg\" (UID: \"bba787ce-4e61-4a5a-a728-af73c68dfdd3\") " pod="openshift-insights/periodic-gathering-zzlf6-kvglg"
Oct 30 00:31:39.895267 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:39.895226    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjms2\" (UniqueName: \"kubernetes.io/projected/94a6f5e3-d553-487e-a6f1-8c2b0fc5c695-kube-api-access-wjms2\") pod \"insights-runtime-extractor-tbjdq\" (UID: \"94a6f5e3-d553-487e-a6f1-8c2b0fc5c695\") " pod="openshift-insights/insights-runtime-extractor-tbjdq"
Oct 30 00:31:39.898852 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:39.898793    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzc86\" (UniqueName: \"kubernetes.io/projected/bba787ce-4e61-4a5a-a728-af73c68dfdd3-kube-api-access-fzc86\") pod \"periodic-gathering-zzlf6-kvglg\" (UID: \"bba787ce-4e61-4a5a-a728-af73c68dfdd3\") " pod="openshift-insights/periodic-gathering-zzlf6-kvglg"
Oct 30 00:31:39.898852 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:39.898049    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-sxknf" podStartSLOduration=201.7998779 podStartE2EDuration="3m26.898030573s" totalImagesPullingTime="5.098152673s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:28:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:31:39.895661784 +0000 UTC m=+36.996968668" watchObservedRunningTime="2026-10-30 00:31:39.898030573 +0000 UTC m=+36.999337455"
Oct 30 00:31:39.901190 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:39.900974    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-khqcq"
Oct 30 00:31:39.936606 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:39.936475    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-bjzgs" podStartSLOduration=201.878372518 podStartE2EDuration="3m26.936461084s" totalImagesPullingTime="5.058088566s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:28:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:31:39.935089036 +0000 UTC m=+37.036395919" watchObservedRunningTime="2026-10-30 00:31:39.936461084 +0000 UTC m=+37.037767964"
Oct 30 00:31:40.020214 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:40.019921    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-zzlf6-kvglg"
Oct 30 00:31:40.209121 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:40.209069    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-wqf5x" podStartSLOduration=3.177956868 podStartE2EDuration="8.209052744s" totalImagesPullingTime="5.031095876s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:31:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:31:39.984253277 +0000 UTC m=+37.085560158" watchObservedRunningTime="2026-10-30 00:31:40.209052744 +0000 UTC m=+37.310359625"
Oct 30 00:31:40.209720 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:40.209699    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-g2dlc"]
Oct 30 00:31:40.211213 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:40.211184    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4d423f3_2077_47fb_bf4d_d9991dd5eac1.slice/crio-3e0825f9e3c3deaa21e8cceb37a1971d348f6672ea7e4ccf29e29caaf09b6cef WatchSource:0}: Error finding container 3e0825f9e3c3deaa21e8cceb37a1971d348f6672ea7e4ccf29e29caaf09b6cef: Status 404 returned error can't find the container with id 3e0825f9e3c3deaa21e8cceb37a1971d348f6672ea7e4ccf29e29caaf09b6cef
Oct 30 00:31:40.279173 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:40.279143    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-khqcq"]
Oct 30 00:31:40.281066 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:40.281023    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod928e22b3_c3fe_453b_bbc8_a67accd2fcb4.slice/crio-208de9f5a67ade098d715f55f9e24682f2698180040d35b44971ed390bfe1312 WatchSource:0}: Error finding container 208de9f5a67ade098d715f55f9e24682f2698180040d35b44971ed390bfe1312: Status 404 returned error can't find the container with id 208de9f5a67ade098d715f55f9e24682f2698180040d35b44971ed390bfe1312
Oct 30 00:31:40.364898 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:40.364871    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-zzlf6-kvglg"]
Oct 30 00:31:40.366219 ip-10-0-128-7 kubenswrapper[2579]: W1030 00:31:40.366185    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbba787ce_4e61_4a5a_a728_af73c68dfdd3.slice/crio-b984324d5c7f9d7f10d6b87625aafb838e43218eaa1db56a88123a6411f19142 WatchSource:0}: Error finding container b984324d5c7f9d7f10d6b87625aafb838e43218eaa1db56a88123a6411f19142: Status 404 returned error can't find the container with id b984324d5c7f9d7f10d6b87625aafb838e43218eaa1db56a88123a6411f19142
Oct 30 00:31:40.372224 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:40.372199    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/94a6f5e3-d553-487e-a6f1-8c2b0fc5c695-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tbjdq\" (UID: \"94a6f5e3-d553-487e-a6f1-8c2b0fc5c695\") " pod="openshift-insights/insights-runtime-extractor-tbjdq"
Oct 30 00:31:40.372322 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:40.372290    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/94a6f5e3-d553-487e-a6f1-8c2b0fc5c695-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tbjdq\" (UID: \"94a6f5e3-d553-487e-a6f1-8c2b0fc5c695\") " pod="openshift-insights/insights-runtime-extractor-tbjdq"
Oct 30 00:31:40.372805 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:40.372443    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 00:31:40.372805 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:40.372471    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 00:31:40.372805 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:40.372742    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94a6f5e3-d553-487e-a6f1-8c2b0fc5c695-insights-runtime-extractor-tls podName:94a6f5e3-d553-487e-a6f1-8c2b0fc5c695 nodeName:}" failed. No retries permitted until 2026-10-30 00:31:41.372726878 +0000 UTC m=+38.474033752 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/94a6f5e3-d553-487e-a6f1-8c2b0fc5c695-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tbjdq" (UID: "94a6f5e3-d553-487e-a6f1-8c2b0fc5c695") : secret "insights-runtime-extractor-tls" not found
Oct 30 00:31:40.372805 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:40.372782    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/94a6f5e3-d553-487e-a6f1-8c2b0fc5c695-kube-rbac-proxy-cm podName:94a6f5e3-d553-487e-a6f1-8c2b0fc5c695 nodeName:}" failed. No retries permitted until 2026-10-30 00:31:41.372763527 +0000 UTC m=+38.474070391 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/94a6f5e3-d553-487e-a6f1-8c2b0fc5c695-kube-rbac-proxy-cm") pod "insights-runtime-extractor-tbjdq" (UID: "94a6f5e3-d553-487e-a6f1-8c2b0fc5c695") : configmap "kube-rbac-proxy" not found
Oct 30 00:31:40.574759 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:40.574684    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9aba3099-fe4c-4846-9a34-25db2c2fe593-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-kjsj6\" (UID: \"9aba3099-fe4c-4846-9a34-25db2c2fe593\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-kjsj6"
Oct 30 00:31:40.574893 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:40.574784    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/251cf359-f9f5-44ae-8ba9-42bf0cabb1ab-metrics-tls\") pod \"dns-default-btj6v\" (UID: \"251cf359-f9f5-44ae-8ba9-42bf0cabb1ab\") " pod="openshift-dns/dns-default-btj6v"
Oct 30 00:31:40.574893 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:40.574818    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3d234b3-76d4-47d5-98a4-dafd83477b17-metrics-certs\") pod \"router-default-85f7b79cf7-h2tpp\" (UID: \"a3d234b3-76d4-47d5-98a4-dafd83477b17\") " pod="openshift-ingress/router-default-85f7b79cf7-h2tpp"
Oct 30 00:31:40.574893 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:40.574845    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a3d234b3-76d4-47d5-98a4-dafd83477b17-service-ca-bundle\") pod \"router-default-85f7b79cf7-h2tpp\" (UID: \"a3d234b3-76d4-47d5-98a4-dafd83477b17\") " pod="openshift-ingress/router-default-85f7b79cf7-h2tpp"
Oct 30 00:31:40.574893 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:40.574862    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d7c5e976-596e-4cb5-81a9-b16262b3c85b-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-zqdbk\" (UID: \"d7c5e976-596e-4cb5-81a9-b16262b3c85b\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-zqdbk"
Oct 30 00:31:40.575015 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:40.574960    2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 00:31:40.575015 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:40.575008    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7c5e976-596e-4cb5-81a9-b16262b3c85b-networking-console-plugin-cert podName:d7c5e976-596e-4cb5-81a9-b16262b3c85b nodeName:}" failed. No retries permitted until 2026-10-30 00:31:48.57499498 +0000 UTC m=+45.676301851 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/d7c5e976-596e-4cb5-81a9-b16262b3c85b-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-zqdbk" (UID: "d7c5e976-596e-4cb5-81a9-b16262b3c85b") : secret "networking-console-plugin-cert" not found
Oct 30 00:31:40.575407 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:40.575351    2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 00:31:40.575510 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:40.575436    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9aba3099-fe4c-4846-9a34-25db2c2fe593-cluster-monitoring-operator-tls podName:9aba3099-fe4c-4846-9a34-25db2c2fe593 nodeName:}" failed. No retries permitted until 2026-10-30 00:31:48.575419146 +0000 UTC m=+45.676726021 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/9aba3099-fe4c-4846-9a34-25db2c2fe593-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-kjsj6" (UID: "9aba3099-fe4c-4846-9a34-25db2c2fe593") : secret "cluster-monitoring-operator-tls" not found
Oct 30 00:31:40.575510 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:40.575496    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 00:31:40.575587 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:40.575531    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/251cf359-f9f5-44ae-8ba9-42bf0cabb1ab-metrics-tls podName:251cf359-f9f5-44ae-8ba9-42bf0cabb1ab nodeName:}" failed. No retries permitted until 2026-10-30 00:31:48.575520464 +0000 UTC m=+45.676827337 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/251cf359-f9f5-44ae-8ba9-42bf0cabb1ab-metrics-tls") pod "dns-default-btj6v" (UID: "251cf359-f9f5-44ae-8ba9-42bf0cabb1ab") : secret "dns-default-metrics-tls" not found
Oct 30 00:31:40.575587 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:40.575577    2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 00:31:40.575651 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:40.575608    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a3d234b3-76d4-47d5-98a4-dafd83477b17-metrics-certs podName:a3d234b3-76d4-47d5-98a4-dafd83477b17 nodeName:}" failed. No retries permitted until 2026-10-30 00:31:48.575598108 +0000 UTC m=+45.676904980 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a3d234b3-76d4-47d5-98a4-dafd83477b17-metrics-certs") pod "router-default-85f7b79cf7-h2tpp" (UID: "a3d234b3-76d4-47d5-98a4-dafd83477b17") : secret "router-metrics-certs-default" not found
Oct 30 00:31:40.575686 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:40.575672    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a3d234b3-76d4-47d5-98a4-dafd83477b17-service-ca-bundle podName:a3d234b3-76d4-47d5-98a4-dafd83477b17 nodeName:}" failed. No retries permitted until 2026-10-30 00:31:48.575652965 +0000 UTC m=+45.676959837 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/a3d234b3-76d4-47d5-98a4-dafd83477b17-service-ca-bundle") pod "router-default-85f7b79cf7-h2tpp" (UID: "a3d234b3-76d4-47d5-98a4-dafd83477b17") : configmap references non-existent config key: service-ca.crt
Oct 30 00:31:40.675706 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:40.675667    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1881dcfe-c2da-4253-bca5-93f3526e14fa-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-2nnk8\" (UID: \"1881dcfe-c2da-4253-bca5-93f3526e14fa\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-2nnk8"
Oct 30 00:31:40.675967 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:40.675841    2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 00:31:40.675967 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:40.675914    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1881dcfe-c2da-4253-bca5-93f3526e14fa-samples-operator-tls podName:1881dcfe-c2da-4253-bca5-93f3526e14fa nodeName:}" failed. No retries permitted until 2026-10-30 00:31:48.675894336 +0000 UTC m=+45.777201214 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/1881dcfe-c2da-4253-bca5-93f3526e14fa-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-2nnk8" (UID: "1881dcfe-c2da-4253-bca5-93f3526e14fa") : secret "samples-operator-tls" not found
Oct 30 00:31:40.738209 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:40.738186    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-g7c2w_5dc56e7e-f3c1-43eb-a21b-8be6bb4706cf/console-operator/1.log"
Oct 30 00:31:40.738636 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:40.738616    2579 scope.go:122] "RemoveContainer" containerID="b9a214f6a9f96ff4dcd1880bfe7e3bbf691313502185f22d37d7989d8110023a"
Oct 30 00:31:40.738861 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:40.738834    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-g7c2w_openshift-console-operator(5dc56e7e-f3c1-43eb-a21b-8be6bb4706cf)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-g7c2w" podUID="5dc56e7e-f3c1-43eb-a21b-8be6bb4706cf"
Oct 30 00:31:40.740177 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:40.740148    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-zzlf6-kvglg" event={"ID":"bba787ce-4e61-4a5a-a728-af73c68dfdd3","Type":"ContainerStarted","Data":"c46ed1ec8ce6982a6892e8f614588ea94fa7e225818413fd0196bced66dec9f3"}
Oct 30 00:31:40.740287 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:40.740231    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-zzlf6-kvglg" event={"ID":"bba787ce-4e61-4a5a-a728-af73c68dfdd3","Type":"ContainerStarted","Data":"b984324d5c7f9d7f10d6b87625aafb838e43218eaa1db56a88123a6411f19142"}
Oct 30 00:31:40.742036 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:40.742009    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-g2dlc" event={"ID":"c4d423f3-2077-47fb-bf4d-d9991dd5eac1","Type":"ContainerStarted","Data":"3e0825f9e3c3deaa21e8cceb37a1971d348f6672ea7e4ccf29e29caaf09b6cef"}
Oct 30 00:31:40.743844 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:40.743810    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-khqcq" event={"ID":"928e22b3-c3fe-453b-bbc8-a67accd2fcb4","Type":"ContainerStarted","Data":"208de9f5a67ade098d715f55f9e24682f2698180040d35b44971ed390bfe1312"}
Oct 30 00:31:40.777593 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:40.777182    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed-registry-tls\") pod \"image-registry-6d4f74cbd-njpvm\" (UID: \"2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed\") " pod="openshift-image-registry/image-registry-6d4f74cbd-njpvm"
Oct 30 00:31:40.777593 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:40.777309    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 00:31:40.777593 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:40.777333    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6d4f74cbd-njpvm: secret "image-registry-tls" not found
Oct 30 00:31:40.777593 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:40.777363    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5ed36f74-5c9e-408e-a7fb-2364144f054e-registry-tls\") pod \"image-registry-6c69f88f85-88bkg\" (UID: \"5ed36f74-5c9e-408e-a7fb-2364144f054e\") " pod="openshift-image-registry/image-registry-6c69f88f85-88bkg"
Oct 30 00:31:40.777593 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:40.777491    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed-registry-tls podName:2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed nodeName:}" failed. No retries permitted until 2026-10-30 00:31:48.777460257 +0000 UTC m=+45.878767132 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed-registry-tls") pod "image-registry-6d4f74cbd-njpvm" (UID: "2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed") : secret "image-registry-tls" not found
Oct 30 00:31:40.777593 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:40.777515    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 00:31:40.777593 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:40.777527    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6c69f88f85-88bkg: secret "image-registry-tls" not found
Oct 30 00:31:40.777593 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:40.777569    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5ed36f74-5c9e-408e-a7fb-2364144f054e-registry-tls podName:5ed36f74-5c9e-408e-a7fb-2364144f054e nodeName:}" failed. No retries permitted until 2026-10-30 00:31:48.777554261 +0000 UTC m=+45.878861124 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5ed36f74-5c9e-408e-a7fb-2364144f054e-registry-tls") pod "image-registry-6c69f88f85-88bkg" (UID: "5ed36f74-5c9e-408e-a7fb-2364144f054e") : secret "image-registry-tls" not found
Oct 30 00:31:40.822605 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:40.822537    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-zzlf6-kvglg" podStartSLOduration=1.82251943 podStartE2EDuration="1.82251943s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:31:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:31:40.821957997 +0000 UTC m=+37.923264873" watchObservedRunningTime="2026-10-30 00:31:40.82251943 +0000 UTC m=+37.923826315"
Oct 30 00:31:41.384772 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:41.384734    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/94a6f5e3-d553-487e-a6f1-8c2b0fc5c695-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tbjdq\" (UID: \"94a6f5e3-d553-487e-a6f1-8c2b0fc5c695\") " pod="openshift-insights/insights-runtime-extractor-tbjdq"
Oct 30 00:31:41.384922 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:41.384832    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/94a6f5e3-d553-487e-a6f1-8c2b0fc5c695-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tbjdq\" (UID: \"94a6f5e3-d553-487e-a6f1-8c2b0fc5c695\") " pod="openshift-insights/insights-runtime-extractor-tbjdq"
Oct 30 00:31:41.385064 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:41.385043    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 00:31:41.385169 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:41.385116    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/94a6f5e3-d553-487e-a6f1-8c2b0fc5c695-kube-rbac-proxy-cm podName:94a6f5e3-d553-487e-a6f1-8c2b0fc5c695 nodeName:}" failed. No retries permitted until 2026-10-30 00:31:43.385096661 +0000 UTC m=+40.486403538 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/94a6f5e3-d553-487e-a6f1-8c2b0fc5c695-kube-rbac-proxy-cm") pod "insights-runtime-extractor-tbjdq" (UID: "94a6f5e3-d553-487e-a6f1-8c2b0fc5c695") : configmap "kube-rbac-proxy" not found
Oct 30 00:31:41.385558 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:41.385535    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 00:31:41.385634 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:41.385586    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94a6f5e3-d553-487e-a6f1-8c2b0fc5c695-insights-runtime-extractor-tls podName:94a6f5e3-d553-487e-a6f1-8c2b0fc5c695 nodeName:}" failed. No retries permitted until 2026-10-30 00:31:43.385571834 +0000 UTC m=+40.486878694 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/94a6f5e3-d553-487e-a6f1-8c2b0fc5c695-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tbjdq" (UID: "94a6f5e3-d553-487e-a6f1-8c2b0fc5c695") : secret "insights-runtime-extractor-tls" not found
Oct 30 00:31:41.748989 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:41.748955    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-lngrr" event={"ID":"d27ee5b1-d1aa-418c-9169-12cac37dcc2c","Type":"ContainerStarted","Data":"d6721beea207e8eab774674a3a2a01b5b884f13a13202ceb16d8b83a5affb536"}
Oct 30 00:31:41.749485 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:41.749309    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-lngrr"
Oct 30 00:31:41.788911 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:41.788853    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-lngrr" podStartSLOduration=35.765090905 podStartE2EDuration="38.788840367s" totalImagesPullingTime="3.023749462s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:31:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:31:41.788525635 +0000 UTC m=+38.889832517" watchObservedRunningTime="2026-10-30 00:31:41.788840367 +0000 UTC m=+38.890147247"
Oct 30 00:31:42.754663 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:42.754628    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-g2dlc" event={"ID":"c4d423f3-2077-47fb-bf4d-d9991dd5eac1","Type":"ContainerStarted","Data":"fd76bebadb6c926e60304aa41ce92e5edcd966ad824a88e285c64c3c70404b26"}
Oct 30 00:31:42.756405 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:42.756355    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-khqcq" event={"ID":"928e22b3-c3fe-453b-bbc8-a67accd2fcb4","Type":"ContainerStarted","Data":"7d68c94ad418bf1a81ba6a3e810c0f0d450be65120a5ff5dc5dd7462e146c052"}
Oct 30 00:31:42.794225 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:42.794158    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-g2dlc" podStartSLOduration=1.867668071 podStartE2EDuration="3.794145373s" totalImagesPullingTime="1.926477302s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:31:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:31:42.792858808 +0000 UTC m=+39.894165688" watchObservedRunningTime="2026-10-30 00:31:42.794145373 +0000 UTC m=+39.895452252"
Oct 30 00:31:42.831589 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:42.831527    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-khqcq" podStartSLOduration=1.974253333 podStartE2EDuration="3.8315074s" totalImagesPullingTime="1.857254067s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:31:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:31:42.831049994 +0000 UTC m=+39.932356904" watchObservedRunningTime="2026-10-30 00:31:42.8315074 +0000 UTC m=+39.932814281"
Oct 30 00:31:43.063939 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:43.063864    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-6gjtz_078fd0e8-1393-45d0-baee-bbcd88095b2b/dns-node-resolver/0.log"
Oct 30 00:31:43.258100 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:43.258064    2579 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-g7c2w"
Oct 30 00:31:43.258282 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:43.258125    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-g7c2w"
Oct 30 00:31:43.258556 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:43.258540    2579 scope.go:122] "RemoveContainer" containerID="b9a214f6a9f96ff4dcd1880bfe7e3bbf691313502185f22d37d7989d8110023a"
Oct 30 00:31:43.258751 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:43.258731    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-g7c2w_openshift-console-operator(5dc56e7e-f3c1-43eb-a21b-8be6bb4706cf)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-g7c2w" podUID="5dc56e7e-f3c1-43eb-a21b-8be6bb4706cf"
Oct 30 00:31:43.402672 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:43.402586    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/94a6f5e3-d553-487e-a6f1-8c2b0fc5c695-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tbjdq\" (UID: \"94a6f5e3-d553-487e-a6f1-8c2b0fc5c695\") " pod="openshift-insights/insights-runtime-extractor-tbjdq"
Oct 30 00:31:43.402672 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:43.402665    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/94a6f5e3-d553-487e-a6f1-8c2b0fc5c695-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tbjdq\" (UID: \"94a6f5e3-d553-487e-a6f1-8c2b0fc5c695\") " pod="openshift-insights/insights-runtime-extractor-tbjdq"
Oct 30 00:31:43.403336 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:43.403311    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 00:31:43.403464 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:43.403358    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 00:31:43.403464 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:43.403405    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/94a6f5e3-d553-487e-a6f1-8c2b0fc5c695-kube-rbac-proxy-cm podName:94a6f5e3-d553-487e-a6f1-8c2b0fc5c695 nodeName:}" failed. No retries permitted until 2026-10-30 00:31:47.40336233 +0000 UTC m=+44.504669202 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/94a6f5e3-d553-487e-a6f1-8c2b0fc5c695-kube-rbac-proxy-cm") pod "insights-runtime-extractor-tbjdq" (UID: "94a6f5e3-d553-487e-a6f1-8c2b0fc5c695") : configmap "kube-rbac-proxy" not found
Oct 30 00:31:43.403585 ip-10-0-128-7 kubenswrapper[2579]: E1030 00:31:43.403470    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94a6f5e3-d553-487e-a6f1-8c2b0fc5c695-insights-runtime-extractor-tls podName:94a6f5e3-d553-487e-a6f1-8c2b0fc5c695 nodeName:}" failed. No retries permitted until 2026-10-30 00:31:47.403450036 +0000 UTC m=+44.504756913 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/94a6f5e3-d553-487e-a6f1-8c2b0fc5c695-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tbjdq" (UID: "94a6f5e3-d553-487e-a6f1-8c2b0fc5c695") : secret "insights-runtime-extractor-tls" not found
Oct 30 00:31:44.263924 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:44.263861    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-8b6d6_14285dfa-dd38-490e-a486-a1fcfc3d0214/node-ca/0.log"
Oct 30 00:31:44.281897 ip-10-0-128-7 kubenswrapper[2579]: I1030 00:31:44.281828    2579 dynamic_cafile_content.go:175] "Shutting down controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 00:31:44.281992 ip-10-0-128-7 systemd[1]: Stopping Kubernetes Kubelet...
Oct 30 00:31:44.296764 ip-10-0-128-7 systemd[1]: kubelet.service: Deactivated successfully.
Oct 30 00:31:44.296932 ip-10-0-128-7 systemd[1]: Stopped Kubernetes Kubelet.
Oct 30 00:31:44.296983 ip-10-0-128-7 systemd[1]: kubelet.service: Consumed 3.141s CPU time.
Oct 30 00:31:44.313057 ip-10-0-128-7 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 00:31:44.412480 ip-10-0-128-7 kubenswrapper[5002]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 00:31:44.412480 ip-10-0-128-7 kubenswrapper[5002]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 00:31:44.412480 ip-10-0-128-7 kubenswrapper[5002]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 00:31:44.412480 ip-10-0-128-7 kubenswrapper[5002]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 00:31:44.414519 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.414431    5002 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 00:31:44.414519 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.414519    5002 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 00:31:44.414519 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.414524    5002 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 00:31:44.414623 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.414527    5002 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 00:31:44.414623 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.414530    5002 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 00:31:44.414623 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.414533    5002 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 00:31:44.414623 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.414536    5002 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 00:31:44.414623 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.414543    5002 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 00:31:44.414623 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.414546    5002 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 00:31:44.414623 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.414549    5002 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 00:31:44.414623 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.414551    5002 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 00:31:44.414623 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.414554    5002 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 00:31:44.414623 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.414557    5002 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 00:31:44.414623 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.414560    5002 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 00:31:44.414623 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.414562    5002 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 00:31:44.414623 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.414565    5002 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 00:31:44.414623 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.414568    5002 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 00:31:44.414623 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.414570    5002 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 00:31:44.414623 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.414573    5002 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 00:31:44.414623 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.414575    5002 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 00:31:44.414623 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.414578    5002 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 00:31:44.414623 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.414581    5002 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 00:31:44.415098 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.414583    5002 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 00:31:44.415098 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.414586    5002 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 00:31:44.415098 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.414589    5002 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 00:31:44.415098 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.414591    5002 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 00:31:44.415098 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.414594    5002 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 00:31:44.415098 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.414597    5002 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 00:31:44.415098 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.414599    5002 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 00:31:44.415098 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.414602    5002 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 00:31:44.415098 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.414604    5002 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 00:31:44.415098 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.414607    5002 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 00:31:44.415098 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.414609    5002 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 00:31:44.415098 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.414613    5002 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 00:31:44.415098 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.414616    5002 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 00:31:44.415098 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.414618    5002 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 00:31:44.415098 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.414621    5002 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 00:31:44.415098 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.414624    5002 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 00:31:44.415098 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.414627    5002 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 00:31:44.415098 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.414630    5002 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 00:31:44.415098 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.414633    5002 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 00:31:44.415098 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.414636    5002 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 00:31:44.415653 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.414638    5002 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 00:31:44.415653 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.414641    5002 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 00:31:44.415653 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.414644    5002 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 00:31:44.415653 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.414646    5002 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 00:31:44.415653 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.414649    5002 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 00:31:44.415653 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.414651    5002 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 00:31:44.415653 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.414654    5002 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 00:31:44.415653 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.414656    5002 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 00:31:44.415653 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.414659    5002 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 00:31:44.415653 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.414662    5002 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 00:31:44.415653 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.414664    5002 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 00:31:44.415653 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.414667    5002 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 00:31:44.415653 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.414669    5002 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 00:31:44.415653 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.414672    5002 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 00:31:44.415653 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.414674    5002 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 00:31:44.415653 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.414677    5002 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 00:31:44.415653 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.414679    5002 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 00:31:44.415653 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.414682    5002 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 00:31:44.415653 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.414685    5002 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 00:31:44.416121 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.414688    5002 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 00:31:44.416121 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.414691    5002 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 00:31:44.416121 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.414693    5002 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 00:31:44.416121 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.414696    5002 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 00:31:44.416121 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.414700    5002 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 00:31:44.416121 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.414703    5002 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 00:31:44.416121 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.414706    5002 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 00:31:44.416121 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.414708    5002 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 00:31:44.416121 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.414711    5002 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 00:31:44.416121 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.414714    5002 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 00:31:44.416121 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.414717    5002 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 00:31:44.416121 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.414719    5002 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 00:31:44.416121 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.414722    5002 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 00:31:44.416121 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.414725    5002 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 00:31:44.416121 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.414727    5002 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 00:31:44.416121 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.414730    5002 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 00:31:44.416121 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.414732    5002 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 00:31:44.416121 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.414735    5002 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 00:31:44.416121 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.414738    5002 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 00:31:44.416121 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.414740    5002 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 00:31:44.416626 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.414743    5002 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 00:31:44.416626 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.414746    5002 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 00:31:44.416626 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.414748    5002 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 00:31:44.416626 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.414751    5002 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 00:31:44.416626 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.414754    5002 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 00:31:44.416626 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.414756    5002 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 00:31:44.416626 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.414759    5002 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 00:31:44.416626 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.414761    5002 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 00:31:44.416626 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.414764    5002 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 00:31:44.416626 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.414767    5002 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 00:31:44.416626 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.414771    5002 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 00:31:44.416626 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.414773    5002 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 00:31:44.416626 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.414778    5002 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 00:31:44.416626 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.414781    5002 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 00:31:44.416626 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.414785    5002 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 00:31:44.416626 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.414788    5002 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 00:31:44.416626 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.414791    5002 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 00:31:44.416626 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.414795    5002 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 00:31:44.416626 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.414799    5002 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 00:31:44.417078 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.414801    5002 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 00:31:44.417078 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.414804    5002 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 00:31:44.417078 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.414807    5002 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 00:31:44.417078 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.414810    5002 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 00:31:44.417078 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.414812    5002 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 00:31:44.417078 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.414815    5002 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 00:31:44.417078 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.415600    5002 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 00:31:44.417078 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.415609    5002 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 00:31:44.417078 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.415612    5002 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 00:31:44.417078 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.415615    5002 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 00:31:44.417078 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.415618    5002 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 00:31:44.417078 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.415620    5002 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 00:31:44.417078 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.415623    5002 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 00:31:44.417078 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.415626    5002 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 00:31:44.417078 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.415629    5002 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 00:31:44.417078 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.415632    5002 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 00:31:44.417078 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.415634    5002 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 00:31:44.417078 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.415637    5002 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 00:31:44.417078 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.415640    5002 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 00:31:44.417078 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.415643    5002 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 00:31:44.417610 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.415646    5002 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 00:31:44.417610 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.415649    5002 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 00:31:44.417610 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.415652    5002 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 00:31:44.417610 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.415654    5002 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 00:31:44.417610 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.415657    5002 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 00:31:44.417610 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.415660    5002 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 00:31:44.417610 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.415663    5002 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 00:31:44.417610 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.415666    5002 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 00:31:44.417610 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.415669    5002 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 00:31:44.417610 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.415672    5002 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 00:31:44.417610 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.415675    5002 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 00:31:44.417610 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.415677    5002 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 00:31:44.417610 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.415680    5002 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 00:31:44.417610 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.415683    5002 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 00:31:44.417610 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.415685    5002 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 00:31:44.417610 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.415688    5002 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 00:31:44.417610 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.415691    5002 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 00:31:44.417610 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.415695    5002 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 00:31:44.417610 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.415697    5002 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 00:31:44.418075 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.415700    5002 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 00:31:44.418075 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.415703    5002 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 00:31:44.418075 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.415705    5002 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 00:31:44.418075 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.415708    5002 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 00:31:44.418075 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.415711    5002 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 00:31:44.418075 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.415713    5002 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 00:31:44.418075 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.415716    5002 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 00:31:44.418075 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.415718    5002 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 00:31:44.418075 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.415721    5002 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 00:31:44.418075 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.415723    5002 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 00:31:44.418075 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.415726    5002 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 00:31:44.418075 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.415729    5002 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 00:31:44.418075 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.415731    5002 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 00:31:44.418075 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.415734    5002 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 00:31:44.418075 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.415737    5002 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 00:31:44.418075 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.415740    5002 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 00:31:44.418075 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.415742    5002 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 00:31:44.418075 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.415745    5002 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 00:31:44.418075 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.415747    5002 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 00:31:44.418575 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.415750    5002 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 00:31:44.418575 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.415753    5002 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 00:31:44.418575 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.415755    5002 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 00:31:44.418575 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.415758    5002 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 00:31:44.418575 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.415761    5002 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 00:31:44.418575 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.415763    5002 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 00:31:44.418575 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.415766    5002 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 00:31:44.418575 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.415768    5002 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 00:31:44.418575 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.415771    5002 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 00:31:44.418575 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.415773    5002 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 00:31:44.418575 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.415776    5002 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 00:31:44.418575 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.415780    5002 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 00:31:44.418575 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.415783    5002 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 00:31:44.418575 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.415785    5002 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 00:31:44.418575 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.415788    5002 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 00:31:44.418575 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.415791    5002 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 00:31:44.418575 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.415794    5002 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 00:31:44.418575 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.415797    5002 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 00:31:44.418575 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.415799    5002 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 00:31:44.418575 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.415802    5002 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 00:31:44.419065 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.415805    5002 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 00:31:44.419065 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.415807    5002 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 00:31:44.419065 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.415810    5002 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 00:31:44.419065 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.415812    5002 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 00:31:44.419065 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.415815    5002 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 00:31:44.419065 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.415817    5002 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 00:31:44.419065 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.415820    5002 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 00:31:44.419065 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.415822    5002 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 00:31:44.419065 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.415825    5002 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 00:31:44.419065 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.415827    5002 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 00:31:44.419065 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.415830    5002 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 00:31:44.419065 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.415832    5002 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 00:31:44.419065 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.415835    5002 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 00:31:44.419065 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.415838    5002 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 00:31:44.419065 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.415840    5002 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 00:31:44.419065 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.415843    5002 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 00:31:44.419065 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.415846    5002 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 00:31:44.419065 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.415848    5002 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 00:31:44.419065 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.415851    5002 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 00:31:44.419556 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.415853    5002 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 00:31:44.419556 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.415857    5002 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 00:31:44.419556 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.415859    5002 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 00:31:44.419556 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.415862    5002 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 00:31:44.419556 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.415864    5002 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 00:31:44.419556 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.415867    5002 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 00:31:44.419556 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.415871    5002 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 00:31:44.419556 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.415875    5002 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 00:31:44.419556 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.415878    5002 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 00:31:44.419556 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.415881    5002 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 00:31:44.419556 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.415884    5002 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 00:31:44.419556 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.415888    5002 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 00:31:44.419556 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.415892    5002 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 00:31:44.419556 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.415896    5002 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 00:31:44.419556 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.415902    5002 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 00:31:44.419556 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.416392    5002 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 00:31:44.419556 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.416397    5002 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 00:31:44.419556 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.416400    5002 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 00:31:44.419556 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.416403    5002 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 00:31:44.419556 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.416406    5002 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 00:31:44.420078 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.416408    5002 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 00:31:44.420078 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.416412    5002 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 00:31:44.420078 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.416415    5002 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 00:31:44.420078 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.416418    5002 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 00:31:44.420078 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.416420    5002 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 00:31:44.420078 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.416423    5002 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 00:31:44.420078 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.416425    5002 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 00:31:44.420078 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.416428    5002 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 00:31:44.420078 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.416431    5002 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 00:31:44.420078 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.416434    5002 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 00:31:44.420078 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.416437    5002 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 00:31:44.420078 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.416440    5002 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 00:31:44.420078 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.416443    5002 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 00:31:44.420078 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.416446    5002 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 00:31:44.420078 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.416448    5002 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 00:31:44.420078 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.416451    5002 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 00:31:44.420078 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.416454    5002 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 00:31:44.420078 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.416457    5002 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 00:31:44.420078 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.416459    5002 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 00:31:44.420078 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.416462    5002 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 00:31:44.420868 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.416465    5002 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 00:31:44.420868 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.416468    5002 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 00:31:44.420868 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.416470    5002 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 00:31:44.420868 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.416473    5002 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 00:31:44.420868 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.416475    5002 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 00:31:44.420868 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.416478    5002 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 00:31:44.420868 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.416481    5002 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 00:31:44.420868 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.416485    5002 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 00:31:44.420868 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.416488    5002 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 00:31:44.420868 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.416492    5002 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 00:31:44.420868 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.416495    5002 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 00:31:44.420868 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.416497    5002 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 00:31:44.420868 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.416500    5002 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 00:31:44.420868 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.416503    5002 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 00:31:44.420868 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.416506    5002 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 00:31:44.420868 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.416509    5002 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 00:31:44.420868 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.416511    5002 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 00:31:44.420868 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.416514    5002 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 00:31:44.420868 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.416517    5002 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 00:31:44.421834 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.416519    5002 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 00:31:44.421834 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.416522    5002 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 00:31:44.421834 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.416525    5002 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 00:31:44.421834 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.416527    5002 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 00:31:44.421834 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.416530    5002 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 00:31:44.421834 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.416532    5002 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 00:31:44.421834 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.416535    5002 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 00:31:44.421834 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.416537    5002 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 00:31:44.421834 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.416540    5002 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 00:31:44.421834 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.416542    5002 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 00:31:44.421834 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.416545    5002 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 00:31:44.421834 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.416548    5002 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 00:31:44.421834 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.416550    5002 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 00:31:44.421834 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.416556    5002 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 00:31:44.421834 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.416558    5002 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 00:31:44.421834 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.416561    5002 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 00:31:44.421834 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.416563    5002 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 00:31:44.421834 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.416566    5002 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 00:31:44.421834 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.416568    5002 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 00:31:44.422703 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.416571    5002 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 00:31:44.422703 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.416574    5002 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 00:31:44.422703 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.416576    5002 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 00:31:44.422703 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.416579    5002 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 00:31:44.422703 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.416581    5002 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 00:31:44.422703 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.416584    5002 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 00:31:44.422703 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.416586    5002 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 00:31:44.422703 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.416589    5002 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 00:31:44.422703 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.416592    5002 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 00:31:44.422703 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.416594    5002 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 00:31:44.422703 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.416597    5002 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 00:31:44.422703 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.416600    5002 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 00:31:44.422703 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.416602    5002 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 00:31:44.422703 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.416605    5002 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 00:31:44.422703 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.416607    5002 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 00:31:44.422703 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.416610    5002 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 00:31:44.422703 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.416612    5002 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 00:31:44.422703 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.416615    5002 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 00:31:44.422703 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.416618    5002 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 00:31:44.422703 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.416620    5002 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 00:31:44.423186 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.416623    5002 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 00:31:44.423186 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.416625    5002 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 00:31:44.423186 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.416628    5002 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 00:31:44.423186 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.416631    5002 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 00:31:44.423186 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.416633    5002 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 00:31:44.423186 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.416637    5002 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 00:31:44.423186 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.416644    5002 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 00:31:44.423186 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.416647    5002 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 00:31:44.423186 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.416650    5002 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 00:31:44.423186 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.416653    5002 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 00:31:44.423186 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.416655    5002 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 00:31:44.423186 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.416658    5002 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 00:31:44.423186 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.416661    5002 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 00:31:44.423186 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.416665    5002 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 00:31:44.423186 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.416669    5002 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 00:31:44.423186 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.416672    5002 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 00:31:44.423186 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.416679    5002 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 00:31:44.423186 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.416682    5002 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 00:31:44.423186 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.416685    5002 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 00:31:44.423691 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.416689    5002 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 00:31:44.423691 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.416692    5002 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 00:31:44.423691 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.416695    5002 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 00:31:44.423691 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.416698    5002 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 00:31:44.423691 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.416793    5002 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 00:31:44.423691 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.416814    5002 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 00:31:44.423691 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.416823    5002 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 00:31:44.423691 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.416828    5002 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 00:31:44.423691 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.416833    5002 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 00:31:44.423691 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.416836    5002 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 00:31:44.423691 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.416840    5002 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 00:31:44.423691 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.416845    5002 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 00:31:44.423691 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.416848    5002 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 00:31:44.423691 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.416851    5002 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 00:31:44.423691 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.416855    5002 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 00:31:44.423691 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.416858    5002 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 00:31:44.423691 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.416862    5002 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 00:31:44.423691 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.416865    5002 flags.go:64] FLAG: --cgroup-root=""
Oct 30 00:31:44.423691 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.416868    5002 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 00:31:44.423691 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.416874    5002 flags.go:64] FLAG: --client-ca-file=""
Oct 30 00:31:44.423691 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.416878    5002 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 00:31:44.423691 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.416881    5002 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 00:31:44.423691 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.416885    5002 flags.go:64] FLAG: --cluster-domain=""
Oct 30 00:31:44.424253 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.416888    5002 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 00:31:44.424253 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.416891    5002 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 00:31:44.424253 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.416894    5002 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 00:31:44.424253 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.416898    5002 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 00:31:44.424253 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.416902    5002 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 00:31:44.424253 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.416905    5002 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 00:31:44.424253 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.416910    5002 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 00:31:44.424253 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.416913    5002 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 00:31:44.424253 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.416916    5002 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 00:31:44.424253 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.416920    5002 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 00:31:44.424253 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.416923    5002 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 00:31:44.424253 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.416926    5002 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 00:31:44.424253 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.416929    5002 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 00:31:44.424253 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.416934    5002 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 00:31:44.424253 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.416937    5002 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 00:31:44.424253 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.416940    5002 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 00:31:44.424253 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.416943    5002 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 00:31:44.424253 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.416946    5002 flags.go:64] FLAG: --enable-server="true"
Oct 30 00:31:44.424253 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.416950    5002 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 00:31:44.424253 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.416954    5002 flags.go:64] FLAG: --event-burst="100"
Oct 30 00:31:44.424253 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.416958    5002 flags.go:64] FLAG: --event-qps="50"
Oct 30 00:31:44.424253 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.416961    5002 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 00:31:44.424253 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.416964    5002 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 00:31:44.424253 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.416967    5002 flags.go:64] FLAG: --eviction-hard=""
Oct 30 00:31:44.424848 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.416971    5002 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 00:31:44.424848 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.416974    5002 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 00:31:44.424848 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.416977    5002 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 00:31:44.424848 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.416980    5002 flags.go:64] FLAG: --eviction-soft=""
Oct 30 00:31:44.424848 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.416985    5002 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 00:31:44.424848 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.416989    5002 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 00:31:44.424848 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.416992    5002 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 00:31:44.424848 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.416995    5002 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 00:31:44.424848 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.416998    5002 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 00:31:44.424848 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.417001    5002 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 00:31:44.424848 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.417005    5002 flags.go:64] FLAG: --feature-gates=""
Oct 30 00:31:44.424848 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.417009    5002 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 00:31:44.424848 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.417012    5002 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 00:31:44.424848 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.417016    5002 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 00:31:44.424848 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.417020    5002 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 00:31:44.424848 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.417023    5002 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 00:31:44.424848 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.417026    5002 flags.go:64] FLAG: --help="false"
Oct 30 00:31:44.424848 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.417029    5002 flags.go:64] FLAG: --hostname-override="ip-10-0-128-7.ec2.internal"
Oct 30 00:31:44.424848 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.417032    5002 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 00:31:44.424848 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.417036    5002 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 00:31:44.424848 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.417039    5002 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 00:31:44.424848 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.417042    5002 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 00:31:44.424848 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.417045    5002 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 00:31:44.425463 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.417049    5002 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 00:31:44.425463 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.417052    5002 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 00:31:44.425463 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.417055    5002 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 00:31:44.425463 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.417058    5002 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 00:31:44.425463 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.417060    5002 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 00:31:44.425463 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.417064    5002 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 00:31:44.425463 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.417066    5002 flags.go:64] FLAG: --kube-reserved=""
Oct 30 00:31:44.425463 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.417069    5002 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 00:31:44.425463 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.417072    5002 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 00:31:44.425463 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.417076    5002 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 00:31:44.425463 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.417079    5002 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 00:31:44.425463 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.417082    5002 flags.go:64] FLAG: --lock-file=""
Oct 30 00:31:44.425463 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.417085    5002 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 00:31:44.425463 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.417090    5002 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 00:31:44.425463 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.417093    5002 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 00:31:44.425463 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.417100    5002 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 00:31:44.425463 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.417103    5002 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 00:31:44.425463 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.417106    5002 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 00:31:44.425463 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.417109    5002 flags.go:64] FLAG: --logging-format="text"
Oct 30 00:31:44.425463 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.417112    5002 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 00:31:44.425463 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.417116    5002 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 00:31:44.425463 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.417119    5002 flags.go:64] FLAG: --manifest-url=""
Oct 30 00:31:44.425463 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.417122    5002 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 00:31:44.425463 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.417129    5002 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 00:31:44.425463 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.417133    5002 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 00:31:44.426072 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.417136    5002 flags.go:64] FLAG: --max-pods="110"
Oct 30 00:31:44.426072 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.417139    5002 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 00:31:44.426072 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.417142    5002 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 00:31:44.426072 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.417145    5002 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 00:31:44.426072 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.417149    5002 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 00:31:44.426072 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.417152    5002 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 00:31:44.426072 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.417155    5002 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 00:31:44.426072 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.417158    5002 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 00:31:44.426072 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.417165    5002 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 00:31:44.426072 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.417168    5002 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 00:31:44.426072 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.417171    5002 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 00:31:44.426072 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.417174    5002 flags.go:64] FLAG: --pod-cidr=""
Oct 30 00:31:44.426072 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.417177    5002 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 00:31:44.426072 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.417180    5002 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 00:31:44.426072 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.417183    5002 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 00:31:44.426072 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.417186    5002 flags.go:64] FLAG: --port="10250"
Oct 30 00:31:44.426072 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.417190    5002 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 00:31:44.426072 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.417193    5002 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-090d92febc2f6edca"
Oct 30 00:31:44.426072 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.417196    5002 flags.go:64] FLAG: --qos-reserved=""
Oct 30 00:31:44.426072 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.417199    5002 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 00:31:44.426072 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.417203    5002 flags.go:64] FLAG: --register-node="true"
Oct 30 00:31:44.426072 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.417206    5002 flags.go:64] FLAG: --register-with-taints=""
Oct 30 00:31:44.426072 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.417210    5002 flags.go:64] FLAG: --registry-burst="10"
Oct 30 00:31:44.426072 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.417216    5002 flags.go:64] FLAG: --registry-qps="5"
Oct 30 00:31:44.426072 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.417220    5002 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 00:31:44.426072 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.417223    5002 flags.go:64] FLAG: --reserved-memory=""
Oct 30 00:31:44.426706 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.417227    5002 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 00:31:44.426706 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.417230    5002 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 00:31:44.426706 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.417233    5002 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 00:31:44.426706 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.417236    5002 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 00:31:44.426706 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.417240    5002 flags.go:64] FLAG: --runonce="false"
Oct 30 00:31:44.426706 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.417243    5002 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 00:31:44.426706 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.417247    5002 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 00:31:44.426706 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.417250    5002 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 00:31:44.426706 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.417253    5002 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 00:31:44.426706 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.417256    5002 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 00:31:44.426706 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.417259    5002 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 00:31:44.426706 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.417262    5002 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 00:31:44.426706 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.417265    5002 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 00:31:44.426706 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.417268    5002 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 00:31:44.426706 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.417271    5002 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 00:31:44.426706 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.417274    5002 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 00:31:44.426706 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.417277    5002 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 00:31:44.426706 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.417280    5002 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 00:31:44.426706 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.417283    5002 flags.go:64] FLAG: --system-cgroups=""
Oct 30 00:31:44.426706 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.417286    5002 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 00:31:44.426706 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.417292    5002 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 00:31:44.426706 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.417295    5002 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 00:31:44.426706 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.417297    5002 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 00:31:44.426706 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.417301    5002 flags.go:64] FLAG: --tls-min-version=""
Oct 30 00:31:44.426706 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.417304    5002 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 00:31:44.427348 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.417307    5002 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 00:31:44.427348 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.417311    5002 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 00:31:44.427348 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.417314    5002 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 00:31:44.427348 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.417318    5002 flags.go:64] FLAG: --v="2"
Oct 30 00:31:44.427348 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.417322    5002 flags.go:64] FLAG: --version="false"
Oct 30 00:31:44.427348 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.417327    5002 flags.go:64] FLAG: --vmodule=""
Oct 30 00:31:44.427348 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.417331    5002 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 00:31:44.427348 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.417334    5002 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 00:31:44.427348 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.417478    5002 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 00:31:44.427348 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.417483    5002 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 00:31:44.427348 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.417486    5002 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 00:31:44.427348 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.417490    5002 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 00:31:44.427348 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.417493    5002 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 00:31:44.427348 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.417496    5002 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 00:31:44.427348 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.417499    5002 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 00:31:44.427348 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.417501    5002 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 00:31:44.427348 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.417504    5002 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 00:31:44.427348 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.417506    5002 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 00:31:44.427348 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.417509    5002 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 00:31:44.427348 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.417512    5002 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 00:31:44.427348 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.417514    5002 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 00:31:44.427894 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.417517    5002 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 00:31:44.427894 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.417520    5002 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 00:31:44.427894 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.417524    5002 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 00:31:44.427894 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.417527    5002 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 00:31:44.427894 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.417530    5002 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 00:31:44.427894 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.417533    5002 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 00:31:44.427894 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.417536    5002 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 00:31:44.427894 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.417539    5002 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 00:31:44.427894 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.417541    5002 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 00:31:44.427894 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.417544    5002 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 00:31:44.427894 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.417546    5002 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 00:31:44.427894 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.417549    5002 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 00:31:44.427894 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.417553    5002 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 00:31:44.427894 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.417556    5002 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 00:31:44.427894 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.417558    5002 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 00:31:44.427894 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.417561    5002 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 00:31:44.427894 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.417564    5002 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 00:31:44.427894 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.417567    5002 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 00:31:44.427894 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.417569    5002 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 00:31:44.428367 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.417572    5002 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 00:31:44.428367 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.417575    5002 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 00:31:44.428367 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.417577    5002 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 00:31:44.428367 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.417581    5002 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 00:31:44.428367 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.417584    5002 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 00:31:44.428367 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.417586    5002 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 00:31:44.428367 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.417589    5002 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 00:31:44.428367 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.417592    5002 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 00:31:44.428367 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.417594    5002 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 00:31:44.428367 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.417597    5002 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 00:31:44.428367 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.417599    5002 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 00:31:44.428367 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.417603    5002 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 00:31:44.428367 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.417605    5002 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 00:31:44.428367 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.417608    5002 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 00:31:44.428367 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.417611    5002 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 00:31:44.428367 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.417614    5002 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 00:31:44.428367 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.417616    5002 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 00:31:44.428367 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.417619    5002 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 00:31:44.428367 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.417622    5002 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 00:31:44.428367 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.417625    5002 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 00:31:44.428872 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.417627    5002 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 00:31:44.428872 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.417630    5002 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 00:31:44.428872 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.417633    5002 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 00:31:44.428872 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.417635    5002 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 00:31:44.428872 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.417639    5002 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 00:31:44.428872 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.417643    5002 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 00:31:44.428872 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.417646    5002 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 00:31:44.428872 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.417648    5002 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 00:31:44.428872 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.417651    5002 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 00:31:44.428872 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.417654    5002 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 00:31:44.428872 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.417656    5002 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 00:31:44.428872 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.417659    5002 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 00:31:44.428872 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.417662    5002 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 00:31:44.428872 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.417665    5002 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 00:31:44.428872 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.417668    5002 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 00:31:44.428872 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.417671    5002 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 00:31:44.428872 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.417674    5002 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 00:31:44.428872 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.417677    5002 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 00:31:44.428872 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.417679    5002 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 00:31:44.428872 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.417682    5002 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 00:31:44.429625 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.417685    5002 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 00:31:44.429625 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.417687    5002 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 00:31:44.429625 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.417690    5002 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 00:31:44.429625 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.417693    5002 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 00:31:44.429625 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.417696    5002 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 00:31:44.429625 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.417698    5002 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 00:31:44.429625 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.417701    5002 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 00:31:44.429625 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.417703    5002 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 00:31:44.429625 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.417706    5002 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 00:31:44.429625 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.417709    5002 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 00:31:44.429625 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.417711    5002 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 00:31:44.429625 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.417714    5002 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 00:31:44.429625 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.417716    5002 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 00:31:44.429625 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.417719    5002 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 00:31:44.429625 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.417721    5002 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 00:31:44.429625 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.417724    5002 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 00:31:44.429625 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.417727    5002 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 00:31:44.429625 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.417731    5002 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 00:31:44.429625 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.417733    5002 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 00:31:44.430087 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.417736    5002 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 00:31:44.430087 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.417738    5002 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 00:31:44.430087 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.417741    5002 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 00:31:44.430087 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.417745    5002 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 00:31:44.430087 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.417749    5002 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 00:31:44.430087 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.417752    5002 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 00:31:44.430087 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.417755    5002 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 00:31:44.430087 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.417758    5002 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 00:31:44.430087 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.417762    5002 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 00:31:44.430087 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.417765    5002 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 00:31:44.430087 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.417768    5002 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 00:31:44.430087 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.417771    5002 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 00:31:44.430087 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.417774    5002 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 00:31:44.430087 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.417777    5002 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 00:31:44.430087 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.417779    5002 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 00:31:44.430738 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.417804    5002 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 00:31:44.430738 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.417840    5002 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 00:31:44.430738 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.417844    5002 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 00:31:44.430738 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.417847    5002 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 00:31:44.430738 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.417850    5002 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 00:31:44.430738 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.417853    5002 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 00:31:44.430738 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.417855    5002 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 00:31:44.430738 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.417858    5002 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 00:31:44.430738 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.417861    5002 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 00:31:44.430738 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.417864    5002 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 00:31:44.430738 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.417867    5002 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 00:31:44.430738 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.417869    5002 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 00:31:44.430738 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.417872    5002 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 00:31:44.430738 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.417874    5002 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 00:31:44.430738 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.417877    5002 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 00:31:44.430738 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.417882    5002 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 00:31:44.431496 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.417884    5002 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 00:31:44.431496 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.417887    5002 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 00:31:44.431496 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.417889    5002 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 00:31:44.431496 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.417892    5002 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 00:31:44.431496 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.417895    5002 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 00:31:44.431496 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.417898    5002 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 00:31:44.431496 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.417900    5002 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 00:31:44.431496 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.417903    5002 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 00:31:44.431496 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.417906    5002 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 00:31:44.431496 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.417909    5002 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 00:31:44.431496 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.417913    5002 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 00:31:44.431496 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.417916    5002 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 00:31:44.431496 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.417920    5002 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 00:31:44.431496 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.417923    5002 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 00:31:44.431496 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.417926    5002 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 00:31:44.431496 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.417929    5002 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 00:31:44.431496 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.417931    5002 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 00:31:44.431496 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.417934    5002 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 00:31:44.431496 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.417937    5002 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 00:31:44.432119 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.417939    5002 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 00:31:44.432119 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.417942    5002 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 00:31:44.432119 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.417945    5002 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 00:31:44.432119 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.417947    5002 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 00:31:44.432119 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.417950    5002 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 00:31:44.432119 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.417952    5002 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 00:31:44.432119 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.417955    5002 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 00:31:44.432119 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.417958    5002 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 00:31:44.432119 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.417960    5002 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 00:31:44.432119 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.417963    5002 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 00:31:44.432119 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.417965    5002 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 00:31:44.432119 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.417968    5002 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 00:31:44.432119 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.417972    5002 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 00:31:44.432119 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.417975    5002 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 00:31:44.432119 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.417978    5002 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 00:31:44.432119 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.417980    5002 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 00:31:44.432119 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.417983    5002 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 00:31:44.432119 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.417986    5002 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 00:31:44.432119 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.417988    5002 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 00:31:44.432681 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.417992    5002 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 00:31:44.432681 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.417996    5002 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 00:31:44.432681 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.417999    5002 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 00:31:44.432681 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.418002    5002 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 00:31:44.432681 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.418006    5002 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 00:31:44.432681 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.418008    5002 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 00:31:44.432681 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.418011    5002 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 00:31:44.432681 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.418013    5002 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 00:31:44.432681 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.418016    5002 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 00:31:44.432681 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.418019    5002 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 00:31:44.432681 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.418021    5002 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 00:31:44.432681 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.418024    5002 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 00:31:44.432681 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.418026    5002 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 00:31:44.432681 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.418029    5002 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 00:31:44.432681 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.418032    5002 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 00:31:44.432681 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.418035    5002 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 00:31:44.432681 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.418048    5002 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 00:31:44.432681 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.418050    5002 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 00:31:44.432681 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.418053    5002 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 00:31:44.433229 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.418056    5002 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 00:31:44.433229 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.418059    5002 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 00:31:44.433229 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.418061    5002 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 00:31:44.433229 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.418064    5002 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 00:31:44.433229 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.418067    5002 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 00:31:44.433229 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.418070    5002 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 00:31:44.433229 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.418073    5002 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 00:31:44.433229 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.418076    5002 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 00:31:44.433229 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.418079    5002 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 00:31:44.433229 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.418082    5002 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 00:31:44.433229 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.418085    5002 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 00:31:44.433229 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.418088    5002 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 00:31:44.433229 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.418092    5002 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 00:31:44.433229 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.418094    5002 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 00:31:44.433229 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.418097    5002 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 00:31:44.433229 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.418099    5002 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 00:31:44.433229 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.418102    5002 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 00:31:44.433229 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.418107    5002 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 00:31:44.433229 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.418110    5002 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 00:31:44.433229 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.418112    5002 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 00:31:44.433743 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.418115    5002 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 00:31:44.433743 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.418118    5002 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 00:31:44.433743 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.418120    5002 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 00:31:44.433743 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.418123    5002 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 00:31:44.433743 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.418126    5002 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 00:31:44.433743 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.418128    5002 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 00:31:44.433743 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.418131    5002 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 00:31:44.433743 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.418134    5002 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 00:31:44.433743 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.418136    5002 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 00:31:44.433743 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.418139    5002 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 00:31:44.433743 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.418143    5002 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 00:31:44.433743 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.418145    5002 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 00:31:44.433743 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.418148    5002 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 00:31:44.433743 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.418151    5002 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 00:31:44.433743 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.423893    5002 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 00:31:44.433743 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.423912    5002 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 00:31:44.433743 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.423940    5002 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 00:31:44.433743 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.423946    5002 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 00:31:44.433743 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424023    5002 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 00:31:44.433743 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424028    5002 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 00:31:44.434265 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424032    5002 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 00:31:44.434265 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424035    5002 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 00:31:44.434265 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424038    5002 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 00:31:44.434265 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424041    5002 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 00:31:44.434265 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424043    5002 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 00:31:44.434265 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424046    5002 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 00:31:44.434265 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424049    5002 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 00:31:44.434265 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424052    5002 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 00:31:44.434265 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424054    5002 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 00:31:44.434265 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424058    5002 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 00:31:44.434265 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424061    5002 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 00:31:44.434265 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424064    5002 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 00:31:44.434265 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424067    5002 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 00:31:44.434265 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424070    5002 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 00:31:44.434265 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424072    5002 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 00:31:44.434265 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424075    5002 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 00:31:44.434265 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424078    5002 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 00:31:44.434265 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424080    5002 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 00:31:44.434265 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424083    5002 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 00:31:44.434741 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424085    5002 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 00:31:44.434741 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424088    5002 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 00:31:44.434741 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424090    5002 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 00:31:44.434741 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424093    5002 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 00:31:44.434741 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424096    5002 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 00:31:44.434741 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424099    5002 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 00:31:44.434741 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424101    5002 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 00:31:44.434741 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424104    5002 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 00:31:44.434741 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424106    5002 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 00:31:44.434741 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424109    5002 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 00:31:44.434741 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424112    5002 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 00:31:44.434741 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424116    5002 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 00:31:44.434741 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424120    5002 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 00:31:44.434741 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424124    5002 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 00:31:44.434741 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424128    5002 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 00:31:44.434741 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424132    5002 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 00:31:44.434741 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424136    5002 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 00:31:44.434741 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424140    5002 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 00:31:44.434741 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424143    5002 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 00:31:44.435195 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424149    5002 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 00:31:44.435195 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424156    5002 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 00:31:44.435195 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424161    5002 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 00:31:44.435195 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424165    5002 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 00:31:44.435195 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424170    5002 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 00:31:44.435195 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424176    5002 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 00:31:44.435195 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424181    5002 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 00:31:44.435195 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424186    5002 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 00:31:44.435195 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424190    5002 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 00:31:44.435195 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424194    5002 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 00:31:44.435195 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424198    5002 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 00:31:44.435195 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424202    5002 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 00:31:44.435195 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424206    5002 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 00:31:44.435195 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424210    5002 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 00:31:44.435195 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424214    5002 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 00:31:44.435195 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424218    5002 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 00:31:44.435195 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424223    5002 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 00:31:44.435195 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424226    5002 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 00:31:44.435195 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424231    5002 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 00:31:44.435750 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424235    5002 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 00:31:44.435750 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424239    5002 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 00:31:44.435750 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424243    5002 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 00:31:44.435750 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424247    5002 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 00:31:44.435750 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424251    5002 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 00:31:44.435750 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424255    5002 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 00:31:44.435750 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424260    5002 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 00:31:44.435750 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424264    5002 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 00:31:44.435750 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424268    5002 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 00:31:44.435750 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424272    5002 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 00:31:44.435750 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424276    5002 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 00:31:44.435750 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424280    5002 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 00:31:44.435750 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424284    5002 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 00:31:44.435750 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424288    5002 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 00:31:44.435750 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424291    5002 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 00:31:44.435750 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424296    5002 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 00:31:44.435750 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424300    5002 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 00:31:44.435750 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424304    5002 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 00:31:44.435750 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424307    5002 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 00:31:44.435750 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424310    5002 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 00:31:44.436576 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424314    5002 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 00:31:44.436576 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424317    5002 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 00:31:44.436576 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424320    5002 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 00:31:44.436576 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424323    5002 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 00:31:44.436576 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424325    5002 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 00:31:44.436576 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424328    5002 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 00:31:44.436576 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424331    5002 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 00:31:44.436576 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424334    5002 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 00:31:44.436576 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424336    5002 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 00:31:44.436576 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424339    5002 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 00:31:44.436576 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424342    5002 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 00:31:44.436576 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424344    5002 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 00:31:44.436576 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424347    5002 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 00:31:44.436576 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424349    5002 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 00:31:44.436576 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424352    5002 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 00:31:44.436576 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424355    5002 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 00:31:44.436576 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424357    5002 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 00:31:44.436576 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424361    5002 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 00:31:44.436576 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424364    5002 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 00:31:44.437258 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424367    5002 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 00:31:44.437258 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424370    5002 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 00:31:44.437258 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424373    5002 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 00:31:44.437258 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424376    5002 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 00:31:44.437258 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424394    5002 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 00:31:44.437258 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424397    5002 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 00:31:44.437258 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424400    5002 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 00:31:44.437258 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424403    5002 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 00:31:44.437258 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.424433    5002 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 00:31:44.437258 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424582    5002 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 00:31:44.437258 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424587    5002 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 00:31:44.437258 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424590    5002 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 00:31:44.437258 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424593    5002 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 00:31:44.437258 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424596    5002 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 00:31:44.437258 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424600    5002 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 00:31:44.437258 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424603    5002 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 00:31:44.437687 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424605    5002 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 00:31:44.437687 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424608    5002 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 00:31:44.437687 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424611    5002 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 00:31:44.437687 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424614    5002 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 00:31:44.437687 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424617    5002 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 00:31:44.437687 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424619    5002 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 00:31:44.437687 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424622    5002 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 00:31:44.437687 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424625    5002 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 00:31:44.437687 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424628    5002 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 00:31:44.437687 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424632    5002 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 00:31:44.437687 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424636    5002 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 00:31:44.437687 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424639    5002 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 00:31:44.437687 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424642    5002 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 00:31:44.437687 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424645    5002 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 00:31:44.437687 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424648    5002 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 00:31:44.437687 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424651    5002 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 00:31:44.437687 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424653    5002 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 00:31:44.437687 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424656    5002 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 00:31:44.437687 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424659    5002 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 00:31:44.438187 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424661    5002 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 00:31:44.438187 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424664    5002 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 00:31:44.438187 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424666    5002 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 00:31:44.438187 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424669    5002 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 00:31:44.438187 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424671    5002 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 00:31:44.438187 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424674    5002 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 00:31:44.438187 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424677    5002 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 00:31:44.438187 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424679    5002 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 00:31:44.438187 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424682    5002 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 00:31:44.438187 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424685    5002 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 00:31:44.438187 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424687    5002 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 00:31:44.438187 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424690    5002 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 00:31:44.438187 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424696    5002 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 00:31:44.438187 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424699    5002 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 00:31:44.438187 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424702    5002 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 00:31:44.438187 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424704    5002 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 00:31:44.438187 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424707    5002 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 00:31:44.438187 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424709    5002 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 00:31:44.438187 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424712    5002 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 00:31:44.438684 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424714    5002 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 00:31:44.438684 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424717    5002 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 00:31:44.438684 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424720    5002 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 00:31:44.438684 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424722    5002 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 00:31:44.438684 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424725    5002 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 00:31:44.438684 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424727    5002 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 00:31:44.438684 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424745    5002 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 00:31:44.438684 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424749    5002 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 00:31:44.438684 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424753    5002 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 00:31:44.438684 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424756    5002 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 00:31:44.438684 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424759    5002 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 00:31:44.438684 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424762    5002 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 00:31:44.438684 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424765    5002 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 00:31:44.438684 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424767    5002 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 00:31:44.438684 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424770    5002 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 00:31:44.438684 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424772    5002 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 00:31:44.438684 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424775    5002 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 00:31:44.438684 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424778    5002 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 00:31:44.438684 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424780    5002 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 00:31:44.438684 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424783    5002 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 00:31:44.439271 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424785    5002 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 00:31:44.439271 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424788    5002 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 00:31:44.439271 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424791    5002 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 00:31:44.439271 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424794    5002 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 00:31:44.439271 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424796    5002 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 00:31:44.439271 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424799    5002 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 00:31:44.439271 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424804    5002 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 00:31:44.439271 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424807    5002 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 00:31:44.439271 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424810    5002 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 00:31:44.439271 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424813    5002 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 00:31:44.439271 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424816    5002 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 00:31:44.439271 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424818    5002 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 00:31:44.439271 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424821    5002 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 00:31:44.439271 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424824    5002 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 00:31:44.439271 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424826    5002 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 00:31:44.439271 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424829    5002 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 00:31:44.439271 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424831    5002 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 00:31:44.439271 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424834    5002 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 00:31:44.439271 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424837    5002 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 00:31:44.439760 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424840    5002 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 00:31:44.439760 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424842    5002 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 00:31:44.439760 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424845    5002 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 00:31:44.439760 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424847    5002 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 00:31:44.439760 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424850    5002 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 00:31:44.439760 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424853    5002 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 00:31:44.439760 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424856    5002 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 00:31:44.439760 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424858    5002 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 00:31:44.439760 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424861    5002 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 00:31:44.439760 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424864    5002 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 00:31:44.439760 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424867    5002 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 00:31:44.439760 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424869    5002 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 00:31:44.439760 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424872    5002 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 00:31:44.439760 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424876    5002 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 00:31:44.439760 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424880    5002 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 00:31:44.439760 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424882    5002 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 00:31:44.439760 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424885    5002 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 00:31:44.439760 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424887    5002 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 00:31:44.439760 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424890    5002 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 00:31:44.439760 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424893    5002 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 00:31:44.440287 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424896    5002 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 00:31:44.440287 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424898    5002 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 00:31:44.440287 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.424922    5002 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 00:31:44.440287 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424954    5002 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 00:31:44.440287 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424957    5002 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 00:31:44.440287 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424960    5002 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 00:31:44.440287 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424963    5002 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 00:31:44.440287 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424966    5002 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 00:31:44.440287 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424969    5002 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 00:31:44.440287 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424971    5002 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 00:31:44.440287 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424974    5002 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 00:31:44.440287 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424977    5002 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 00:31:44.440287 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424979    5002 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 00:31:44.440287 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424982    5002 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 00:31:44.440287 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424985    5002 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 00:31:44.440287 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424987    5002 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 00:31:44.440698 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424990    5002 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 00:31:44.440698 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424992    5002 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 00:31:44.440698 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424995    5002 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 00:31:44.440698 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.424998    5002 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 00:31:44.440698 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.425002    5002 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 00:31:44.440698 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.425005    5002 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 00:31:44.440698 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.425008    5002 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 00:31:44.440698 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.425011    5002 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 00:31:44.440698 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.425013    5002 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 00:31:44.440698 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.425021    5002 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 00:31:44.440698 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.425023    5002 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 00:31:44.440698 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.425026    5002 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 00:31:44.440698 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.425029    5002 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 00:31:44.440698 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.425033    5002 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 00:31:44.440698 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.425037    5002 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 00:31:44.440698 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.425041    5002 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 00:31:44.440698 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.425045    5002 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 00:31:44.440698 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.425049    5002 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 00:31:44.440698 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.425052    5002 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 00:31:44.441167 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.425055    5002 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 00:31:44.441167 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.425057    5002 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 00:31:44.441167 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.425060    5002 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 00:31:44.441167 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.425063    5002 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 00:31:44.441167 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.425066    5002 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 00:31:44.441167 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.425068    5002 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 00:31:44.441167 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.425071    5002 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 00:31:44.441167 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.425074    5002 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 00:31:44.441167 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.425076    5002 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 00:31:44.441167 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.425079    5002 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 00:31:44.441167 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.425082    5002 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 00:31:44.441167 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.425084    5002 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 00:31:44.441167 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.425087    5002 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 00:31:44.441167 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.425089    5002 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 00:31:44.441167 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.425092    5002 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 00:31:44.441167 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.425095    5002 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 00:31:44.441167 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.425097    5002 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 00:31:44.441167 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.425100    5002 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 00:31:44.441167 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.425102    5002 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 00:31:44.441743 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.425105    5002 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 00:31:44.441743 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.425108    5002 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 00:31:44.441743 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.425110    5002 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 00:31:44.441743 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.425116    5002 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 00:31:44.441743 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.425119    5002 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 00:31:44.441743 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.425121    5002 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 00:31:44.441743 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.425124    5002 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 00:31:44.441743 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.425127    5002 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 00:31:44.441743 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.425130    5002 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 00:31:44.441743 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.425132    5002 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 00:31:44.441743 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.425135    5002 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 00:31:44.441743 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.425138    5002 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 00:31:44.441743 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.425141    5002 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 00:31:44.441743 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.425143    5002 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 00:31:44.441743 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.425146    5002 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 00:31:44.441743 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.425149    5002 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 00:31:44.441743 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.425151    5002 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 00:31:44.441743 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.425154    5002 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 00:31:44.441743 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.425157    5002 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 00:31:44.441743 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.425160    5002 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 00:31:44.442234 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.425163    5002 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 00:31:44.442234 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.425167    5002 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 00:31:44.442234 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.425170    5002 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 00:31:44.442234 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.425173    5002 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 00:31:44.442234 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.425176    5002 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 00:31:44.442234 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.425179    5002 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 00:31:44.442234 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.425181    5002 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 00:31:44.442234 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.425184    5002 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 00:31:44.442234 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.425186    5002 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 00:31:44.442234 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.425189    5002 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 00:31:44.442234 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.425191    5002 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 00:31:44.442234 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.425194    5002 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 00:31:44.442234 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.425196    5002 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 00:31:44.442234 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.425199    5002 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 00:31:44.442234 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.425202    5002 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 00:31:44.442234 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.425208    5002 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 00:31:44.442234 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.425211    5002 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 00:31:44.442234 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.425213    5002 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 00:31:44.442234 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.425216    5002 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 00:31:44.442714 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.425219    5002 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 00:31:44.442714 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.425221    5002 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 00:31:44.442714 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.425224    5002 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 00:31:44.442714 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.425227    5002 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 00:31:44.442714 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.425230    5002 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 00:31:44.442714 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.425233    5002 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 00:31:44.442714 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.425237    5002 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 00:31:44.442714 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.425239    5002 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 00:31:44.442714 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.425242    5002 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 00:31:44.442714 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.425245    5002 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 00:31:44.442714 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.425247    5002 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 00:31:44.442714 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.425250    5002 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 00:31:44.442714 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.425253    5002 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 00:31:44.442714 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.425255    5002 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 00:31:44.442714 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.425258    5002 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 00:31:44.442714 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:44.425261    5002 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 00:31:44.442714 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.425319    5002 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 00:31:44.442714 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.426287    5002 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Oct 30 00:31:44.442714 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.426348    5002 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Oct 30 00:31:44.443172 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.426557    5002 server.go:1014] "Starting client certificate rotation"
Oct 30 00:31:44.443172 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.426693    5002 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 00:31:44.443172 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.426745    5002 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 00:26:03 +0000 UTC" deadline="2028-05-18 04:15:05.99569131 +0000 UTC"
Oct 30 00:31:44.443172 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.426781    5002 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13587h43m21.56891271s"
Oct 30 00:31:44.443172 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.427012    5002 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 00:31:44.443172 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.428158    5002 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 00:31:44.443172 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.429539    5002 log.go:25] "Validated CRI v1 runtime API"
Oct 30 00:31:44.443172 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.431719    5002 log.go:25] "Validated CRI v1 image API"
Oct 30 00:31:44.443172 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.432495    5002 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 00:31:44.443172 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.434695    5002 fs.go:135] Filesystem UUIDs: map[4444e55e-7462-4627-8c3d-fd7f95cef134:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 89f2892b-758d-4770-80e8-4d7019907450:/dev/nvme0n1p4]
Oct 30 00:31:44.443484 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.434718    5002 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0a1b48da841c11cc03ad3c6dd62932dd0b7de30c4e34fc221f366ec25b3a9ee5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0a1b48da841c11cc03ad3c6dd62932dd0b7de30c4e34fc221f366ec25b3a9ee5/userdata/shm major:0 minor:219 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/208de9f5a67ade098d715f55f9e24682f2698180040d35b44971ed390bfe1312/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/208de9f5a67ade098d715f55f9e24682f2698180040d35b44971ed390bfe1312/userdata/shm major:0 minor:338 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3e0825f9e3c3deaa21e8cceb37a1971d348f6672ea7e4ccf29e29caaf09b6cef/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3e0825f9e3c3deaa21e8cceb37a1971d348f6672ea7e4ccf29e29caaf09b6cef/userdata/shm major:0 minor:336 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/43e781b035175dffab13f8dbbfd8ba443ef3114ebef4867715bd8514bcd1217e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/43e781b035175dffab13f8dbbfd8ba443ef3114ebef4867715bd8514bcd1217e/userdata/shm major:0 minor:221 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/453fcdc0a3bdbd70d0b6a86ba356202ebd1abd1c2e08b95bc5fd8533b3550ac8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/453fcdc0a3bdbd70d0b6a86ba356202ebd1abd1c2e08b95bc5fd8533b3550ac8/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/57c013576400090006215c678b7fcc105a57fbc3a4f73eb482fe1d1da5fcf302/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/57c013576400090006215c678b7fcc105a57fbc3a4f73eb482fe1d1da5fcf302/userdata/shm major:0 minor:298 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/650dac582e52282d5b674397229aa50ac75a19dc3768efb9eb967a5f252fc938/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/650dac582e52282d5b674397229aa50ac75a19dc3768efb9eb967a5f252fc938/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/70a7f9743975c1002c0feb8db36a717b4dc312d0b220be0a3b8dfca5e7122ebd/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/70a7f9743975c1002c0feb8db36a717b4dc312d0b220be0a3b8dfca5e7122ebd/userdata/shm major:0 minor:246 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7e631fdaf2b86d82764221a535cde6bc365d8253137f5469f9370b342ecc6c6d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7e631fdaf2b86d82764221a535cde6bc365d8253137f5469f9370b342ecc6c6d/userdata/shm major:0 minor:242 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/83e0c503316f6412a51e1550577df70d9ffcbe1c49b03e2cbec69f56d9cbce9e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/83e0c503316f6412a51e1550577df70d9ffcbe1c49b03e2cbec69f56d9cbce9e/userdata/shm major:0 minor:95 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/94e5870821441f01b75fd1499c09731d42f372db8af82d71e594850adef78e99/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/94e5870821441f01b75fd1499c09731d42f372db8af82d71e594850adef78e99/userdata/shm major:0 minor:88 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a342adc28b63c1d46dd71a11bcf85d3d52a05101220c43ef41724801c48c9fa4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a342adc28b63c1d46dd71a11bcf85d3d52a05101220c43ef41724801c48c9fa4/userdata/shm major:0 minor:228 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b1739fe883553d159fc4a3b364b09e5ac408869eefc63b025dccf4c5273b7f8b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b1739fe883553d159fc4a3b364b09e5ac408869eefc63b025dccf4c5273b7f8b/userdata/shm major:0 minor:78 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b984324d5c7f9d7f10d6b87625aafb838e43218eaa1db56a88123a6411f19142/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b984324d5c7f9d7f10d6b87625aafb838e43218eaa1db56a88123a6411f19142/userdata/shm major:0 minor:340 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ba10707cdb72fd5b7212466e2efb1d9bf1fe0a9077139125e96f32d1d0295515/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ba10707cdb72fd5b7212466e2efb1d9bf1fe0a9077139125e96f32d1d0295515/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bb5c2326fc6156453ddd2fd1052eb32784198ed45e9d6f91e0b3e693c64e57ef/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bb5c2326fc6156453ddd2fd1052eb32784198ed45e9d6f91e0b3e693c64e57ef/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c0ca8a0286da3206a1984da7ca07bcd8f48346aeb32382aa1ac127f0b959599f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c0ca8a0286da3206a1984da7ca07bcd8f48346aeb32382aa1ac127f0b959599f/userdata/shm major:0 minor:237 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c401bfa9817c86ba3e9537e9932491b450d77a68e1d6d855d93aed8ae46c3445/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c401bfa9817c86ba3e9537e9932491b450d77a68e1d6d855d93aed8ae46c3445/userdata/shm major:0 minor:211 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c42e0453a3ed59a2dec4c08c3a2acc4b6c3a846e1abb8dc1b4db077d05741d0c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c42e0453a3ed59a2dec4c08c3a2acc4b6c3a846e1abb8dc1b4db077d05741d0c/userdata/shm major:0 minor:230 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c72433bacc952d570b3676b773c9b174fec945357ecf73c909e453c4a02d6577/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c72433bacc952d570b3676b773c9b174fec945357ecf73c909e453c4a02d6577/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cc9244c17a2ca2d72c4199cc0781aaa7e22b7859adfb3b8c52ee0fb4348e35ef/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cc9244c17a2ca2d72c4199cc0781aaa7e22b7859adfb3b8c52ee0fb4348e35ef/userdata/shm major:0 minor:243 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d4b5066e0524802f2cab4f8c5119b9785e3cd5bbdac5d0b5fd0b350f47f5baaa/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d4b5066e0524802f2cab4f8c5119b9785e3cd5bbdac5d0b5fd0b350f47f5baaa/userdata/shm major:0 minor:293 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ea478e19d2f09e1d112adbc723346aa5cdea945578e9f53cf3b2cb2fda738476/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ea478e19d2f09e1d112adbc723346aa5cdea945578e9f53cf3b2cb2fda738476/userdata/shm major:0 minor:74 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f43de3eccd69933959e622c93a59fc85020fb8a8b6088a8cfc2cad3cccb1eff0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f43de3eccd69933959e622c93a59fc85020fb8a8b6088a8cfc2cad3cccb1eff0/userdata/shm major:0 minor:240 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fe1ce4b21e681d416ffba2a01a9f38924ef7afc592d8e253123303359bbdf086/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fe1ce4b21e681d416ffba2a01a9f38924ef7afc592d8e253123303359bbdf086/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/078fd0e8-1393-45d0-baee-bbcd88095b2b/volumes/kubernetes.io~projected/kube-api-access-crppr:{mountpoint:/var/lib/kubelet/pods/078fd0e8-1393-45d0-baee-bbcd88095b2b/volumes/kubernetes.io~projected/kube-api-access-crppr major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0d1b347f-c22e-489b-986d-d3b3e01fa36b/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/0d1b347f-c22e-489b-986d-d3b3e01fa36b/volumes/kubernetes.io~secret/agent-certs major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/139583a1-f622-49ab-8c1e-af168fcdf6c4/volumes/kubernetes.io~projected/kube-api-access-xds6d:{mountpoint:/var/lib/kubelet/pods/139583a1-f622-49ab-8c1e-af168fcdf6c4/volumes/kubernetes.io~projected/kube-api-access-xds6d major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/14285dfa-dd38-490e-a486-a1fcfc3d0214/volumes/kubernetes.io~projected/kube-api-access-6llvc:{mountpoint:/var/lib/kubelet/pods/14285dfa-dd38-490e-a486-a1fcfc3d0214/volumes/kubernetes.io~projected/kube-api-access-6llvc major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1881dcfe-c2da-4253-bca5-93f3526e14fa/volumes/kubernetes.io~projected/kube-api-access-566pb:{mountpoint:/var/lib/kubelet/pods/1881dcfe-c2da-4253-bca5-93f3526e14fa/volumes/kubernetes.io~projected/kube-api-access-566pb major:0 minor:216 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/251cf359-f9f5-44ae-8ba9-42bf0cabb1ab/volumes/kubernetes.io~projected/kube-api-access-6kpm2:{mountpoint:/var/lib/kubelet/pods/251cf359-f9f5-44ae-8ba9-42bf0cabb1ab/volumes/kubernetes.io~projected/kube-api-access-6kpm2 major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:234 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed/volumes/kubernetes.io~projected/kube-api-access-84rlf:{mountpoint:/var/lib/kubelet/pods/2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed/volumes/kubernetes.io~projected/kube-api-access-84rlf major:0 minor:233 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:223 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:226 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4606f172-d949-471e-842c-b340b50e179b/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/4606f172-d949-471e-842c-b340b50e179b/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4606f172-d949-471e-842c-b340b50e179b/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/4606f172-d949-471e-842c-b340b50e179b/volumes/kubernetes.io~empty-dir/tmp major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4606f172-d949-471e-842c-b340b50e179b/volumes/kubernetes.io~projected/kube-api-access-6rl4f:{mountpoint:/var/lib/kubelet/pods/4606f172-d949-471e-842c-b340b50e179b/volumes/kubernetes.io~projected/kube-api-access-6rl4f major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/559377b2-e3fe-4889-995e-55127771d05e/volumes/kubernetes.io~projected/kube-api-access-zmzz7:{mountpoint:/var/lib/kubelet/pods/559377b2-e3fe-4889-995e-55127771d05e/volumes/kubernetes.io~projected/kube-api-access-zmzz7 major:0 minor:231 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/559377b2-e3fe-4889-995e-55127771d05e/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/559377b2-e3fe-4889-995e-55127771d05e/volumes/kubernetes.io~secret/cert major:0 minor:224 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5dc56e7e-f3c1-43eb-a21b-8be6bb4706cf/volumes/kubernetes.io~projected/kube-api-access-pjrlm:{mountpoint:/var/lib/kubelet/pods/5dc56e7e-f3c1-43eb-a21b-8be6bb4706cf/volumes/kubernetes.io~projected/kube-api-access-pjrlm major:0 minor:215 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5dc56e7e-f3c1-43eb-a21b-8be6bb4706cf/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/5dc56e7e-f3c1-43eb-a21b-8be6bb4706cf/volumes/kubernetes.io~secret/serving-cert major:0 minor:210 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5ed36f74-5c9e-408e-a7fb-2364144f054e/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/5ed36f74-5c9e-408e-a7fb-2364144f054e/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:238 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5ed36f74-5c9e-408e-a7fb-2364144f054e/volumes/kubernetes.io~projected/kube-api-access-7l585:{mountpoint:/var/lib/kubelet/pods/5ed36f74-5c9e-408e-a7fb-2364144f054e/volumes/kubernetes.io~projected/kube-api-access-7l585 major:0 minor:235 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5ed36f74-5c9e-408e-a7fb-2364144f054e/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/5ed36f74-5c9e-408e-a7fb-2364144f054e/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5ed36f74-5c9e-408e-a7fb-2364144f054e/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/5ed36f74-5c9e-408e-a7fb-2364144f054e/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:227 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/61903cf2-2b92-4f40-ab52-25509de587ef/volumes/kubernetes.io~projected/kube-api-access-qgb9w:{mountpoint:/var/lib/kubelet/pods/61903cf2-2b92-4f40-ab52-25509de587ef/volumes/kubernetes.io~projected/kube-api-access-qgb9w major:0 minor:217 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/61903cf2-2b92-4f40-ab52-25509de587ef/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/61903cf2-2b92-4f40-ab52-25509de587ef/volumes/kubernetes.io~secret/serving-cert major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6af42f7e-20f7-4b52-9759-0bf1af4d85b4/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/6af42f7e-20f7-4b52-9759-0bf1af4d85b4/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6af42f7e-20f7-4b52-9759-0bf1af4d85b4/volumes/kubernetes.io~projected/kube-api-access-5lbm4:{mountpoint:/var/lib/kubelet/pods/6af42f7e-20f7-4b52-9759-0bf1af4d85b4/volumes/kubernetes.io~projected/kube-api-access-5lbm4 major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6af42f7e-20f7-4b52-9759-0bf1af4d85b4/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/6af42f7e-20f7-4b52-9759-0bf1af4d85b4/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/783c7a98-5ebf-44dd-8da9-797e71c28169/volumes/kubernetes.io~projected/kube-api-access-vvgs7:{mountpoint:/var/lib/kubelet/pods/783c7a98-5ebf-44dd-8da9-797e71c28169/volumes/kubernetes.io~projected/kube-api-access-vvgs7 major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/80603987-09a1-4ca1-b8b4-3c9ac8d0efc6/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/80603987-09a1-4ca1-b8b4-3c9ac8d0efc6/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:288 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/88de9f3a-e7d0-400f-9091-46c0b89c1a3c/volumes/kubernetes.io~projected/kube-api-access-nxcbj:{mountpoint:/var/lib/kubelet/pods/88de9f3a-e7d0-400f-9091-46c0b89c1a3c/volumes/kubernetes.io~projected/kube-api-access-nxcbj major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/928e22b3-c3fe-453b-bbc8-a67accd2fcb4/volumes/kubernetes.io~projected/kube-api-access-vkng8:{mountpoint:/var/lib/kubelet/pods/928e22b3-c3fe-453b-bbc8-a67accd2fcb4/volumes/kubernetes.io~projected/kube-api-access-vkng8 major:0 minor:312 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/94a6f5e3-d553-487e-a6f1-8c2b0fc5c695/volumes/kubernetes.io~projected/kube-api-access-wjms2:{mountpoint:/var/lib/kubelet/pods/94a6f5e3-d553-487e-a6f1-8c2b0fc5c695/volumes/kubernetes.io~projected/kube-api-access-wjms2 major:0 minor:334 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9822c484-fd06-4e5e-94b4-ef0f5ade9a13/volumes/kubernetes.io~projected/kube-api-access-xbj4z:{mountpoint:/var/lib/kubelet/pods/9822c484-fd06-4e5e-94b4-ef0f5ade9a13/volumes/kubernetes.io~projected/kube-api-access-xbj4z major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9aba3099-fe4c-4846-9a34-25db2c2fe593/volumes/kubernetes.io~projected/kube-api-access-2jjgr:{mountpoint:/var/lib/kubelet/pods/9aba3099-fe4c-4846-9a34-25db2c2fe593/volumes/kubernetes.io~projected/kube-api-access-2jjgr major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9b2b3b14-8027-489a-88f0-d056226c2f67/volumes/kubernetes.io~projected/kube-api-access-bhs2g:{mountpoint:/var/lib/kubelet/pods/9b2b3b14-8027-489a-88f0-d056226c2f67/volumes/kubernetes.io~projected/kube-api-access-bhs2g major:0 minor:236 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a3d234b3-76d4-47d5-98a4-dafd83477b17/volumes/kubernetes.io~projected/kube-api-access-bjmbq:{mountpoint:/var/lib/kubelet/pods/a3d234b3-76d4-47d5-98a4-dafd83477b17/volumes/kubernetes.io~projected/kube-api-access-bjmbq major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a3d234b3-76d4-47d5-98a4-dafd83477b17/volumes/kubernetes.io~secret/default-certificate:{mountpoint:/var/lib/kubelet/pods/a3d234b3-76d4-47d5-98a4-dafd83477b17/volumes/kubernetes.io~secret/default-certificate major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a3d234b3-76d4-47d5-98a4-dafd83477b17/volumes/kubernetes.io~secret/stats-auth:{mountpoint:/var/lib/kubelet/pods/a3d234b3-76d4-47d5-98a4-dafd83477b17/volumes/kubernetes.io~secret/stats-auth major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a707a5d7-313f-4bef-8baf-a99aeae40d54/volumes/kubernetes.io~projected/kube-api-access-h4cc6:{mountpoint:/var/lib/kubelet/pods/a707a5d7-313f-4bef-8baf-a99aeae40d54/volumes/kubernetes.io~projected/kube-api-access-h4cc6 major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/af2d6f84-aea3-4698-be12-11bd6767caec/volumes/kubernetes.io~projected/kube-api-access-4d7j7:{mountpoint:/var/lib/kubelet/pods/af2d6f84-aea3-4698-be12-11bd6767caec/volumes/kubernetes.io~projected/kube-api-access-4d7j7 major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b874991e-9562-4848-b8f1-4b5a2b7cddc4/volumes/kubernetes.io~projected/kube-api-access-zvtmg:{mountpoint:/var/lib/kubelet/pods/b874991e-9562-4848-b8f1-4b5a2b7cddc4/volumes/kubernetes.io~projected/kube-api-access-zvtmg major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b874991e-9562-4848-b8f1-4b5a2b7cddc4/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/b874991e-9562-4848-b8f1-4b5a2b7cddc4/volumes/kubernetes.io~secret/serving-cert major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bba787ce-4e61-4a5a-a728-af73c68dfdd3/volumes/kubernetes.io~projected/kube-api-access-fzc86:{mountpoint:/var/lib/kubelet/pods/bba787ce-4e61-4a5a-a728-af73c68dfdd3/volumes/kubernetes.io~projected/kube-api-access-fzc86 major:0 minor:335 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c1e4fab8-6072-4fcd-9624-a60dee7aed86/volumes/kubernetes.io~projected/kube-api-access-2jzc7:{mountpoint:/var/lib/kubelet/pods/c1e4fab8-6072-4fcd-9624-a60dee7aed86/volumes/kubernetes.io~projected/kube-api-access-2jzc7 major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c4d423f3-2077-47fb-bf4d-d9991dd5eac1/volumes/kubernetes.io~projected/kube-api-access-dj9wz:{mountpoint:/var/lib/kubelet/pods/c4d423f3-2077-47fb-bf4d-d9991dd5eac1/volumes/kubernetes.io~projected/kube-api-access-dj9wz major:0 minor:333 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c5c3a55b-e259-4550-9b7e-f666a28d217e/volumes/kubernetes.io~projected/kube-api-access-mqmpd:{mountpoint:/var/lib/kubelet/pods/c5c3a55b-e259-4550-9b7e-f666a28d217e/volumes/kubernetes.io~projected/kube-api-access-mqmpd major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cea1b649-9c31-4434-9448-7f6652496830/volumes/kubernetes.io~projected/kube-api-access-2ngcm:{mountpoint:/var/lib/kubelet/pods/cea1b649-9c31-4434-9448-7f6652496830/volumes/kubernetes.io~projected/kube-api-access-2ngcm major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d27ee5b1-d1aa-418c-9169-12cac37dcc2c/volumes/kubernetes.io~projected/kube-api-access-r998s:{mountpoint:/var/lib/kubelet/pods/d27ee5b1-d1aa-418c-9169-12cac37dcc2c/volumes/kubernetes.io~projected/kube-api-access-r998s major:0 minor:295 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e4ca0ea9-b57d-4d4c-be3a-2da5fc26a65d/volumes/kubernetes.io~projected/kube-api-access-qzg5q:{mountpoint:/var/lib/kubelet/pods/e4ca0ea9-b57d-4d4c-be3a-2da5fc26a65d/volumes/kubernetes.io~projected/kube-api-access-qzg5q major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e4ca0ea9-b57d-4d4c-be3a-2da5fc26a65d/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/e4ca0ea9-b57d-4d4c-be3a-2da5fc26a65d/volumes/kubernetes.io~secret/serving-cert major:0 minor:208 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/3ef72d4c4dbc3efc206b1f83dce8c18aac655dfbf4629b4d405d1af5a0f5ac84/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/7bafad985c26aa0ad98bfbe877947c91693abaaf7e82fb15396699d90ad07183/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/1cb2363bc67f4c4a14bbf053fee04b4815453103c04507d807801e8966d09114/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/1be7e092fbc87d62d98ffca1c380396d9ec0d60d577a3a2cc3d6bb30ffbbbbec/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/51d42fe0a7cc32b3ad304749d054dcdacb8070e6ea7735f256eb0ada0b57577e/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/a9072824895ecc410422c3d04e06d790a76b21463c38e0a4de579173d670d113/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/fd4c94f7faa0757a78f1b237f56f9bf9facbe4d64380eb22ebb13a48c4be8cb0/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/93229e88dd8814a98659c50c2177e2bc611bbfa81486c0ae9f700d54d0ff553e/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/f6a509b5d7bf91f516a67e1a969832818cdc56985f924a4beea71c57608ff96e/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/6cbb69d243086d38e4159ae09bce21505dc094ac7125a632b6f6a0e6a0baa71d/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/49110398de5b51a8200d0f5dde7feb45d2ec278d42d3afdcc7323bd1b9efba72/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-143:{mountpoint:/var/lib/containers/storage/overlay/3d54daa60c26eee1e07beda9e0fdcf07888d8f3dbe708bd72a9f8ef795963094/merged major:0 minor:143 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/39558e078f9d7703ff225b37ec4eae193d04eb76c583c020866397184afb6ca8/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-154:{mountpoint:/var/lib/containers/storage/overlay/7afcf9e1e7300c6174fd965f3dd0c75f36e81eea3a47899db7c603c7d24ae850/merged major:0 minor:154 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/c87ef8563b0a6f698b8efe1ab85682e4fe247e5a8bf774a3aa4c07669973f5e8/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/886329c11741aca4add7deb367393736cbc4805e07df743851d74273c60f4bdf/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/9cc7e0fd72ca347e672ee87a341659de1479c3dab4cb766431f5304fde3ad099/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/99f5d89c812cc0b87e2d369b6a5a16deb402a865689d3205a3c142afe5efa99f/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/1a4e22cfe9f3a10c4d899b38c8d34866ac4b08d3e0d86aaf03078988b6f7a844/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/7510bec549a89aea7f7a66821522affc699699c78f06dbe3c2226d88671d5dc6/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/d1a7c284c0c7fb425004f5c01ea9a817f4dc9bac0ac581d41145dc422483a050/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/cd5fba668fbfe7c821951a41e960392584e62de9bda83b9881c8e165ae3159f6/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/57b5ab05d1431085f597d2eff9faef4c8dafb2e4c5ee60e254328856955560a7/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-252:{mountpoint:/var/lib/containers/storage/overlay/883849a89d192b1ba12e20cb72e3ceb8d0841dd6a832bc97c4bf19e8283bf70d/merged major:0 minor:252 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/201f5b292f4f13e82d24d63b95147914d3e054b078e6770e058d8ffc0962ddc7/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/79c77bcb61d4f6a553cd9862fbc65283c8b07524faa4e85dc98b35bebfec5170/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/607477c8d17fd940bfdabdd19f38b4b8d38be515892ee6a5690e0c902a0826c2/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-260:{mountpoint:/var/lib/containers/storage/overlay/7848b420332f33901bdcdac5d11557332c1fdae509e93dd2c32a10fe3f6f0439/merged major:0 minor:260 fsType:overlay blockSize:0} overlay_0-262:{mountpoint:/var/lib/containers/storage/overlay/2f8d4ce9c7485a12caae2c2034826e40d94330f6a7d9daebc0b94a8de01a40ca/merged major:0 minor:262 fsType:overlay blockSize:0} overlay_0-264:{mountpoint:/var/lib/containers/storage/overlay/910bcba7aa6eba8a8d45dd08c8ddaa5070e11c960c405468ebf0ccbe7cdcd003/merged major:0 minor:264 fsType:overlay blockSize:0} overlay_0-266:{mountpoint:/var/lib/containers/storage/overlay/f147d624f1bdc702ecc79abe3f91dc1284f0242859500ba3adbb7b3de59548d1/merged major:0 minor:266 fsType:overlay blockSize:0} overlay_0-276:{mountpoint:/var/lib/containers/storage/overlay/7fc7bd2267916aa701552963c473dc92228d86a3c8d95f9507486be41821046d/merged major:0 minor:276 fsType:overlay blockSize:0} overlay_0-281:{mountpoint:/var/lib/containers/storage/overlay/fb58da05f3bbb67f9d659a3a9b0ee0053c5975e71d207f92df200badf22325e1/merged major:0 minor:281 fsType:overlay blockSize:0} overlay_0-284:{mountpoint:/var/lib/containers/storage/overlay/dd9bf1336e12a317e78ad06f3a42eaf5e9f92cd14ad64a909ab0fe492d8cc6e8/merged major:0 minor:284 fsType:overlay blockSize:0} overlay_0-286:{mountpoint:/var/lib/containers/storage/overlay/037151fe8896163a937f6ccf2cc9f3a4d293020e4fe20d1c45862398d0c58ca7/merged major:0 minor:286 fsType:overlay blockSize:0} overlay_0-296:{mountpoint:/var/lib/containers/storage/overlay/4d97292c06f58d599fb0b9ca22de267f7f03cbf346f32f9e1444fff428f0be4b/merged major:0 minor:296 fsType:overlay blockSize:0} overlay_0-300:{mountpoint:/var/lib/containers/storage/overlay/46f4c40be6f65e960df50230922b0192a2947ebe5635001f847a2f6ab0c6482c/merged major:0 minor:300 fsType:overlay blockSize:0} overlay_0-302:{mountpoint:/var/lib/containers/storage/overlay/d994fedef3397d8e78a8e9f70aeb2bcbe4a6eba528a3b786a28026f51f57b0a3/merged major:0 minor:302 fsType:overlay blockSize:0} overlay_0-304:{mountpoint:/var/lib/containers/storage/overlay/d962e7c8a1289c00bebf97798a75becf47f5b9252361b605bb1f6a491354a0e2/merged major:0 minor:304 fsType:overlay blockSize:0} overlay_0-306:{mountpoint:/var/lib/containers/storage/overlay/a9c1263e11d7c6affed38bc63c744cc2c4250317ad263cd344977a085a814cb7/merged major:0 minor:306 fsType:overlay blockSize:0} overlay_0-308:{mountpoint:/var/lib/containers/storage/overlay/97651ae04b5949c653f01f3ddfd72f746c51249006b9d2c18728f0738d054ad3/merged major:0 minor:308 fsType:overlay blockSize:0} overlay_0-310:{mountpoint:/var/lib/containers/storage/overlay/44536afb082fccbbed17ed9bab6615f5dfcaf95c6633a54b09facd80aa66c8b0/merged major:0 minor:310 fsType:overlay blockSize:0} overlay_0-342:{mountpoint:/var/lib/containers/storage/overlay/91b88f8520243e9a8302d02a5c517113463f69ddf6f74d3fbeebf54e68d008b3/merged major:0 minor:342 fsType:overlay blockSize:0} overlay_0-344:{mountpoint:/var/lib/containers/storage/overlay/49a93272aa3f1c2ef1d637df48a4c04b223c88998a9ef126b91d3c42d65fde38/merged major:0 minor:344 fsType:overlay blockSize:0} overlay_0-346:{mountpoint:/var/lib/containers/storage/overlay/f3e04ca6212e6af050e3f187f79512dbf887c4102e1e9b806b763f06bf30b5a9/merged major:0 minor:346 fsType:overlay blockSize:0} overlay_0-348:{mountpoint:/var/lib/containers/storage/overlay/5f47686c64afd4d3d24244cf0889d8b67e4564d2cc2493a95d6cc71b0b27b0b5/merged major:0 minor:348 fsType:overlay blockSize:0} overlay_0-350:{mountpoint:/var/lib/containers/storage/overlay/1a1acca4d0b4e4b9d95cab5a535ebec7e93d54bf915467cb3317952d2cccdbb7/merged major:0 minor:350 fsType:overlay blockSize:0} overlay_0-356:{mountpoint:/var/lib/containers/storage/overlay/1eca7b785b8052ed3e443961ec0b5d75d1030fd4bb9ad54b7c67e75d591eac37/merged major:0 minor:356 fsType:overlay blockSize:0} overlay_0-362:{mountpoint:/var/lib/containers/storage/overlay/ee873b82c1ef8ad78549cb0ab0e5a0f444487b1033d5a8bb91b9fc88df6997d5/merged major:0 minor:362 fsType:overlay blockSize:0} overlay_0-364:{mountpoint:/var/lib/containers/storage/overlay/e87a603fdaae7b1f087c2c8c6002f8a875d4361589fb68089a7e45d917434d64/merged major:0 minor:364 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/9472749989eea6a08f60426ace2d9458a0229d229c1d3eecd1d9e84759d9266e/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/f03b0f9126ee68008217cff053130094d127856f030a837ec489b612c05f71d2/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/61946debb46bd89f4dfcd529b05c5c41b5e94a06bfe5e22cdbca63efe45ac6c7/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-72:{mountpoint:/var/lib/containers/storage/overlay/740ceb90fda1abb949dd1ce67ef9d1a5b092c28d493186284316f9613a9a6fe0/merged major:0 minor:72 fsType:overlay blockSize:0} overlay_0-76:{mountpoint:/var/lib/containers/storage/overlay/0ebf26c7c4f5ddbd2248afd53c6b11d28cd0302f85338393e40394b4d9757a48/merged major:0 minor:76 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/fef024e081fa86cd22c0bffeb692ebf9b1101c9cbcd04ae3321b3516469c2c17/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/1db31f4f0ed47159939ff84da54f527619dd0e8d319eb8b9548f68179917cf8c/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-89:{mountpoint:/var/lib/containers/storage/overlay/69844fc1b1f9ac0c18f915abd7c4be0e05ee698bb002622c28c24d1ec09accdb/merged major:0 minor:89 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/a4eff049844940614440afde1268b0721993134798b734592277c3cde8b33289/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-98:{mountpoint:/var/lib/containers/storage/overlay/21e85107983750987a44965503618fc5e3e82d61a138672b43328d682c20a184/merged major:0 minor:98 fsType:overlay blockSize:0}]
Oct 30 00:31:44.444557 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.444159    5002 manager.go:217] Machine: {Timestamp:2026-10-30 00:31:44.443701994 +0000 UTC m=+0.060134865 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099060 MemoryCapacity:33164333056 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2f5f3fefea6c82c96c9899e764860b SystemUUID:ec2f5f3f-efea-6c82-c96c-9899e764860b BootID:c43aa09f-cd5a-4eb6-bc6e-43c1a088b226 Filesystems:[{Device:/var/lib/kubelet/pods/078fd0e8-1393-45d0-baee-bbcd88095b2b/volumes/kubernetes.io~projected/kube-api-access-crppr DeviceMajor:0 DeviceMinor:62 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/928e22b3-c3fe-453b-bbc8-a67accd2fcb4/volumes/kubernetes.io~projected/kube-api-access-vkng8 DeviceMajor:0 DeviceMinor:312 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/14285dfa-dd38-490e-a486-a1fcfc3d0214/volumes/kubernetes.io~projected/kube-api-access-6llvc DeviceMajor:0 DeviceMinor:63 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/c1e4fab8-6072-4fcd-9624-a60dee7aed86/volumes/kubernetes.io~projected/kube-api-access-2jzc7 DeviceMajor:0 DeviceMinor:67 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/88de9f3a-e7d0-400f-9091-46c0b89c1a3c/volumes/kubernetes.io~projected/kube-api-access-nxcbj DeviceMajor:0 DeviceMinor:59 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/650dac582e52282d5b674397229aa50ac75a19dc3768efb9eb967a5f252fc938/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0a1b48da841c11cc03ad3c6dd62932dd0b7de30c4e34fc221f366ec25b3a9ee5/userdata/shm DeviceMajor:0 DeviceMinor:219 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a342adc28b63c1d46dd71a11bcf85d3d52a05101220c43ef41724801c48c9fa4/userdata/shm DeviceMajor:0 DeviceMinor:228 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/9822c484-fd06-4e5e-94b4-ef0f5ade9a13/volumes/kubernetes.io~projected/kube-api-access-xbj4z DeviceMajor:0 DeviceMinor:206 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-264 DeviceMajor:0 DeviceMinor:264 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-260 DeviceMajor:0 DeviceMinor:260 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bba787ce-4e61-4a5a-a728-af73c68dfdd3/volumes/kubernetes.io~projected/kube-api-access-fzc86 DeviceMajor:0 DeviceMinor:335 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-350 DeviceMajor:0 DeviceMinor:350 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-364 DeviceMajor:0 DeviceMinor:364 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-266 DeviceMajor:0 DeviceMinor:266 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-304 DeviceMajor:0 DeviceMinor:304 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-276 DeviceMajor:0 DeviceMinor:276 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-154 DeviceMajor:0 DeviceMinor:154 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4606f172-d949-471e-842c-b340b50e179b/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:56 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/5ed36f74-5c9e-408e-a7fb-2364144f054e/volumes/kubernetes.io~projected/kube-api-access-7l585 DeviceMajor:0 DeviceMinor:235 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-262 DeviceMajor:0 DeviceMinor:262 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-302 DeviceMajor:0 DeviceMinor:302 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/94e5870821441f01b75fd1499c09731d42f372db8af82d71e594850adef78e99/userdata/shm DeviceMajor:0 DeviceMinor:88 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-89 DeviceMajor:0 DeviceMinor:89 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:234 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/af2d6f84-aea3-4698-be12-11bd6767caec/volumes/kubernetes.io~projected/kube-api-access-4d7j7 DeviceMajor:0 DeviceMinor:60 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-76 DeviceMajor:0 DeviceMinor:76 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-252 DeviceMajor:0 DeviceMinor:252 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b984324d5c7f9d7f10d6b87625aafb838e43218eaa1db56a88123a6411f19142/userdata/shm DeviceMajor:0 DeviceMinor:340 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-362 DeviceMajor:0 DeviceMinor:362 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9b2b3b14-8027-489a-88f0-d056226c2f67/volumes/kubernetes.io~projected/kube-api-access-bhs2g DeviceMajor:0 DeviceMinor:236 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-296 DeviceMajor:0 DeviceMinor:296 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e4ca0ea9-b57d-4d4c-be3a-2da5fc26a65d/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:208 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed/volumes/kubernetes.io~projected/kube-api-access-84rlf DeviceMajor:0 DeviceMinor:233 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/83e0c503316f6412a51e1550577df70d9ffcbe1c49b03e2cbec69f56d9cbce9e/userdata/shm DeviceMajor:0 DeviceMinor:95 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/9aba3099-fe4c-4846-9a34-25db2c2fe593/volumes/kubernetes.io~projected/kube-api-access-2jjgr DeviceMajor:0 DeviceMinor:201 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/783c7a98-5ebf-44dd-8da9-797e71c28169/volumes/kubernetes.io~projected/kube-api-access-vvgs7 DeviceMajor:0 DeviceMinor:202 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/80603987-09a1-4ca1-b8b4-3c9ac8d0efc6/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:288 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-308 DeviceMajor:0 DeviceMinor:308 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-346 DeviceMajor:0 DeviceMinor:346 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:overlay_0-98 DeviceMajor:0 DeviceMinor:98 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7e631fdaf2b86d82764221a535cde6bc365d8253137f5469f9370b342ecc6c6d/userdata/shm DeviceMajor:0 DeviceMinor:242 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-310 DeviceMajor:0 DeviceMinor:310 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6af42f7e-20f7-4b52-9759-0bf1af4d85b4/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/e4ca0ea9-b57d-4d4c-be3a-2da5fc26a65d/volumes/kubernetes.io~projected/kube-api-access-qzg5q DeviceMajor:0 DeviceMinor:218 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/5ed36f74-5c9e-408e-a7fb-2364144f054e/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:225 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/4606f172-d949-471e-842c-b340b50e179b/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:58 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4606f172-d949-471e-842c-b340b50e179b/volumes/kubernetes.io~projected/kube-api-access-6rl4f DeviceMajor:0 DeviceMinor:66 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c72433bacc952d570b3676b773c9b174fec945357ecf73c909e453c4a02d6577/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/5dc56e7e-f3c1-43eb-a21b-8be6bb4706cf/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:210 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-281 DeviceMajor:0 DeviceMinor:281 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-348 DeviceMajor:0 DeviceMinor:348 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ea478e19d2f09e1d112adbc723346aa5cdea945578e9f53cf3b2cb2fda738476/userdata/shm DeviceMajor:0 DeviceMinor:74 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/cea1b649-9c31-4434-9448-7f6652496830/volumes/kubernetes.io~projected/kube-api-access-2ngcm DeviceMajor:0 DeviceMinor:214 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-342 DeviceMajor:0 DeviceMinor:342 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d4b5066e0524802f2cab4f8c5119b9785e3cd5bbdac5d0b5fd0b350f47f5baaa/userdata/shm DeviceMajor:0 DeviceMinor:293 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c401bfa9817c86ba3e9537e9932491b450d77a68e1d6d855d93aed8ae46c3445/userdata/shm DeviceMajor:0 DeviceMinor:211 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3e0825f9e3c3deaa21e8cceb37a1971d348f6672ea7e4ccf29e29caaf09b6cef/userdata/shm DeviceMajor:0 DeviceMinor:336 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/c5c3a55b-e259-4550-9b7e-f666a28d217e/volumes/kubernetes.io~projected/kube-api-access-mqmpd DeviceMajor:0 DeviceMinor:64 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-286 DeviceMajor:0 DeviceMinor:286 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/208de9f5a67ade098d715f55f9e24682f2698180040d35b44971ed390bfe1312/userdata/shm DeviceMajor:0 DeviceMinor:338 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/6af42f7e-20f7-4b52-9759-0bf1af4d85b4/volumes/kubernetes.io~projected/kube-api-access-5lbm4 DeviceMajor:0 DeviceMinor:61 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/a707a5d7-313f-4bef-8baf-a99aeae40d54/volumes/kubernetes.io~projected/kube-api-access-h4cc6 DeviceMajor:0 DeviceMinor:65 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/61903cf2-2b92-4f40-ab52-25509de587ef/volumes/kubernetes.io~projected/kube-api-access-qgb9w DeviceMajor:0 DeviceMinor:217 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-284 DeviceMajor:0 DeviceMinor:284 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a3d234b3-76d4-47d5-98a4-dafd83477b17/volumes/kubernetes.io~secret/default-certificate DeviceMajor:0 DeviceMinor:45 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/61903cf2-2b92-4f40-ab52-25509de587ef/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:207 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-300 DeviceMajor:0 DeviceMinor:300 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/559377b2-e3fe-4889-995e-55127771d05e/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:224 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/70a7f9743975c1002c0feb8db36a717b4dc312d0b220be0a3b8dfca5e7122ebd/userdata/shm DeviceMajor:0 DeviceMinor:246 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/a3d234b3-76d4-47d5-98a4-dafd83477b17/volumes/kubernetes.io~projected/kube-api-access-bjmbq DeviceMajor:0 DeviceMinor:204 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a3d234b3-76d4-47d5-98a4-dafd83477b17/volumes/kubernetes.io~secret/stats-auth DeviceMajor:0 DeviceMinor:197 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/139583a1-f622-49ab-8c1e-af168fcdf6c4/volumes/kubernetes.io~projected/kube-api-access-xds6d DeviceMajor:0 DeviceMinor:205 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/5ed36f74-5c9e-408e-a7fb-2364144f054e/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:227 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c0ca8a0286da3206a1984da7ca07bcd8f48346aeb32382aa1ac127f0b959599f/userdata/shm DeviceMajor:0 DeviceMinor:237 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f43de3eccd69933959e622c93a59fc85020fb8a8b6088a8cfc2cad3cccb1eff0/userdata/shm DeviceMajor:0 DeviceMinor:240 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-344 DeviceMajor:0 DeviceMinor:344 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:223 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/0d1b347f-c22e-489b-986d-d3b3e01fa36b/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:55 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b1739fe883553d159fc4a3b364b09e5ac408869eefc63b025dccf4c5273b7f8b/userdata/shm DeviceMajor:0 DeviceMinor:78 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1881dcfe-c2da-4253-bca5-93f3526e14fa/volumes/kubernetes.io~projected/kube-api-access-566pb DeviceMajor:0 DeviceMinor:216 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c42e0453a3ed59a2dec4c08c3a2acc4b6c3a846e1abb8dc1b4db077d05741d0c/userdata/shm DeviceMajor:0 DeviceMinor:230 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/d27ee5b1-d1aa-418c-9169-12cac37dcc2c/volumes/kubernetes.io~projected/kube-api-access-r998s DeviceMajor:0 DeviceMinor:295 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-306 DeviceMajor:0 DeviceMinor:306 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-143 DeviceMajor:0 DeviceMinor:143 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/251cf359-f9f5-44ae-8ba9-42bf0cabb1ab/volumes/kubernetes.io~projected/kube-api-access-6kpm2 DeviceMajor:0 DeviceMinor:203 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-356 DeviceMajor:0 DeviceMinor:356 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/453fcdc0a3bdbd70d0b6a86ba356202ebd1abd1c2e08b95bc5fd8533b3550ac8/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5dc56e7e-f3c1-43eb-a21b-8be6bb4706cf/volumes/kubernetes.io~projected/kube-api-access-pjrlm DeviceMajor:0 DeviceMinor:215 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:226 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/559377b2-e3fe-4889-995e-55127771d05e/volumes/kubernetes.io~projected/kube-api-access-zmzz7 DeviceMajor:0 DeviceMinor:231 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bb5c2326fc6156453ddd2fd1052eb32784198ed45e9d6f91e0b3e693c64e57ef/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fe1ce4b21e681d416ffba2a01a9f38924ef7afc592d8e253123303359bbdf086/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/6af42f7e-20f7-4b52-9759-0bf1af4d85b4/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:57 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/b874991e-9562-4848-b8f1-4b5a2b7cddc4/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:209 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/5ed36f74-5c9e-408e-a7fb-2364144f054e/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:238 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-72 DeviceMajor:0 DeviceMinor:72 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ba10707cdb72fd5b7212466e2efb1d9bf1fe0a9077139125e96f32d1d0295515/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cc9244c17a2ca2d72c4199cc0781aaa7e22b7859adfb3b8c52ee0fb4348e35ef/userdata/shm DeviceMajor:0 DeviceMinor:243 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/57c013576400090006215c678b7fcc105a57fbc3a4f73eb482fe1d1da5fcf302/userdata/shm DeviceMajor:0 DeviceMinor:298 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b874991e-9562-4848-b8f1-4b5a2b7cddc4/volumes/kubernetes.io~projected/kube-api-access-zvtmg DeviceMajor:0 DeviceMinor:213 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/43e781b035175dffab13f8dbbfd8ba443ef3114ebef4867715bd8514bcd1217e/userdata/shm DeviceMajor:0 DeviceMinor:221 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/c4d423f3-2077-47fb-bf4d-d9991dd5eac1/volumes/kubernetes.io~projected/kube-api-access-dj9wz DeviceMajor:0 DeviceMinor:333 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/94a6f5e3-d553-487e-a6f1-8c2b0fc5c695/volumes/kubernetes.io~projected/kube-api-access-wjms2 DeviceMajor:0 DeviceMinor:334 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:0a1b48da841c11c MacAddress:e6:8c:6a:36:6e:d7 Speed:10000 Mtu:8901} {Name:208de9f5a67ade0 MacAddress:d2:f0:5d:62:d2:e1 Speed:10000 Mtu:8901} {Name:3e0825f9e3c3dea MacAddress:9e:4c:44:e3:bc:82 Speed:10000 Mtu:8901} {Name:43e781b035175df MacAddress:1e:e0:14:21:7f:f4 Speed:10000 Mtu:8901} {Name:57c013576400090 MacAddress:e6:d7:40:2e:44:9e Speed:10000 Mtu:8901} {Name:70a7f9743975c10 MacAddress:4e:04:e7:9e:83:3f Speed:10000 Mtu:8901} {Name:7e631fdaf2b86d8 MacAddress:22:a6:e9:14:a7:01 Speed:10000 Mtu:8901} {Name:a342adc28b63c1d MacAddress:02:e7:98:71:b0:10 Speed:10000 Mtu:8901} {Name:b984324d5c7f9d7 MacAddress:62:f1:c7:6a:68:cf Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:d8:58:65:ed Speed:0 Mtu:9001} {Name:br-int MacAddress:fe:cf:66:cc:30:e3 Speed:0 Mtu:8901} {Name:c0ca8a0286da320 MacAddress:aa:3d:68:a0:a7:8f Speed:10000 Mtu:8901} {Name:c401bfa9817c86b MacAddress:b2:60:bb:14:0c:ba Speed:10000 Mtu:8901} {Name:c42e0453a3ed59a MacAddress:c6:9c:eb:7d:53:80 Speed:10000 Mtu:8901} {Name:cc9244c17a2ca2d MacAddress:b2:ff:38:e4:ce:85 Speed:10000 Mtu:8901} {Name:d4b5066e0524802 MacAddress:0a:d0:60:0e:25:76 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:d8:58:65:ed Speed:0 Mtu:9001} {Name:f43de3eccd69933 MacAddress:76:30:89:fb:c4:12 Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:f2:82:9c:36:f0:5b Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:84:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:16:80:d1:87:0b:00 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164333056 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 00:31:44.444777 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.444562    5002 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 00:31:44.444777 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.444613    5002 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 00:31:44.444964 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.444935    5002 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 00:31:44.445101 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.444966    5002 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-7.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 00:31:44.445145 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.445103    5002 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 00:31:44.445145 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.445111    5002 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 00:31:44.445145 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.445121    5002 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 00:31:44.445145 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.445135    5002 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 00:31:44.445246 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.445154    5002 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 00:31:44.445311 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.445301    5002 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 00:31:44.445450 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.445440    5002 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 00:31:44.445500 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.445477    5002 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 00:31:44.445500 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.445487    5002 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 00:31:44.445591 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.445501    5002 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 00:31:44.445591 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.445510    5002 kubelet.go:405] "Adding apiserver pod source"
Oct 30 00:31:44.445591 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.445521    5002 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 00:31:44.446457 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.446438    5002 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 00:31:44.446457 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.446456    5002 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 00:31:44.447052 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.446999    5002 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 00:31:44.449608 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.449594    5002 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 00:31:44.449608 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.449608    5002 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 00:31:44.449716 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.449706    5002 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Oct 30 00:31:44.449898 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.449887    5002 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 00:31:44.449931 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.449916    5002 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 00:31:44.450019 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.450010    5002 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 00:31:44.450054 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.450024    5002 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 00:31:44.450054 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.450032    5002 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 00:31:44.450054 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.450038    5002 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 00:31:44.450054 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.450045    5002 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 00:31:44.450153 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.450055    5002 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 00:31:44.450153 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.450064    5002 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 00:31:44.450153 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.450071    5002 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 00:31:44.450153 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.450077    5002 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 00:31:44.450153 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.450083    5002 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 00:31:44.450153 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.450097    5002 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 00:31:44.450153 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.450107    5002 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 00:31:44.450153 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.450131    5002 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 00:31:44.450153 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.450138    5002 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 00:31:44.454211 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.454187    5002 server.go:1263] "Started kubelet"
Oct 30 00:31:44.454307 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.454264    5002 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 00:31:44.454363 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.454295    5002 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 00:31:44.454425 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.454404    5002 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 00:31:44.454712 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.454683    5002 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 00:31:44.454741 ip-10-0-128-7 systemd[1]: Started Kubernetes Kubelet.
Oct 30 00:31:44.455662 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.455647    5002 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 00:31:44.459103 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.459083    5002 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 00:31:44.459103 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.459097    5002 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 00:31:44.459103 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.459187    5002 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 00:31:44.459103 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.459200    5002 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 00:31:44.459103 ip-10-0-128-7 kubenswrapper[5002]: E1030 00:31:44.459234    5002 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-7.ec2.internal\" not found"
Oct 30 00:31:44.459103 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.459269    5002 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 00:26:05 +0000 UTC" deadline="2028-08-07 08:25:58.462591724 +0000 UTC"
Oct 30 00:31:44.459103 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.459289    5002 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15535h54m14.003305746s"
Oct 30 00:31:44.459103 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.459324    5002 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 00:31:44.461890 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.461863    5002 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="251cf359-f9f5-44ae-8ba9-42bf0cabb1ab" volumeName="kubernetes.io/empty-dir/251cf359-f9f5-44ae-8ba9-42bf0cabb1ab-tmp-dir" seLinuxMountContext=""
Oct 30 00:31:44.461988 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.461893    5002 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed" volumeName="kubernetes.io/configmap/2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed-trusted-ca" seLinuxMountContext=""
Oct 30 00:31:44.461988 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.461909    5002 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5dc56e7e-f3c1-43eb-a21b-8be6bb4706cf" volumeName="kubernetes.io/configmap/5dc56e7e-f3c1-43eb-a21b-8be6bb4706cf-trusted-ca" seLinuxMountContext=""
Oct 30 00:31:44.461988 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.461920    5002 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c1e4fab8-6072-4fcd-9624-a60dee7aed86" volumeName="kubernetes.io/configmap/c1e4fab8-6072-4fcd-9624-a60dee7aed86-iptables-alerter-script" seLinuxMountContext=""
Oct 30 00:31:44.461988 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.461935    5002 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9b2b3b14-8027-489a-88f0-d056226c2f67" volumeName="kubernetes.io/projected/9b2b3b14-8027-489a-88f0-d056226c2f67-kube-api-access-bhs2g" seLinuxMountContext=""
Oct 30 00:31:44.461988 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.461946    5002 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a3d234b3-76d4-47d5-98a4-dafd83477b17" volumeName="kubernetes.io/projected/a3d234b3-76d4-47d5-98a4-dafd83477b17-kube-api-access-bjmbq" seLinuxMountContext=""
Oct 30 00:31:44.461988 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.461959    5002 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5dc56e7e-f3c1-43eb-a21b-8be6bb4706cf" volumeName="kubernetes.io/projected/5dc56e7e-f3c1-43eb-a21b-8be6bb4706cf-kube-api-access-pjrlm" seLinuxMountContext=""
Oct 30 00:31:44.461988 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.461971    5002 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0d1b347f-c22e-489b-986d-d3b3e01fa36b" volumeName="kubernetes.io/configmap/0d1b347f-c22e-489b-986d-d3b3e01fa36b-konnectivity-ca" seLinuxMountContext=""
Oct 30 00:31:44.461988 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.461985    5002 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed" volumeName="kubernetes.io/projected/2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed-bound-sa-token" seLinuxMountContext=""
Oct 30 00:31:44.462411 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.461997    5002 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5ed36f74-5c9e-408e-a7fb-2364144f054e" volumeName="kubernetes.io/projected/5ed36f74-5c9e-408e-a7fb-2364144f054e-bound-sa-token" seLinuxMountContext=""
Oct 30 00:31:44.462411 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.462008    5002 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e4ca0ea9-b57d-4d4c-be3a-2da5fc26a65d" volumeName="kubernetes.io/secret/e4ca0ea9-b57d-4d4c-be3a-2da5fc26a65d-serving-cert" seLinuxMountContext=""
Oct 30 00:31:44.462411 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.462020    5002 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6af42f7e-20f7-4b52-9759-0bf1af4d85b4" volumeName="kubernetes.io/secret/6af42f7e-20f7-4b52-9759-0bf1af4d85b4-ovn-node-metrics-cert" seLinuxMountContext=""
Oct 30 00:31:44.462411 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.462040    5002 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a3d234b3-76d4-47d5-98a4-dafd83477b17" volumeName="kubernetes.io/secret/a3d234b3-76d4-47d5-98a4-dafd83477b17-stats-auth" seLinuxMountContext=""
Oct 30 00:31:44.462411 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.462051    5002 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bba787ce-4e61-4a5a-a728-af73c68dfdd3" volumeName="kubernetes.io/empty-dir/bba787ce-4e61-4a5a-a728-af73c68dfdd3-archives-path" seLinuxMountContext=""
Oct 30 00:31:44.462411 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.462065    5002 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c1e4fab8-6072-4fcd-9624-a60dee7aed86" volumeName="kubernetes.io/projected/c1e4fab8-6072-4fcd-9624-a60dee7aed86-kube-api-access-2jzc7" seLinuxMountContext=""
Oct 30 00:31:44.462411 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.462076    5002 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c4d423f3-2077-47fb-bf4d-d9991dd5eac1" volumeName="kubernetes.io/projected/c4d423f3-2077-47fb-bf4d-d9991dd5eac1-kube-api-access-dj9wz" seLinuxMountContext=""
Oct 30 00:31:44.462411 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.462088    5002 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5ed36f74-5c9e-408e-a7fb-2364144f054e" volumeName="kubernetes.io/projected/5ed36f74-5c9e-408e-a7fb-2364144f054e-kube-api-access-7l585" seLinuxMountContext=""
Oct 30 00:31:44.462411 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.462100    5002 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5ed36f74-5c9e-408e-a7fb-2364144f054e" volumeName="kubernetes.io/secret/5ed36f74-5c9e-408e-a7fb-2364144f054e-installation-pull-secrets" seLinuxMountContext=""
Oct 30 00:31:44.462411 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.462113    5002 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="61903cf2-2b92-4f40-ab52-25509de587ef" volumeName="kubernetes.io/configmap/61903cf2-2b92-4f40-ab52-25509de587ef-config" seLinuxMountContext=""
Oct 30 00:31:44.462411 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.462125    5002 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="61903cf2-2b92-4f40-ab52-25509de587ef" volumeName="kubernetes.io/projected/61903cf2-2b92-4f40-ab52-25509de587ef-kube-api-access-qgb9w" seLinuxMountContext=""
Oct 30 00:31:44.462411 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.462137    5002 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6af42f7e-20f7-4b52-9759-0bf1af4d85b4" volumeName="kubernetes.io/projected/6af42f7e-20f7-4b52-9759-0bf1af4d85b4-kube-api-access-5lbm4" seLinuxMountContext=""
Oct 30 00:31:44.462411 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.462148    5002 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bba787ce-4e61-4a5a-a728-af73c68dfdd3" volumeName="kubernetes.io/configmap/bba787ce-4e61-4a5a-a728-af73c68dfdd3-trusted-ca-bundle" seLinuxMountContext=""
Oct 30 00:31:44.462411 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.462160    5002 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e4ca0ea9-b57d-4d4c-be3a-2da5fc26a65d" volumeName="kubernetes.io/empty-dir/e4ca0ea9-b57d-4d4c-be3a-2da5fc26a65d-snapshots" seLinuxMountContext=""
Oct 30 00:31:44.462411 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.462178    5002 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a3d234b3-76d4-47d5-98a4-dafd83477b17" volumeName="kubernetes.io/secret/a3d234b3-76d4-47d5-98a4-dafd83477b17-default-certificate" seLinuxMountContext=""
Oct 30 00:31:44.462411 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.462190    5002 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed" volumeName="kubernetes.io/secret/2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed-image-registry-private-configuration" seLinuxMountContext=""
Oct 30 00:31:44.462411 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.462202    5002 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5ed36f74-5c9e-408e-a7fb-2364144f054e" volumeName="kubernetes.io/secret/5ed36f74-5c9e-408e-a7fb-2364144f054e-image-registry-private-configuration" seLinuxMountContext=""
Oct 30 00:31:44.462411 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.462214    5002 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6af42f7e-20f7-4b52-9759-0bf1af4d85b4" volumeName="kubernetes.io/configmap/6af42f7e-20f7-4b52-9759-0bf1af4d85b4-ovnkube-config" seLinuxMountContext=""
Oct 30 00:31:44.462411 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.462228    5002 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c5c3a55b-e259-4550-9b7e-f666a28d217e" volumeName="kubernetes.io/configmap/c5c3a55b-e259-4550-9b7e-f666a28d217e-cni-binary-copy" seLinuxMountContext=""
Oct 30 00:31:44.462411 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.462250    5002 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5dc56e7e-f3c1-43eb-a21b-8be6bb4706cf" volumeName="kubernetes.io/configmap/5dc56e7e-f3c1-43eb-a21b-8be6bb4706cf-config" seLinuxMountContext=""
Oct 30 00:31:44.462411 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.462262    5002 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed" volumeName="kubernetes.io/configmap/2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed-registry-certificates" seLinuxMountContext=""
Oct 30 00:31:44.462411 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.462275    5002 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="251cf359-f9f5-44ae-8ba9-42bf0cabb1ab" volumeName="kubernetes.io/configmap/251cf359-f9f5-44ae-8ba9-42bf0cabb1ab-config-volume" seLinuxMountContext=""
Oct 30 00:31:44.462411 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.462287    5002 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="251cf359-f9f5-44ae-8ba9-42bf0cabb1ab" volumeName="kubernetes.io/projected/251cf359-f9f5-44ae-8ba9-42bf0cabb1ab-kube-api-access-6kpm2" seLinuxMountContext=""
Oct 30 00:31:44.462411 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.462300    5002 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5ed36f74-5c9e-408e-a7fb-2364144f054e" volumeName="kubernetes.io/configmap/5ed36f74-5c9e-408e-a7fb-2364144f054e-trusted-ca" seLinuxMountContext=""
Oct 30 00:31:44.462411 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.462312    5002 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bba787ce-4e61-4a5a-a728-af73c68dfdd3" volumeName="kubernetes.io/projected/bba787ce-4e61-4a5a-a728-af73c68dfdd3-kube-api-access-fzc86" seLinuxMountContext=""
Oct 30 00:31:44.462411 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.462324    5002 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1881dcfe-c2da-4253-bca5-93f3526e14fa" volumeName="kubernetes.io/projected/1881dcfe-c2da-4253-bca5-93f3526e14fa-kube-api-access-566pb" seLinuxMountContext=""
Oct 30 00:31:44.462411 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.462335    5002 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c5c3a55b-e259-4550-9b7e-f666a28d217e" volumeName="kubernetes.io/configmap/c5c3a55b-e259-4550-9b7e-f666a28d217e-whereabouts-flatfile-configmap" seLinuxMountContext=""
Oct 30 00:31:44.462411 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.462348    5002 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e4ca0ea9-b57d-4d4c-be3a-2da5fc26a65d" volumeName="kubernetes.io/configmap/e4ca0ea9-b57d-4d4c-be3a-2da5fc26a65d-trusted-ca-bundle" seLinuxMountContext=""
Oct 30 00:31:44.462411 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.462360    5002 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="139583a1-f622-49ab-8c1e-af168fcdf6c4" volumeName="kubernetes.io/projected/139583a1-f622-49ab-8c1e-af168fcdf6c4-kube-api-access-xds6d" seLinuxMountContext=""
Oct 30 00:31:44.462411 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.462372    5002 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="14285dfa-dd38-490e-a486-a1fcfc3d0214" volumeName="kubernetes.io/projected/14285dfa-dd38-490e-a486-a1fcfc3d0214-kube-api-access-6llvc" seLinuxMountContext=""
Oct 30 00:31:44.462411 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.462405    5002 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed" volumeName="kubernetes.io/secret/2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed-installation-pull-secrets" seLinuxMountContext=""
Oct 30 00:31:44.462411 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.462427    5002 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4606f172-d949-471e-842c-b340b50e179b" volumeName="kubernetes.io/empty-dir/4606f172-d949-471e-842c-b340b50e179b-tmp" seLinuxMountContext=""
Oct 30 00:31:44.463735 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.462438    5002 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="559377b2-e3fe-4889-995e-55127771d05e" volumeName="kubernetes.io/secret/559377b2-e3fe-4889-995e-55127771d05e-cert" seLinuxMountContext=""
Oct 30 00:31:44.463735 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.462452    5002 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="94a6f5e3-d553-487e-a6f1-8c2b0fc5c695" volumeName="kubernetes.io/empty-dir/94a6f5e3-d553-487e-a6f1-8c2b0fc5c695-data-volume" seLinuxMountContext=""
Oct 30 00:31:44.463735 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.462465    5002 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a707a5d7-313f-4bef-8baf-a99aeae40d54" volumeName="kubernetes.io/configmap/a707a5d7-313f-4bef-8baf-a99aeae40d54-multus-daemon-config" seLinuxMountContext=""
Oct 30 00:31:44.463735 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.462477    5002 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5dc56e7e-f3c1-43eb-a21b-8be6bb4706cf" volumeName="kubernetes.io/secret/5dc56e7e-f3c1-43eb-a21b-8be6bb4706cf-serving-cert" seLinuxMountContext=""
Oct 30 00:31:44.463735 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.462488    5002 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="80603987-09a1-4ca1-b8b4-3c9ac8d0efc6" volumeName="kubernetes.io/secret/80603987-09a1-4ca1-b8b4-3c9ac8d0efc6-original-pull-secret" seLinuxMountContext=""
Oct 30 00:31:44.463735 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.462500    5002 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="928e22b3-c3fe-453b-bbc8-a67accd2fcb4" volumeName="kubernetes.io/projected/928e22b3-c3fe-453b-bbc8-a67accd2fcb4-kube-api-access-vkng8" seLinuxMountContext=""
Oct 30 00:31:44.463735 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.462512    5002 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9aba3099-fe4c-4846-9a34-25db2c2fe593" volumeName="kubernetes.io/configmap/9aba3099-fe4c-4846-9a34-25db2c2fe593-telemetry-config" seLinuxMountContext=""
Oct 30 00:31:44.463735 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.462525    5002 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed" volumeName="kubernetes.io/projected/2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed-kube-api-access-84rlf" seLinuxMountContext=""
Oct 30 00:31:44.463735 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.462537    5002 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1881dcfe-c2da-4253-bca5-93f3526e14fa" volumeName="kubernetes.io/configmap/1881dcfe-c2da-4253-bca5-93f3526e14fa-samples-operator-config" seLinuxMountContext=""
Oct 30 00:31:44.463735 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.462549    5002 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6af42f7e-20f7-4b52-9759-0bf1af4d85b4" volumeName="kubernetes.io/configmap/6af42f7e-20f7-4b52-9759-0bf1af4d85b4-ovnkube-script-lib" seLinuxMountContext=""
Oct 30 00:31:44.463735 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.462560    5002 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="783c7a98-5ebf-44dd-8da9-797e71c28169" volumeName="kubernetes.io/projected/783c7a98-5ebf-44dd-8da9-797e71c28169-kube-api-access-vvgs7" seLinuxMountContext=""
Oct 30 00:31:44.463735 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.462572    5002 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e4ca0ea9-b57d-4d4c-be3a-2da5fc26a65d" volumeName="kubernetes.io/empty-dir/e4ca0ea9-b57d-4d4c-be3a-2da5fc26a65d-tmp" seLinuxMountContext=""
Oct 30 00:31:44.463735 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.462589    5002 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="88de9f3a-e7d0-400f-9091-46c0b89c1a3c" volumeName="kubernetes.io/projected/88de9f3a-e7d0-400f-9091-46c0b89c1a3c-kube-api-access-nxcbj" seLinuxMountContext=""
Oct 30 00:31:44.463735 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.462603    5002 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b874991e-9562-4848-b8f1-4b5a2b7cddc4" volumeName="kubernetes.io/projected/b874991e-9562-4848-b8f1-4b5a2b7cddc4-kube-api-access-zvtmg" seLinuxMountContext=""
Oct 30 00:31:44.463735 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.462614    5002 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e4ca0ea9-b57d-4d4c-be3a-2da5fc26a65d" volumeName="kubernetes.io/projected/e4ca0ea9-b57d-4d4c-be3a-2da5fc26a65d-kube-api-access-qzg5q" seLinuxMountContext=""
Oct 30 00:31:44.463735 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.462635    5002 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6af42f7e-20f7-4b52-9759-0bf1af4d85b4" volumeName="kubernetes.io/configmap/6af42f7e-20f7-4b52-9759-0bf1af4d85b4-env-overrides" seLinuxMountContext=""
Oct 30 00:31:44.463735 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.462646    5002 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9822c484-fd06-4e5e-94b4-ef0f5ade9a13" volumeName="kubernetes.io/projected/9822c484-fd06-4e5e-94b4-ef0f5ade9a13-kube-api-access-xbj4z" seLinuxMountContext=""
Oct 30 00:31:44.463735 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.462657    5002 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bba787ce-4e61-4a5a-a728-af73c68dfdd3" volumeName="kubernetes.io/configmap/bba787ce-4e61-4a5a-a728-af73c68dfdd3-service-ca-bundle" seLinuxMountContext=""
Oct 30 00:31:44.463735 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.462668    5002 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cea1b649-9c31-4434-9448-7f6652496830" volumeName="kubernetes.io/projected/cea1b649-9c31-4434-9448-7f6652496830-kube-api-access-2ngcm" seLinuxMountContext=""
Oct 30 00:31:44.463735 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.462680    5002 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d27ee5b1-d1aa-418c-9169-12cac37dcc2c" volumeName="kubernetes.io/projected/d27ee5b1-d1aa-418c-9169-12cac37dcc2c-kube-api-access-r998s" seLinuxMountContext=""
Oct 30 00:31:44.463735 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.462691    5002 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="078fd0e8-1393-45d0-baee-bbcd88095b2b" volumeName="kubernetes.io/projected/078fd0e8-1393-45d0-baee-bbcd88095b2b-kube-api-access-crppr" seLinuxMountContext=""
Oct 30 00:31:44.463735 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.462702    5002 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="14285dfa-dd38-490e-a486-a1fcfc3d0214" volumeName="kubernetes.io/configmap/14285dfa-dd38-490e-a486-a1fcfc3d0214-serviceca" seLinuxMountContext=""
Oct 30 00:31:44.463735 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.462713    5002 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="559377b2-e3fe-4889-995e-55127771d05e" volumeName="kubernetes.io/projected/559377b2-e3fe-4889-995e-55127771d05e-kube-api-access-zmzz7" seLinuxMountContext=""
Oct 30 00:31:44.463735 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.462726    5002 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5ed36f74-5c9e-408e-a7fb-2364144f054e" volumeName="kubernetes.io/configmap/5ed36f74-5c9e-408e-a7fb-2364144f054e-registry-certificates" seLinuxMountContext=""
Oct 30 00:31:44.463735 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.462738    5002 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="94a6f5e3-d553-487e-a6f1-8c2b0fc5c695" volumeName="kubernetes.io/projected/94a6f5e3-d553-487e-a6f1-8c2b0fc5c695-kube-api-access-wjms2" seLinuxMountContext=""
Oct 30 00:31:44.463735 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.462750    5002 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b874991e-9562-4848-b8f1-4b5a2b7cddc4" volumeName="kubernetes.io/configmap/b874991e-9562-4848-b8f1-4b5a2b7cddc4-config" seLinuxMountContext=""
Oct 30 00:31:44.463735 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.462762    5002 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b874991e-9562-4848-b8f1-4b5a2b7cddc4" volumeName="kubernetes.io/secret/b874991e-9562-4848-b8f1-4b5a2b7cddc4-serving-cert" seLinuxMountContext=""
Oct 30 00:31:44.463735 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.462773    5002 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c5c3a55b-e259-4550-9b7e-f666a28d217e" volumeName="kubernetes.io/projected/c5c3a55b-e259-4550-9b7e-f666a28d217e-kube-api-access-mqmpd" seLinuxMountContext=""
Oct 30 00:31:44.463735 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.462785    5002 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4606f172-d949-471e-842c-b340b50e179b" volumeName="kubernetes.io/projected/4606f172-d949-471e-842c-b340b50e179b-kube-api-access-6rl4f" seLinuxMountContext=""
Oct 30 00:31:44.463735 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.462796    5002 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="078fd0e8-1393-45d0-baee-bbcd88095b2b" volumeName="kubernetes.io/empty-dir/078fd0e8-1393-45d0-baee-bbcd88095b2b-tmp-dir" seLinuxMountContext=""
Oct 30 00:31:44.463735 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.462808    5002 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0d1b347f-c22e-489b-986d-d3b3e01fa36b" volumeName="kubernetes.io/secret/0d1b347f-c22e-489b-986d-d3b3e01fa36b-agent-certs" seLinuxMountContext=""
Oct 30 00:31:44.463735 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.462821    5002 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a707a5d7-313f-4bef-8baf-a99aeae40d54" volumeName="kubernetes.io/configmap/a707a5d7-313f-4bef-8baf-a99aeae40d54-cni-binary-copy" seLinuxMountContext=""
Oct 30 00:31:44.463735 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.462832    5002 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="af2d6f84-aea3-4698-be12-11bd6767caec" volumeName="kubernetes.io/projected/af2d6f84-aea3-4698-be12-11bd6767caec-kube-api-access-4d7j7" seLinuxMountContext=""
Oct 30 00:31:44.463735 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.462844    5002 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d7c5e976-596e-4cb5-81a9-b16262b3c85b" volumeName="kubernetes.io/configmap/d7c5e976-596e-4cb5-81a9-b16262b3c85b-nginx-conf" seLinuxMountContext=""
Oct 30 00:31:44.463735 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.462856    5002 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e4ca0ea9-b57d-4d4c-be3a-2da5fc26a65d" volumeName="kubernetes.io/configmap/e4ca0ea9-b57d-4d4c-be3a-2da5fc26a65d-service-ca-bundle" seLinuxMountContext=""
Oct 30 00:31:44.463735 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.462868    5002 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed" volumeName="kubernetes.io/empty-dir/2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed-ca-trust-extracted" seLinuxMountContext=""
Oct 30 00:31:44.463735 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.462879    5002 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5ed36f74-5c9e-408e-a7fb-2364144f054e" volumeName="kubernetes.io/empty-dir/5ed36f74-5c9e-408e-a7fb-2364144f054e-ca-trust-extracted" seLinuxMountContext=""
Oct 30 00:31:44.463735 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.462891    5002 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a707a5d7-313f-4bef-8baf-a99aeae40d54" volumeName="kubernetes.io/projected/a707a5d7-313f-4bef-8baf-a99aeae40d54-kube-api-access-h4cc6" seLinuxMountContext=""
Oct 30 00:31:44.463735 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.462902    5002 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c5c3a55b-e259-4550-9b7e-f666a28d217e" volumeName="kubernetes.io/configmap/c5c3a55b-e259-4550-9b7e-f666a28d217e-cni-sysctl-allowlist" seLinuxMountContext=""
Oct 30 00:31:44.463735 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.462915    5002 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4606f172-d949-471e-842c-b340b50e179b" volumeName="kubernetes.io/empty-dir/4606f172-d949-471e-842c-b340b50e179b-etc-tuned" seLinuxMountContext=""
Oct 30 00:31:44.463735 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.462927    5002 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="61903cf2-2b92-4f40-ab52-25509de587ef" volumeName="kubernetes.io/secret/61903cf2-2b92-4f40-ab52-25509de587ef-serving-cert" seLinuxMountContext=""
Oct 30 00:31:44.463735 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.462938    5002 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9aba3099-fe4c-4846-9a34-25db2c2fe593" volumeName="kubernetes.io/projected/9aba3099-fe4c-4846-9a34-25db2c2fe593-kube-api-access-2jjgr" seLinuxMountContext=""
Oct 30 00:31:44.463735 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.462950    5002 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 00:31:44.463735 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.462957    5002 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 00:31:44.465671 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.463784    5002 factory.go:153] Registering CRI-O factory
Oct 30 00:31:44.465671 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.463799    5002 factory.go:223] Registration of the crio container factory successfully
Oct 30 00:31:44.465671 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.463841    5002 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 00:31:44.465671 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.463849    5002 factory.go:55] Registering systemd factory
Oct 30 00:31:44.465671 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.463856    5002 factory.go:223] Registration of the systemd container factory successfully
Oct 30 00:31:44.465671 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.463870    5002 factory.go:103] Registering Raw factory
Oct 30 00:31:44.465671 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.463883    5002 manager.go:1199] Started watching for new ooms in manager
Oct 30 00:31:44.465671 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.464647    5002 manager.go:319] Starting recovery of all containers
Oct 30 00:31:44.473946 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.473917    5002 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 00:31:44.480507 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.480485    5002 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-g7c2w_5dc56e7e-f3c1-43eb-a21b-8be6bb4706cf/console-operator/1.log"
Oct 30 00:31:44.480769 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.480750    5002 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-g7c2w_5dc56e7e-f3c1-43eb-a21b-8be6bb4706cf/console-operator/1.log"
Oct 30 00:31:44.486486 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.486464    5002 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 00:31:44.486486 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.486487    5002 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 00:31:44.486639 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.486505    5002 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 00:31:44.486639 ip-10-0-128-7 kubenswrapper[5002]: E1030 00:31:44.486564    5002 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Oct 30 00:31:44.506608 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.506572    5002 generic.go:356] "Generic (PLEG): container finished" podID="c5c3a55b-e259-4550-9b7e-f666a28d217e" containerID="0d6fce3030a94ab80f89f6dd8b2466fc094db4bc0f801aad3abe83f650b258f9" exitCode=0
Oct 30 00:31:44.506608 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.506603    5002 generic.go:356] "Generic (PLEG): container finished" podID="c5c3a55b-e259-4550-9b7e-f666a28d217e" containerID="491a677a8fe80eed5b32874c6d29a09b4ced36528c56603427e807956d22e14a" exitCode=0
Oct 30 00:31:44.506608 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.506613    5002 generic.go:356] "Generic (PLEG): container finished" podID="c5c3a55b-e259-4550-9b7e-f666a28d217e" containerID="a86eb29daa94f829c573baa241891539cd0aff7b9834a1bac65d2b9267c79082" exitCode=0
Oct 30 00:31:44.506844 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.506622    5002 generic.go:356] "Generic (PLEG): container finished" podID="c5c3a55b-e259-4550-9b7e-f666a28d217e" containerID="ff59758a0d4edcf893f2a339ed6b9d1afe635bdb842b9d7bf4fac94408084cd7" exitCode=0
Oct 30 00:31:44.506844 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.506630    5002 generic.go:356] "Generic (PLEG): container finished" podID="c5c3a55b-e259-4550-9b7e-f666a28d217e" containerID="7eecf80608ddf0d75051dd83a9c697ebb3371db9c26c54e64a6bdbb819d9bfd4" exitCode=0
Oct 30 00:31:44.506844 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.506638    5002 generic.go:356] "Generic (PLEG): container finished" podID="c5c3a55b-e259-4550-9b7e-f666a28d217e" containerID="1a3e6d299f1bab1bdd66a93dc0a01da38b8a0fc3ac0dec8c3281e45a1cf57945" exitCode=0
Oct 30 00:31:44.515025 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.514969    5002 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Oct 30 00:31:44.527718 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.527692    5002 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 00:31:44.529033 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.529013    5002 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 00:31:44.529903 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.529421    5002 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 00:31:44.529903 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.529785    5002 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 00:31:44.540504 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.540483    5002 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-g7c2w_5dc56e7e-f3c1-43eb-a21b-8be6bb4706cf/console-operator/1.log"
Oct 30 00:31:44.540598 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.540517    5002 generic.go:356] "Generic (PLEG): container finished" podID="5dc56e7e-f3c1-43eb-a21b-8be6bb4706cf" containerID="b9a214f6a9f96ff4dcd1880bfe7e3bbf691313502185f22d37d7989d8110023a" exitCode=255
Oct 30 00:31:44.547974 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.547943    5002 generic.go:356] "Generic (PLEG): container finished" podID="70c88bf4a25fa0f049c2cc951e3cbb56" containerID="7375b6e950fa89b8e5c9eff221fc4e4c8532d97031eb86c570e03bfe63bb5b21" exitCode=0
Oct 30 00:31:44.565102 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.565083    5002 manager.go:324] Recovery completed
Oct 30 00:31:44.586038 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.586017    5002 cpu_manager.go:226] "Starting" policy="none"
Oct 30 00:31:44.586038 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.586034    5002 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 00:31:44.586174 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.586050    5002 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 00:31:44.586174 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.586163    5002 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Oct 30 00:31:44.586262 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.586173    5002 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Oct 30 00:31:44.586262 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.586190    5002 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Oct 30 00:31:44.586262 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.586195    5002 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Oct 30 00:31:44.586262 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.586200    5002 policy_none.go:50] "Start"
Oct 30 00:31:44.586262 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.586215    5002 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 00:31:44.586262 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.586224    5002 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 00:31:44.586471 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.586356    5002 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Oct 30 00:31:44.586471 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.586364    5002 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Oct 30 00:31:44.586471 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.586370    5002 policy_none.go:44] "Start"
Oct 30 00:31:44.587578 ip-10-0-128-7 kubenswrapper[5002]: E1030 00:31:44.587560    5002 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet"
Oct 30 00:31:44.590596 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.590565    5002 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 00:31:44.595044 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.595030    5002 manager.go:341] "Starting Device Plugin manager"
Oct 30 00:31:44.595731 ip-10-0-128-7 kubenswrapper[5002]: E1030 00:31:44.595713    5002 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 00:31:44.595776 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.595734    5002 server.go:87] "Starting device plugin registration server"
Oct 30 00:31:44.596054 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.596042    5002 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 00:31:44.596104 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.596057    5002 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 00:31:44.596163 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.596140    5002 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 00:31:44.596349 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.596303    5002 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 00:31:44.596431 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.596360    5002 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 00:31:44.596431 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.596368    5002 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 00:31:44.596525 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.596438    5002 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T00:31:44.596345924Z","UUID":"c03ebba0-ffe3-4e63-ba7c-88446d9a0aba","Handler":null,"Name":"","Endpoint":""}
Oct 30 00:31:44.597707 ip-10-0-128-7 kubenswrapper[5002]: E1030 00:31:44.597688    5002 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 00:31:44.597803 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.597788    5002 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 00:31:44.597843 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.597813    5002 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 00:31:44.697053 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.697023    5002 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 00:31:44.698322 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.698301    5002 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-7.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 00:31:44.698445 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.698333    5002 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-7.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 00:31:44.698445 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.698348    5002 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-7.ec2.internal" event="NodeHasSufficientPID"
Oct 30 00:31:44.698445 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.698416    5002 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-128-7.ec2.internal"
Oct 30 00:31:44.735897 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.735873    5002 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-128-7.ec2.internal"
Oct 30 00:31:44.736024 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.735942    5002 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-128-7.ec2.internal"
Oct 30 00:31:44.788462 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.788365    5002 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-7.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-128-7.ec2.internal"]
Oct 30 00:31:44.789018 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.788977    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-7.ec2.internal" event={"ID":"74c81293f9236d86a7e367354c3aa34d","Type":"ContainerStarted","Data":"47de7b78b4a22546f051141196d2e5cb33bdc3753050239027f020d7c3b47306"}
Oct 30 00:31:44.789148 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.789024    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-7.ec2.internal" event={"ID":"74c81293f9236d86a7e367354c3aa34d","Type":"ContainerStarted","Data":"c72433bacc952d570b3676b773c9b174fec945357ecf73c909e453c4a02d6577"}
Oct 30 00:31:44.789148 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.789044    5002 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-7.ec2.internal"
Oct 30 00:31:44.789148 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.789128    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-7.ec2.internal" event={"ID":"70c88bf4a25fa0f049c2cc951e3cbb56","Type":"ContainerStarted","Data":"b7d66edb9a643c62c5f736c2f427a27e5f0213c1ef36ea340207f7db63f6399c"}
Oct 30 00:31:44.789284 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.789150    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-7.ec2.internal" event={"ID":"70c88bf4a25fa0f049c2cc951e3cbb56","Type":"ContainerDied","Data":"7375b6e950fa89b8e5c9eff221fc4e4c8532d97031eb86c570e03bfe63bb5b21"}
Oct 30 00:31:44.789284 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.789168    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-7.ec2.internal" event={"ID":"70c88bf4a25fa0f049c2cc951e3cbb56","Type":"ContainerStarted","Data":"453fcdc0a3bdbd70d0b6a86ba356202ebd1abd1c2e08b95bc5fd8533b3550ac8"}
Oct 30 00:31:44.789284 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.789232    5002 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-7.ec2.internal"
Oct 30 00:31:44.816638 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.816618    5002 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 00:31:44.816719 ip-10-0-128-7 kubenswrapper[5002]: E1030 00:31:44.816660    5002 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-128-7.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-7.ec2.internal"
Oct 30 00:31:44.817189 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.817173    5002 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 00:31:44.817280 ip-10-0-128-7 kubenswrapper[5002]: E1030 00:31:44.817216    5002 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-128-7.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-7.ec2.internal"
Oct 30 00:31:44.917608 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.917573    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/70c88bf4a25fa0f049c2cc951e3cbb56-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-7.ec2.internal\" (UID: \"70c88bf4a25fa0f049c2cc951e3cbb56\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-7.ec2.internal"
Oct 30 00:31:44.917738 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.917624    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/70c88bf4a25fa0f049c2cc951e3cbb56-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-7.ec2.internal\" (UID: \"70c88bf4a25fa0f049c2cc951e3cbb56\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-7.ec2.internal"
Oct 30 00:31:44.917738 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:44.917646    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/74c81293f9236d86a7e367354c3aa34d-config\") pod \"kube-apiserver-proxy-ip-10-0-128-7.ec2.internal\" (UID: \"74c81293f9236d86a7e367354c3aa34d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-7.ec2.internal"
Oct 30 00:31:45.018122 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.018092    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/70c88bf4a25fa0f049c2cc951e3cbb56-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-7.ec2.internal\" (UID: \"70c88bf4a25fa0f049c2cc951e3cbb56\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-7.ec2.internal"
Oct 30 00:31:45.018288 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.018143    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/70c88bf4a25fa0f049c2cc951e3cbb56-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-7.ec2.internal\" (UID: \"70c88bf4a25fa0f049c2cc951e3cbb56\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-7.ec2.internal"
Oct 30 00:31:45.018288 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.018168    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/74c81293f9236d86a7e367354c3aa34d-config\") pod \"kube-apiserver-proxy-ip-10-0-128-7.ec2.internal\" (UID: \"74c81293f9236d86a7e367354c3aa34d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-7.ec2.internal"
Oct 30 00:31:45.018288 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.018101    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/70c88bf4a25fa0f049c2cc951e3cbb56-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-7.ec2.internal\" (UID: \"70c88bf4a25fa0f049c2cc951e3cbb56\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-7.ec2.internal"
Oct 30 00:31:45.018288 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.018216    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/74c81293f9236d86a7e367354c3aa34d-config\") pod \"kube-apiserver-proxy-ip-10-0-128-7.ec2.internal\" (UID: \"74c81293f9236d86a7e367354c3aa34d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-7.ec2.internal"
Oct 30 00:31:45.018288 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.018234    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/70c88bf4a25fa0f049c2cc951e3cbb56-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-7.ec2.internal\" (UID: \"70c88bf4a25fa0f049c2cc951e3cbb56\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-7.ec2.internal"
Oct 30 00:31:45.445894 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.445859    5002 apiserver.go:52] "Watching apiserver"
Oct 30 00:31:45.519519 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.519488    5002 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 00:31:45.520317 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.520287    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-lxg8k","openshift-ingress/router-default-85f7b79cf7-h2tpp","openshift-network-diagnostics/network-check-source-6f78d59f99-jg5h6","openshift-network-diagnostics/network-check-target-lngrr","openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-246g2","kube-system/kube-apiserver-proxy-ip-10-0-128-7.ec2.internal","openshift-dns/node-resolver-6gjtz","kube-system/kas-connection-checker-685895c7c9-f2b4d","openshift-console-operator/console-operator-7cd56b8dbc-g7c2w","kube-system/kas-connection-checker-685895c7c9-fbdvl","kube-system/kas-connection-checker-685895c7c9-cqjlz","openshift-multus/multus-additional-cni-plugins-zvcpp","openshift-ingress-canary/ingress-canary-wqf5x","openshift-kube-storage-version-migrator/migrator-66df5d496b-g2dlc","openshift-monitoring/cluster-monitoring-operator-884576c57-kjsj6","openshift-image-registry/node-ca-8b6d6","openshift-image-registry/image-registry-6c69f88f85-88bkg","openshift-insights/insights-runtime-extractor-tbjdq","openshift-multus/multus-h4hbh","openshift-multus/network-metrics-daemon-z58bb","openshift-network-console/networking-console-plugin-86c478bb88-zqdbk","openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-2nnk8","openshift-insights/periodic-gathering-zzlf6-kvglg","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v24xq","openshift-cluster-node-tuning-operator/tuned-5hvvb","openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-sxknf","openshift-dns/dns-default-btj6v","openshift-network-operator/iptables-alerter-kph7f","openshift-image-registry/image-registry-6d4f74cbd-njpvm","openshift-service-ca-operator/service-ca-operator-544d749c8c-bjzgs","kube-system/global-pull-secret-syncer-tz6g8","openshift-insights/insights-operator-7c4986d449-sklth","openshift-kube-storage-version-migrator/migrator-66df5d496b-khqcq","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-7.ec2.internal","kube-system/konnectivity-agent-m5qgr"]
Oct 30 00:31:45.525234 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.525201    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-kjsj6"
Oct 30 00:31:45.525482 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.525454    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6d4f74cbd-njpvm"
Oct 30 00:31:45.526954 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.526930    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-85f7b79cf7-h2tpp"
Oct 30 00:31:45.527699 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.527679    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-2nnk8"
Oct 30 00:31:45.528107 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.528091    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-zqdbk"
Oct 30 00:31:45.530330 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.530310    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z58bb"
Oct 30 00:31:45.534096 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.534076    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6c69f88f85-88bkg"
Oct 30 00:31:45.534564 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.534551    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-btj6v"
Oct 30 00:31:45.536880 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.536826    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-tbjdq"
Oct 30 00:31:45.538216 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.538200    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Oct 30 00:31:45.538708 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.538690    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 30 00:31:45.538780 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.538692    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Oct 30 00:31:45.538780 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.538715    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Oct 30 00:31:45.538780 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.538724    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\""
Oct 30 00:31:45.538780 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.538724    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\""
Oct 30 00:31:45.538902 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.538793    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 30 00:31:45.539713 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.539699    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Oct 30 00:31:45.539913 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.539897    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Oct 30 00:31:45.542254 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.542237    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Oct 30 00:31:45.552630 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.552611    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Oct 30 00:31:45.553065 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.553043    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Oct 30 00:31:45.553252 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.553235    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Oct 30 00:31:45.553344 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.553053    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Oct 30 00:31:45.554125 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.554104    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 00:31:45.554293 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.554278    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 30 00:31:45.554350 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.554331    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 00:31:45.554463 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.554447    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 00:31:45.554761 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.554737    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 00:31:45.554991 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.554976    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Oct 30 00:31:45.555049 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.555007    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Oct 30 00:31:45.555542 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.555525    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Oct 30 00:31:45.555542 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.555538    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Oct 30 00:31:45.555655 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.555605    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Oct 30 00:31:45.555655 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.555640    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Oct 30 00:31:45.555752 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.555720    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Oct 30 00:31:45.558062 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.558047    5002 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-7.ec2.internal"
Oct 30 00:31:45.561824 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.561810    5002 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 00:31:45.567056 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.567039    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-ttql7\""
Oct 30 00:31:45.567132 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.567079    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Oct 30 00:31:45.567132 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.567089    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Oct 30 00:31:45.567132 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.567128    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Oct 30 00:31:45.567287 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.567185    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 30 00:31:45.567287 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.567221    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Oct 30 00:31:45.567287 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.567282    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 30 00:31:45.567441 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.567307    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Oct 30 00:31:45.567532 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.567516    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-fch7k\""
Oct 30 00:31:45.567597 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.567534    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 00:31:45.567597 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.567552    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-d7h4r\""
Oct 30 00:31:45.567597 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.567581    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-rw2kd\""
Oct 30 00:31:45.567735 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.567648    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-h6pr4\""
Oct 30 00:31:45.567735 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.567695    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Oct 30 00:31:45.567831 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.567765    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-xlz4l\""
Oct 30 00:31:45.567831 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.567773    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-mt5kv\""
Oct 30 00:31:45.567924 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.567839    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 30 00:31:45.567924 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.567831    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 00:31:45.567924 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.567845    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 00:31:45.567924 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.567793    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-xrsjq\""
Oct 30 00:31:45.568094 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.567926    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Oct 30 00:31:45.568094 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.567926    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Oct 30 00:31:45.568094 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.567939    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-9hmc7\""
Oct 30 00:31:45.568094 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.567959    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 00:31:45.568094 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.567959    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Oct 30 00:31:45.568094 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.568059    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 30 00:31:45.568094 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.568067    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 00:31:45.568094 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.568077    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 00:31:45.568094 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.568062    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 00:31:45.568094 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.568092    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-j8zc4\""
Oct 30 00:31:45.568547 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.568093    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-ngd4v\""
Oct 30 00:31:45.568547 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.568082    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Oct 30 00:31:45.568547 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.568096    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-nrfgz\""
Oct 30 00:31:45.568547 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.568170    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Oct 30 00:31:45.568547 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.568176    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 00:31:45.568547 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.568171    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 00:31:45.568547 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.568177    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-z22kq\""
Oct 30 00:31:45.568547 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.568192    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Oct 30 00:31:45.568547 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.568216    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 00:31:45.568547 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.568224    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\""
Oct 30 00:31:45.568547 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.568232    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Oct 30 00:31:45.568547 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.568235    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Oct 30 00:31:45.568547 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.568251    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 00:31:45.568547 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.568251    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 00:31:45.568547 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.568273    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 00:31:45.568547 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.568287    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 00:31:45.568547 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.568296    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 00:31:45.568547 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.568291    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Oct 30 00:31:45.568547 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.568310    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 30 00:31:45.568547 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.568310    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Oct 30 00:31:45.568547 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.568321    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Oct 30 00:31:45.568547 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.568337    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 30 00:31:45.568547 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.568363    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Oct 30 00:31:45.568547 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.568364    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Oct 30 00:31:45.568547 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.568412    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Oct 30 00:31:45.568547 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.568417    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 00:31:45.568547 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.568433    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 00:31:45.568547 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.568479    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 00:31:45.568547 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.568533    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 00:31:45.568547 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.568536    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 00:31:45.569687 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.568596    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 00:31:45.569687 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.568624    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-8g889\""
Oct 30 00:31:45.569687 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.568669    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 30 00:31:45.569687 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.568714    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-6gpzw\""
Oct 30 00:31:45.569687 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.568826    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 00:31:45.569687 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.568898    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 00:31:45.569687 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.568954    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 00:31:45.569687 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.569009    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 00:31:45.569687 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.569091    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 00:31:45.569687 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.569174    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-6rb57\""
Oct 30 00:31:45.569687 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.569243    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-8njw4\""
Oct 30 00:31:45.569687 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.569301    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 00:31:45.569687 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.569474    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-f65k2\""
Oct 30 00:31:45.569687 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.569495    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-vk9xp\""
Oct 30 00:31:45.569687 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.569529    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 00:31:45.582356 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.582339    5002 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 00:31:45.582417 ip-10-0-128-7 kubenswrapper[5002]: E1030 00:31:45.582400    5002 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-128-7.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-7.ec2.internal"
Oct 30 00:31:45.621022 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.620993    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/e4ca0ea9-b57d-4d4c-be3a-2da5fc26a65d-snapshots\") pod \"insights-operator-7c4986d449-sklth\" (UID: \"e4ca0ea9-b57d-4d4c-be3a-2da5fc26a65d\") " pod="openshift-insights/insights-operator-7c4986d449-sklth"
Oct 30 00:31:45.621166 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.621029    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjmbq\" (UniqueName: \"kubernetes.io/projected/a3d234b3-76d4-47d5-98a4-dafd83477b17-kube-api-access-bjmbq\") pod \"router-default-85f7b79cf7-h2tpp\" (UID: \"a3d234b3-76d4-47d5-98a4-dafd83477b17\") " pod="openshift-ingress/router-default-85f7b79cf7-h2tpp"
Oct 30 00:31:45.621166 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.621053    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6af42f7e-20f7-4b52-9759-0bf1af4d85b4-ovnkube-script-lib\") pod \"ovnkube-node-lxg8k\" (UID: \"6af42f7e-20f7-4b52-9759-0bf1af4d85b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxg8k"
Oct 30 00:31:45.621166 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.621081    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4606f172-d949-471e-842c-b340b50e179b-etc-sysconfig\") pod \"tuned-5hvvb\" (UID: \"4606f172-d949-471e-842c-b340b50e179b\") " pod="openshift-cluster-node-tuning-operator/tuned-5hvvb"
Oct 30 00:31:45.621307 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.621175    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/e4ca0ea9-b57d-4d4c-be3a-2da5fc26a65d-snapshots\") pod \"insights-operator-7c4986d449-sklth\" (UID: \"e4ca0ea9-b57d-4d4c-be3a-2da5fc26a65d\") " pod="openshift-insights/insights-operator-7c4986d449-sklth"
Oct 30 00:31:45.621307 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.621242    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4606f172-d949-471e-842c-b340b50e179b-etc-kubernetes\") pod \"tuned-5hvvb\" (UID: \"4606f172-d949-471e-842c-b340b50e179b\") " pod="openshift-cluster-node-tuning-operator/tuned-5hvvb"
Oct 30 00:31:45.621307 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.621292    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4606f172-d949-471e-842c-b340b50e179b-run\") pod \"tuned-5hvvb\" (UID: \"4606f172-d949-471e-842c-b340b50e179b\") " pod="openshift-cluster-node-tuning-operator/tuned-5hvvb"
Oct 30 00:31:45.621544 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.621324    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/61903cf2-2b92-4f40-ab52-25509de587ef-config\") pod \"service-ca-operator-544d749c8c-bjzgs\" (UID: \"61903cf2-2b92-4f40-ab52-25509de587ef\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-bjzgs"
Oct 30 00:31:45.621544 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.621348    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c1e4fab8-6072-4fcd-9624-a60dee7aed86-iptables-alerter-script\") pod \"iptables-alerter-kph7f\" (UID: \"c1e4fab8-6072-4fcd-9624-a60dee7aed86\") " pod="openshift-network-operator/iptables-alerter-kph7f"
Oct 30 00:31:45.621544 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.621369    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5ed36f74-5c9e-408e-a7fb-2364144f054e-ca-trust-extracted\") pod \"image-registry-6c69f88f85-88bkg\" (UID: \"5ed36f74-5c9e-408e-a7fb-2364144f054e\") " pod="openshift-image-registry/image-registry-6c69f88f85-88bkg"
Oct 30 00:31:45.621544 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.621406    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a707a5d7-313f-4bef-8baf-a99aeae40d54-host-var-lib-kubelet\") pod \"multus-h4hbh\" (UID: \"a707a5d7-313f-4bef-8baf-a99aeae40d54\") " pod="openshift-multus/multus-h4hbh"
Oct 30 00:31:45.621544 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.621424    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5dc56e7e-f3c1-43eb-a21b-8be6bb4706cf-trusted-ca\") pod \"console-operator-7cd56b8dbc-g7c2w\" (UID: \"5dc56e7e-f3c1-43eb-a21b-8be6bb4706cf\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-g7c2w"
Oct 30 00:31:45.621544 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.621446    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5ed36f74-5c9e-408e-a7fb-2364144f054e-ca-trust-extracted\") pod \"image-registry-6c69f88f85-88bkg\" (UID: \"5ed36f74-5c9e-408e-a7fb-2364144f054e\") " pod="openshift-image-registry/image-registry-6c69f88f85-88bkg"
Oct 30 00:31:45.621544 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.621447    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rl4f\" (UniqueName: \"kubernetes.io/projected/4606f172-d949-471e-842c-b340b50e179b-kube-api-access-6rl4f\") pod \"tuned-5hvvb\" (UID: \"4606f172-d949-471e-842c-b340b50e179b\") " pod="openshift-cluster-node-tuning-operator/tuned-5hvvb"
Oct 30 00:31:45.621544 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.621490    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0d1b347f-c22e-489b-986d-d3b3e01fa36b-konnectivity-ca\") pod \"konnectivity-agent-m5qgr\" (UID: \"0d1b347f-c22e-489b-986d-d3b3e01fa36b\") " pod="kube-system/konnectivity-agent-m5qgr"
Oct 30 00:31:45.621900 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.621627    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d7c5e976-596e-4cb5-81a9-b16262b3c85b-nginx-conf\") pod \"networking-console-plugin-86c478bb88-zqdbk\" (UID: \"d7c5e976-596e-4cb5-81a9-b16262b3c85b\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-zqdbk"
Oct 30 00:31:45.621900 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.621639    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6af42f7e-20f7-4b52-9759-0bf1af4d85b4-ovnkube-script-lib\") pod \"ovnkube-node-lxg8k\" (UID: \"6af42f7e-20f7-4b52-9759-0bf1af4d85b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxg8k"
Oct 30 00:31:45.621900 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.621661    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6af42f7e-20f7-4b52-9759-0bf1af4d85b4-host-run-netns\") pod \"ovnkube-node-lxg8k\" (UID: \"6af42f7e-20f7-4b52-9759-0bf1af4d85b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxg8k"
Oct 30 00:31:45.621900 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.621694    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0d1b347f-c22e-489b-986d-d3b3e01fa36b-konnectivity-ca\") pod \"konnectivity-agent-m5qgr\" (UID: \"0d1b347f-c22e-489b-986d-d3b3e01fa36b\") " pod="kube-system/konnectivity-agent-m5qgr"
Oct 30 00:31:45.621900 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.621697    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/61903cf2-2b92-4f40-ab52-25509de587ef-config\") pod \"service-ca-operator-544d749c8c-bjzgs\" (UID: \"61903cf2-2b92-4f40-ab52-25509de587ef\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-bjzgs"
Oct 30 00:31:45.621900 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.621732    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c1e4fab8-6072-4fcd-9624-a60dee7aed86-iptables-alerter-script\") pod \"iptables-alerter-kph7f\" (UID: \"c1e4fab8-6072-4fcd-9624-a60dee7aed86\") " pod="openshift-network-operator/iptables-alerter-kph7f"
Oct 30 00:31:45.621900 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.621743    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lbm4\" (UniqueName: \"kubernetes.io/projected/6af42f7e-20f7-4b52-9759-0bf1af4d85b4-kube-api-access-5lbm4\") pod \"ovnkube-node-lxg8k\" (UID: \"6af42f7e-20f7-4b52-9759-0bf1af4d85b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxg8k"
Oct 30 00:31:45.621900 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.621777    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzc86\" (UniqueName: \"kubernetes.io/projected/bba787ce-4e61-4a5a-a728-af73c68dfdd3-kube-api-access-fzc86\") pod \"periodic-gathering-zzlf6-kvglg\" (UID: \"bba787ce-4e61-4a5a-a728-af73c68dfdd3\") " pod="openshift-insights/periodic-gathering-zzlf6-kvglg"
Oct 30 00:31:45.621900 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.621803    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6af42f7e-20f7-4b52-9759-0bf1af4d85b4-run-ovn\") pod \"ovnkube-node-lxg8k\" (UID: \"6af42f7e-20f7-4b52-9759-0bf1af4d85b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxg8k"
Oct 30 00:31:45.621900 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.621830    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4606f172-d949-471e-842c-b340b50e179b-etc-tuned\") pod \"tuned-5hvvb\" (UID: \"4606f172-d949-471e-842c-b340b50e179b\") " pod="openshift-cluster-node-tuning-operator/tuned-5hvvb"
Oct 30 00:31:45.621900 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.621831    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5dc56e7e-f3c1-43eb-a21b-8be6bb4706cf-trusted-ca\") pod \"console-operator-7cd56b8dbc-g7c2w\" (UID: \"5dc56e7e-f3c1-43eb-a21b-8be6bb4706cf\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-g7c2w"
Oct 30 00:31:45.621900 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.621857    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9aba3099-fe4c-4846-9a34-25db2c2fe593-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-kjsj6\" (UID: \"9aba3099-fe4c-4846-9a34-25db2c2fe593\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-kjsj6"
Oct 30 00:31:45.621900 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.621878    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d7c5e976-596e-4cb5-81a9-b16262b3c85b-nginx-conf\") pod \"networking-console-plugin-86c478bb88-zqdbk\" (UID: \"d7c5e976-596e-4cb5-81a9-b16262b3c85b\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-zqdbk"
Oct 30 00:31:45.621900 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.621883    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a707a5d7-313f-4bef-8baf-a99aeae40d54-host-var-lib-cni-bin\") pod \"multus-h4hbh\" (UID: \"a707a5d7-313f-4bef-8baf-a99aeae40d54\") " pod="openshift-multus/multus-h4hbh"
Oct 30 00:31:45.622354 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.621924    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/251cf359-f9f5-44ae-8ba9-42bf0cabb1ab-config-volume\") pod \"dns-default-btj6v\" (UID: \"251cf359-f9f5-44ae-8ba9-42bf0cabb1ab\") " pod="openshift-dns/dns-default-btj6v"
Oct 30 00:31:45.622354 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.621924    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4606f172-d949-471e-842c-b340b50e179b-etc-tuned\") pod \"tuned-5hvvb\" (UID: \"4606f172-d949-471e-842c-b340b50e179b\") " pod="openshift-cluster-node-tuning-operator/tuned-5hvvb"
Oct 30 00:31:45.622354 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.621958    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a707a5d7-313f-4bef-8baf-a99aeae40d54-host-run-k8s-cni-cncf-io\") pod \"multus-h4hbh\" (UID: \"a707a5d7-313f-4bef-8baf-a99aeae40d54\") " pod="openshift-multus/multus-h4hbh"
Oct 30 00:31:45.622354 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.622002    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a707a5d7-313f-4bef-8baf-a99aeae40d54-multus-conf-dir\") pod \"multus-h4hbh\" (UID: \"a707a5d7-313f-4bef-8baf-a99aeae40d54\") " pod="openshift-multus/multus-h4hbh"
Oct 30 00:31:45.622354 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.622035    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmzz7\" (UniqueName: \"kubernetes.io/projected/559377b2-e3fe-4889-995e-55127771d05e-kube-api-access-zmzz7\") pod \"ingress-canary-wqf5x\" (UID: \"559377b2-e3fe-4889-995e-55127771d05e\") " pod="openshift-ingress-canary/ingress-canary-wqf5x"
Oct 30 00:31:45.622354 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.622076    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6af42f7e-20f7-4b52-9759-0bf1af4d85b4-ovn-node-metrics-cert\") pod \"ovnkube-node-lxg8k\" (UID: \"6af42f7e-20f7-4b52-9759-0bf1af4d85b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxg8k"
Oct 30 00:31:45.622354 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.622133    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c5c3a55b-e259-4550-9b7e-f666a28d217e-cni-binary-copy\") pod \"multus-additional-cni-plugins-zvcpp\" (UID: \"c5c3a55b-e259-4550-9b7e-f666a28d217e\") " pod="openshift-multus/multus-additional-cni-plugins-zvcpp"
Oct 30 00:31:45.622354 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.622154    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/251cf359-f9f5-44ae-8ba9-42bf0cabb1ab-config-volume\") pod \"dns-default-btj6v\" (UID: \"251cf359-f9f5-44ae-8ba9-42bf0cabb1ab\") " pod="openshift-dns/dns-default-btj6v"
Oct 30 00:31:45.622354 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.622217    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a707a5d7-313f-4bef-8baf-a99aeae40d54-etc-kubernetes\") pod \"multus-h4hbh\" (UID: \"a707a5d7-313f-4bef-8baf-a99aeae40d54\") " pod="openshift-multus/multus-h4hbh"
Oct 30 00:31:45.622354 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.622265    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5ed36f74-5c9e-408e-a7fb-2364144f054e-registry-tls\") pod \"image-registry-6c69f88f85-88bkg\" (UID: \"5ed36f74-5c9e-408e-a7fb-2364144f054e\") " pod="openshift-image-registry/image-registry-6c69f88f85-88bkg"
Oct 30 00:31:45.622354 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.622323    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5ed36f74-5c9e-408e-a7fb-2364144f054e-bound-sa-token\") pod \"image-registry-6c69f88f85-88bkg\" (UID: \"5ed36f74-5c9e-408e-a7fb-2364144f054e\") " pod="openshift-image-registry/image-registry-6c69f88f85-88bkg"
Oct 30 00:31:45.622771 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.622357    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crppr\" (UniqueName: \"kubernetes.io/projected/078fd0e8-1393-45d0-baee-bbcd88095b2b-kube-api-access-crppr\") pod \"node-resolver-6gjtz\" (UID: \"078fd0e8-1393-45d0-baee-bbcd88095b2b\") " pod="openshift-dns/node-resolver-6gjtz"
Oct 30 00:31:45.622771 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.622400    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a707a5d7-313f-4bef-8baf-a99aeae40d54-hostroot\") pod \"multus-h4hbh\" (UID: \"a707a5d7-313f-4bef-8baf-a99aeae40d54\") " pod="openshift-multus/multus-h4hbh"
Oct 30 00:31:45.622771 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.622425    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jzc7\" (UniqueName: \"kubernetes.io/projected/c1e4fab8-6072-4fcd-9624-a60dee7aed86-kube-api-access-2jzc7\") pod \"iptables-alerter-kph7f\" (UID: \"c1e4fab8-6072-4fcd-9624-a60dee7aed86\") " pod="openshift-network-operator/iptables-alerter-kph7f"
Oct 30 00:31:45.622771 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.622450    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbj4z\" (UniqueName: \"kubernetes.io/projected/9822c484-fd06-4e5e-94b4-ef0f5ade9a13-kube-api-access-xbj4z\") pod \"kas-connection-checker-685895c7c9-cqjlz\" (UID: \"9822c484-fd06-4e5e-94b4-ef0f5ade9a13\") " pod="kube-system/kas-connection-checker-685895c7c9-cqjlz"
Oct 30 00:31:45.622771 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.622472    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c5c3a55b-e259-4550-9b7e-f666a28d217e-cni-binary-copy\") pod \"multus-additional-cni-plugins-zvcpp\" (UID: \"c5c3a55b-e259-4550-9b7e-f666a28d217e\") " pod="openshift-multus/multus-additional-cni-plugins-zvcpp"
Oct 30 00:31:45.622771 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.622479    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c5c3a55b-e259-4550-9b7e-f666a28d217e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zvcpp\" (UID: \"c5c3a55b-e259-4550-9b7e-f666a28d217e\") " pod="openshift-multus/multus-additional-cni-plugins-zvcpp"
Oct 30 00:31:45.622771 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.622526    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6llvc\" (UniqueName: \"kubernetes.io/projected/14285dfa-dd38-490e-a486-a1fcfc3d0214-kube-api-access-6llvc\") pod \"node-ca-8b6d6\" (UID: \"14285dfa-dd38-490e-a486-a1fcfc3d0214\") " pod="openshift-image-registry/node-ca-8b6d6"
Oct 30 00:31:45.622771 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.622535    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6af42f7e-20f7-4b52-9759-0bf1af4d85b4-ovn-node-metrics-cert\") pod \"ovnkube-node-lxg8k\" (UID: \"6af42f7e-20f7-4b52-9759-0bf1af4d85b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxg8k"
Oct 30 00:31:45.622771 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.622560    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6af42f7e-20f7-4b52-9759-0bf1af4d85b4-host-slash\") pod \"ovnkube-node-lxg8k\" (UID: \"6af42f7e-20f7-4b52-9759-0bf1af4d85b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxg8k"
Oct 30 00:31:45.622771 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.622586    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r998s\" (UniqueName: \"kubernetes.io/projected/d27ee5b1-d1aa-418c-9169-12cac37dcc2c-kube-api-access-r998s\") pod \"network-check-target-lngrr\" (UID: \"d27ee5b1-d1aa-418c-9169-12cac37dcc2c\") " pod="openshift-network-diagnostics/network-check-target-lngrr"
Oct 30 00:31:45.622771 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.622609    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4606f172-d949-471e-842c-b340b50e179b-etc-sysctl-d\") pod \"tuned-5hvvb\" (UID: \"4606f172-d949-471e-842c-b340b50e179b\") " pod="openshift-cluster-node-tuning-operator/tuned-5hvvb"
Oct 30 00:31:45.622771 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.622635    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed-image-registry-private-configuration\") pod \"image-registry-6d4f74cbd-njpvm\" (UID: \"2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed\") " pod="openshift-image-registry/image-registry-6d4f74cbd-njpvm"
Oct 30 00:31:45.622771 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.622673    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/251cf359-f9f5-44ae-8ba9-42bf0cabb1ab-metrics-tls\") pod \"dns-default-btj6v\" (UID: \"251cf359-f9f5-44ae-8ba9-42bf0cabb1ab\") " pod="openshift-dns/dns-default-btj6v"
Oct 30 00:31:45.622771 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.622727    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-566pb\" (UniqueName: \"kubernetes.io/projected/1881dcfe-c2da-4253-bca5-93f3526e14fa-kube-api-access-566pb\") pod \"cluster-samples-operator-6ddb6c8b94-2nnk8\" (UID: \"1881dcfe-c2da-4253-bca5-93f3526e14fa\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-2nnk8"
Oct 30 00:31:45.622771 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.622757    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bba787ce-4e61-4a5a-a728-af73c68dfdd3-service-ca-bundle\") pod \"periodic-gathering-zzlf6-kvglg\" (UID: \"bba787ce-4e61-4a5a-a728-af73c68dfdd3\") " pod="openshift-insights/periodic-gathering-zzlf6-kvglg"
Oct 30 00:31:45.623432 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.622819    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4606f172-d949-471e-842c-b340b50e179b-etc-modprobe-d\") pod \"tuned-5hvvb\" (UID: \"4606f172-d949-471e-842c-b340b50e179b\") " pod="openshift-cluster-node-tuning-operator/tuned-5hvvb"
Oct 30 00:31:45.623432 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.622866    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed-image-registry-private-configuration\") pod \"image-registry-6d4f74cbd-njpvm\" (UID: \"2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed\") " pod="openshift-image-registry/image-registry-6d4f74cbd-njpvm"
Oct 30 00:31:45.623432 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.622878    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4606f172-d949-471e-842c-b340b50e179b-sys\") pod \"tuned-5hvvb\" (UID: \"4606f172-d949-471e-842c-b340b50e179b\") " pod="openshift-cluster-node-tuning-operator/tuned-5hvvb"
Oct 30 00:31:45.623432 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.622887    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bba787ce-4e61-4a5a-a728-af73c68dfdd3-service-ca-bundle\") pod \"periodic-gathering-zzlf6-kvglg\" (UID: \"bba787ce-4e61-4a5a-a728-af73c68dfdd3\") " pod="openshift-insights/periodic-gathering-zzlf6-kvglg"
Oct 30 00:31:45.623432 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.622924    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed-registry-tls\") pod \"image-registry-6d4f74cbd-njpvm\" (UID: \"2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed\") " pod="openshift-image-registry/image-registry-6d4f74cbd-njpvm"
Oct 30 00:31:45.623432 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.622967    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhs2g\" (UniqueName: \"kubernetes.io/projected/9b2b3b14-8027-489a-88f0-d056226c2f67-kube-api-access-bhs2g\") pod \"network-check-source-6f78d59f99-jg5h6\" (UID: \"9b2b3b14-8027-489a-88f0-d056226c2f67\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-jg5h6"
Oct 30 00:31:45.623432 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.622990    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kpm2\" (UniqueName: \"kubernetes.io/projected/251cf359-f9f5-44ae-8ba9-42bf0cabb1ab-kube-api-access-6kpm2\") pod \"dns-default-btj6v\" (UID: \"251cf359-f9f5-44ae-8ba9-42bf0cabb1ab\") " pod="openshift-dns/dns-default-btj6v"
Oct 30 00:31:45.623432 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.623008    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4cc6\" (UniqueName: \"kubernetes.io/projected/a707a5d7-313f-4bef-8baf-a99aeae40d54-kube-api-access-h4cc6\") pod \"multus-h4hbh\" (UID: \"a707a5d7-313f-4bef-8baf-a99aeae40d54\") " pod="openshift-multus/multus-h4hbh"
Oct 30 00:31:45.623432 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.623031    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c1e4fab8-6072-4fcd-9624-a60dee7aed86-host-slash\") pod \"iptables-alerter-kph7f\" (UID: \"c1e4fab8-6072-4fcd-9624-a60dee7aed86\") " pod="openshift-network-operator/iptables-alerter-kph7f"
Oct 30 00:31:45.623432 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.623047    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/559377b2-e3fe-4889-995e-55127771d05e-cert\") pod \"ingress-canary-wqf5x\" (UID: \"559377b2-e3fe-4889-995e-55127771d05e\") " pod="openshift-ingress-canary/ingress-canary-wqf5x"
Oct 30 00:31:45.623432 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.623064    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4606f172-d949-471e-842c-b340b50e179b-lib-modules\") pod \"tuned-5hvvb\" (UID: \"4606f172-d949-471e-842c-b340b50e179b\") " pod="openshift-cluster-node-tuning-operator/tuned-5hvvb"
Oct 30 00:31:45.623432 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.623080    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/88de9f3a-e7d0-400f-9091-46c0b89c1a3c-etc-selinux\") pod \"aws-ebs-csi-driver-node-v24xq\" (UID: \"88de9f3a-e7d0-400f-9091-46c0b89c1a3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v24xq"
Oct 30 00:31:45.623432 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.623112    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6af42f7e-20f7-4b52-9759-0bf1af4d85b4-host-cni-bin\") pod \"ovnkube-node-lxg8k\" (UID: \"6af42f7e-20f7-4b52-9759-0bf1af4d85b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxg8k"
Oct 30 00:31:45.623432 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.623200    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/14285dfa-dd38-490e-a486-a1fcfc3d0214-host\") pod \"node-ca-8b6d6\" (UID: \"14285dfa-dd38-490e-a486-a1fcfc3d0214\") " pod="openshift-image-registry/node-ca-8b6d6"
Oct 30 00:31:45.623432 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.623277    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6af42f7e-20f7-4b52-9759-0bf1af4d85b4-var-lib-openvswitch\") pod \"ovnkube-node-lxg8k\" (UID: \"6af42f7e-20f7-4b52-9759-0bf1af4d85b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxg8k"
Oct 30 00:31:45.623432 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.623314    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5ed36f74-5c9e-408e-a7fb-2364144f054e-registry-certificates\") pod \"image-registry-6c69f88f85-88bkg\" (UID: \"5ed36f74-5c9e-408e-a7fb-2364144f054e\") " pod="openshift-image-registry/image-registry-6c69f88f85-88bkg"
Oct 30 00:31:45.623432 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.623347    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/94a6f5e3-d553-487e-a6f1-8c2b0fc5c695-data-volume\") pod \"insights-runtime-extractor-tbjdq\" (UID: \"94a6f5e3-d553-487e-a6f1-8c2b0fc5c695\") " pod="openshift-insights/insights-runtime-extractor-tbjdq"
Oct 30 00:31:45.623432 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.623363    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/559377b2-e3fe-4889-995e-55127771d05e-cert\") pod \"ingress-canary-wqf5x\" (UID: \"559377b2-e3fe-4889-995e-55127771d05e\") " pod="openshift-ingress-canary/ingress-canary-wqf5x"
Oct 30 00:31:45.623432 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.623418    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4606f172-d949-471e-842c-b340b50e179b-host\") pod \"tuned-5hvvb\" (UID: \"4606f172-d949-471e-842c-b340b50e179b\") " pod="openshift-cluster-node-tuning-operator/tuned-5hvvb"
Oct 30 00:31:45.623957 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.623445    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/94a6f5e3-d553-487e-a6f1-8c2b0fc5c695-data-volume\") pod \"insights-runtime-extractor-tbjdq\" (UID: \"94a6f5e3-d553-487e-a6f1-8c2b0fc5c695\") " pod="openshift-insights/insights-runtime-extractor-tbjdq"
Oct 30 00:31:45.623957 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.623479    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5ed36f74-5c9e-408e-a7fb-2364144f054e-registry-certificates\") pod \"image-registry-6c69f88f85-88bkg\" (UID: \"5ed36f74-5c9e-408e-a7fb-2364144f054e\") " pod="openshift-image-registry/image-registry-6c69f88f85-88bkg"
Oct 30 00:31:45.623957 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.623485    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/9aba3099-fe4c-4846-9a34-25db2c2fe593-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-kjsj6\" (UID: \"9aba3099-fe4c-4846-9a34-25db2c2fe593\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-kjsj6"
Oct 30 00:31:45.623957 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.623645    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/80603987-09a1-4ca1-b8b4-3c9ac8d0efc6-original-pull-secret\") pod \"global-pull-secret-syncer-tz6g8\" (UID: \"80603987-09a1-4ca1-b8b4-3c9ac8d0efc6\") " pod="kube-system/global-pull-secret-syncer-tz6g8"
Oct 30 00:31:45.623957 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.623672    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/9aba3099-fe4c-4846-9a34-25db2c2fe593-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-kjsj6\" (UID: \"9aba3099-fe4c-4846-9a34-25db2c2fe593\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-kjsj6"
Oct 30 00:31:45.623957 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.623704    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5ed36f74-5c9e-408e-a7fb-2364144f054e-installation-pull-secrets\") pod \"image-registry-6c69f88f85-88bkg\" (UID: \"5ed36f74-5c9e-408e-a7fb-2364144f054e\") " pod="openshift-image-registry/image-registry-6c69f88f85-88bkg"
Oct 30 00:31:45.623957 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.623733    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c5c3a55b-e259-4550-9b7e-f666a28d217e-cnibin\") pod \"multus-additional-cni-plugins-zvcpp\" (UID: \"c5c3a55b-e259-4550-9b7e-f666a28d217e\") " pod="openshift-multus/multus-additional-cni-plugins-zvcpp"
Oct 30 00:31:45.623957 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.623760    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c5c3a55b-e259-4550-9b7e-f666a28d217e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zvcpp\" (UID: \"c5c3a55b-e259-4550-9b7e-f666a28d217e\") " pod="openshift-multus/multus-additional-cni-plugins-zvcpp"
Oct 30 00:31:45.623957 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.623786    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e4ca0ea9-b57d-4d4c-be3a-2da5fc26a65d-tmp\") pod \"insights-operator-7c4986d449-sklth\" (UID: \"e4ca0ea9-b57d-4d4c-be3a-2da5fc26a65d\") " pod="openshift-insights/insights-operator-7c4986d449-sklth"
Oct 30 00:31:45.623957 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.623812    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a3d234b3-76d4-47d5-98a4-dafd83477b17-service-ca-bundle\") pod \"router-default-85f7b79cf7-h2tpp\" (UID: \"a3d234b3-76d4-47d5-98a4-dafd83477b17\") " pod="openshift-ingress/router-default-85f7b79cf7-h2tpp"
Oct 30 00:31:45.623957 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.623841    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/a3d234b3-76d4-47d5-98a4-dafd83477b17-stats-auth\") pod \"router-default-85f7b79cf7-h2tpp\" (UID: \"a3d234b3-76d4-47d5-98a4-dafd83477b17\") " pod="openshift-ingress/router-default-85f7b79cf7-h2tpp"
Oct 30 00:31:45.623957 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.623862    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/80603987-09a1-4ca1-b8b4-3c9ac8d0efc6-original-pull-secret\") pod \"global-pull-secret-syncer-tz6g8\" (UID: \"80603987-09a1-4ca1-b8b4-3c9ac8d0efc6\") " pod="kube-system/global-pull-secret-syncer-tz6g8"
Oct 30 00:31:45.623957 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.623871    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgb9w\" (UniqueName: \"kubernetes.io/projected/61903cf2-2b92-4f40-ab52-25509de587ef-kube-api-access-qgb9w\") pod \"service-ca-operator-544d749c8c-bjzgs\" (UID: \"61903cf2-2b92-4f40-ab52-25509de587ef\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-bjzgs"
Oct 30 00:31:45.623957 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.623880    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5ed36f74-5c9e-408e-a7fb-2364144f054e-installation-pull-secrets\") pod \"image-registry-6c69f88f85-88bkg\" (UID: \"5ed36f74-5c9e-408e-a7fb-2364144f054e\") " pod="openshift-image-registry/image-registry-6c69f88f85-88bkg"
Oct 30 00:31:45.623957 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.623904    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e4ca0ea9-b57d-4d4c-be3a-2da5fc26a65d-tmp\") pod \"insights-operator-7c4986d449-sklth\" (UID: \"e4ca0ea9-b57d-4d4c-be3a-2da5fc26a65d\") " pod="openshift-insights/insights-operator-7c4986d449-sklth"
Oct 30 00:31:45.623957 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.623905    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6af42f7e-20f7-4b52-9759-0bf1af4d85b4-run-openvswitch\") pod \"ovnkube-node-lxg8k\" (UID: \"6af42f7e-20f7-4b52-9759-0bf1af4d85b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxg8k"
Oct 30 00:31:45.623957 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.623938    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed-ca-trust-extracted\") pod \"image-registry-6d4f74cbd-njpvm\" (UID: \"2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed\") " pod="openshift-image-registry/image-registry-6d4f74cbd-njpvm"
Oct 30 00:31:45.623957 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.623958    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c5c3a55b-e259-4550-9b7e-f666a28d217e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zvcpp\" (UID: \"c5c3a55b-e259-4550-9b7e-f666a28d217e\") " pod="openshift-multus/multus-additional-cni-plugins-zvcpp"
Oct 30 00:31:45.624594 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.623967    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3d234b3-76d4-47d5-98a4-dafd83477b17-metrics-certs\") pod \"router-default-85f7b79cf7-h2tpp\" (UID: \"a3d234b3-76d4-47d5-98a4-dafd83477b17\") " pod="openshift-ingress/router-default-85f7b79cf7-h2tpp"
Oct 30 00:31:45.624594 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.624007    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed-ca-trust-extracted\") pod \"image-registry-6d4f74cbd-njpvm\" (UID: \"2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed\") " pod="openshift-image-registry/image-registry-6d4f74cbd-njpvm"
Oct 30 00:31:45.624594 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.624015    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a707a5d7-313f-4bef-8baf-a99aeae40d54-system-cni-dir\") pod \"multus-h4hbh\" (UID: \"a707a5d7-313f-4bef-8baf-a99aeae40d54\") " pod="openshift-multus/multus-h4hbh"
Oct 30 00:31:45.624594 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.624050    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a707a5d7-313f-4bef-8baf-a99aeae40d54-multus-socket-dir-parent\") pod \"multus-h4hbh\" (UID: \"a707a5d7-313f-4bef-8baf-a99aeae40d54\") " pod="openshift-multus/multus-h4hbh"
Oct 30 00:31:45.624594 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.624072    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/a3d234b3-76d4-47d5-98a4-dafd83477b17-stats-auth\") pod \"router-default-85f7b79cf7-h2tpp\" (UID: \"a3d234b3-76d4-47d5-98a4-dafd83477b17\") " pod="openshift-ingress/router-default-85f7b79cf7-h2tpp"
Oct 30 00:31:45.624594 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.624083    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5ed36f74-5c9e-408e-a7fb-2364144f054e-image-registry-private-configuration\") pod \"image-registry-6c69f88f85-88bkg\" (UID: \"5ed36f74-5c9e-408e-a7fb-2364144f054e\") " pod="openshift-image-registry/image-registry-6c69f88f85-88bkg"
Oct 30 00:31:45.624594 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.624205    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5ed36f74-5c9e-408e-a7fb-2364144f054e-trusted-ca\") pod \"image-registry-6c69f88f85-88bkg\" (UID: \"5ed36f74-5c9e-408e-a7fb-2364144f054e\") " pod="openshift-image-registry/image-registry-6c69f88f85-88bkg"
Oct 30 00:31:45.624594 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.624233    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjms2\" (UniqueName: \"kubernetes.io/projected/94a6f5e3-d553-487e-a6f1-8c2b0fc5c695-kube-api-access-wjms2\") pod \"insights-runtime-extractor-tbjdq\" (UID: \"94a6f5e3-d553-487e-a6f1-8c2b0fc5c695\") " pod="openshift-insights/insights-runtime-extractor-tbjdq"
Oct 30 00:31:45.624594 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.624240    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5ed36f74-5c9e-408e-a7fb-2364144f054e-image-registry-private-configuration\") pod \"image-registry-6c69f88f85-88bkg\" (UID: \"5ed36f74-5c9e-408e-a7fb-2364144f054e\") " pod="openshift-image-registry/image-registry-6c69f88f85-88bkg"
Oct 30 00:31:45.624594 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.624251    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/88de9f3a-e7d0-400f-9091-46c0b89c1a3c-registration-dir\") pod \"aws-ebs-csi-driver-node-v24xq\" (UID: \"88de9f3a-e7d0-400f-9091-46c0b89c1a3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v24xq"
Oct 30 00:31:45.624594 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.624295    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/88de9f3a-e7d0-400f-9091-46c0b89c1a3c-sys-fs\") pod \"aws-ebs-csi-driver-node-v24xq\" (UID: \"88de9f3a-e7d0-400f-9091-46c0b89c1a3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v24xq"
Oct 30 00:31:45.624594 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.624373    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkng8\" (UniqueName: \"kubernetes.io/projected/928e22b3-c3fe-453b-bbc8-a67accd2fcb4-kube-api-access-vkng8\") pod \"migrator-66df5d496b-khqcq\" (UID: \"928e22b3-c3fe-453b-bbc8-a67accd2fcb4\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-khqcq"
Oct 30 00:31:45.624594 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.624505    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5dc56e7e-f3c1-43eb-a21b-8be6bb4706cf-config\") pod \"console-operator-7cd56b8dbc-g7c2w\" (UID: \"5dc56e7e-f3c1-43eb-a21b-8be6bb4706cf\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-g7c2w"
Oct 30 00:31:45.624594 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.624553    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed-registry-certificates\") pod \"image-registry-6d4f74cbd-njpvm\" (UID: \"2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed\") " pod="openshift-image-registry/image-registry-6d4f74cbd-njpvm"
Oct 30 00:31:45.624594 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.624562    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5ed36f74-5c9e-408e-a7fb-2364144f054e-trusted-ca\") pod \"image-registry-6c69f88f85-88bkg\" (UID: \"5ed36f74-5c9e-408e-a7fb-2364144f054e\") " pod="openshift-image-registry/image-registry-6c69f88f85-88bkg"
Oct 30 00:31:45.624594 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.624588    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e4ca0ea9-b57d-4d4c-be3a-2da5fc26a65d-serving-cert\") pod \"insights-operator-7c4986d449-sklth\" (UID: \"e4ca0ea9-b57d-4d4c-be3a-2da5fc26a65d\") " pod="openshift-insights/insights-operator-7c4986d449-sklth"
Oct 30 00:31:45.625164 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.624665    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5dc56e7e-f3c1-43eb-a21b-8be6bb4706cf-config\") pod \"console-operator-7cd56b8dbc-g7c2w\" (UID: \"5dc56e7e-f3c1-43eb-a21b-8be6bb4706cf\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-g7c2w"
Oct 30 00:31:45.625164 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.624678    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed-registry-certificates\") pod \"image-registry-6d4f74cbd-njpvm\" (UID: \"2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed\") " pod="openshift-image-registry/image-registry-6d4f74cbd-njpvm"
Oct 30 00:31:45.625164 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.624697    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a707a5d7-313f-4bef-8baf-a99aeae40d54-host-var-lib-cni-multus\") pod \"multus-h4hbh\" (UID: \"a707a5d7-313f-4bef-8baf-a99aeae40d54\") " pod="openshift-multus/multus-h4hbh"
Oct 30 00:31:45.625164 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.624717    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6af42f7e-20f7-4b52-9759-0bf1af4d85b4-node-log\") pod \"ovnkube-node-lxg8k\" (UID: \"6af42f7e-20f7-4b52-9759-0bf1af4d85b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxg8k"
Oct 30 00:31:45.625164 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.624733    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af2d6f84-aea3-4698-be12-11bd6767caec-metrics-certs\") pod \"network-metrics-daemon-z58bb\" (UID: \"af2d6f84-aea3-4698-be12-11bd6767caec\") " pod="openshift-multus/network-metrics-daemon-z58bb"
Oct 30 00:31:45.625164 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.624749    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/a3d234b3-76d4-47d5-98a4-dafd83477b17-default-certificate\") pod \"router-default-85f7b79cf7-h2tpp\" (UID: \"a3d234b3-76d4-47d5-98a4-dafd83477b17\") " pod="openshift-ingress/router-default-85f7b79cf7-h2tpp"
Oct 30 00:31:45.625164 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.624760    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e4ca0ea9-b57d-4d4c-be3a-2da5fc26a65d-serving-cert\") pod \"insights-operator-7c4986d449-sklth\" (UID: \"e4ca0ea9-b57d-4d4c-be3a-2da5fc26a65d\") " pod="openshift-insights/insights-operator-7c4986d449-sklth"
Oct 30 00:31:45.625164 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.624766    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a707a5d7-313f-4bef-8baf-a99aeae40d54-host-run-netns\") pod \"multus-h4hbh\" (UID: \"a707a5d7-313f-4bef-8baf-a99aeae40d54\") " pod="openshift-multus/multus-h4hbh"
Oct 30 00:31:45.625164 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.624876    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/14285dfa-dd38-490e-a486-a1fcfc3d0214-serviceca\") pod \"node-ca-8b6d6\" (UID: \"14285dfa-dd38-490e-a486-a1fcfc3d0214\") " pod="openshift-image-registry/node-ca-8b6d6"
Oct 30 00:31:45.625164 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.624916    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/a3d234b3-76d4-47d5-98a4-dafd83477b17-default-certificate\") pod \"router-default-85f7b79cf7-h2tpp\" (UID: \"a3d234b3-76d4-47d5-98a4-dafd83477b17\") " pod="openshift-ingress/router-default-85f7b79cf7-h2tpp"
Oct 30 00:31:45.625164 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.624945    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6af42f7e-20f7-4b52-9759-0bf1af4d85b4-ovnkube-config\") pod \"ovnkube-node-lxg8k\" (UID: \"6af42f7e-20f7-4b52-9759-0bf1af4d85b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxg8k"
Oct 30 00:31:45.625164 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.624971    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6af42f7e-20f7-4b52-9759-0bf1af4d85b4-env-overrides\") pod \"ovnkube-node-lxg8k\" (UID: \"6af42f7e-20f7-4b52-9759-0bf1af4d85b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxg8k"
Oct 30 00:31:45.625164 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.624987    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bba787ce-4e61-4a5a-a728-af73c68dfdd3-trusted-ca-bundle\") pod \"periodic-gathering-zzlf6-kvglg\" (UID: \"bba787ce-4e61-4a5a-a728-af73c68dfdd3\") " pod="openshift-insights/periodic-gathering-zzlf6-kvglg"
Oct 30 00:31:45.625164 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.625042    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4606f172-d949-471e-842c-b340b50e179b-etc-sysctl-conf\") pod \"tuned-5hvvb\" (UID: \"4606f172-d949-471e-842c-b340b50e179b\") " pod="openshift-cluster-node-tuning-operator/tuned-5hvvb"
Oct 30 00:31:45.625164 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.625079    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed-bound-sa-token\") pod \"image-registry-6d4f74cbd-njpvm\" (UID: \"2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed\") " pod="openshift-image-registry/image-registry-6d4f74cbd-njpvm"
Oct 30 00:31:45.625164 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.625116    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xds6d\" (UniqueName: \"kubernetes.io/projected/139583a1-f622-49ab-8c1e-af168fcdf6c4-kube-api-access-xds6d\") pod \"kas-connection-checker-685895c7c9-f2b4d\" (UID: \"139583a1-f622-49ab-8c1e-af168fcdf6c4\") " pod="kube-system/kas-connection-checker-685895c7c9-f2b4d"
Oct 30 00:31:45.625164 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.625123    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/14285dfa-dd38-490e-a486-a1fcfc3d0214-serviceca\") pod \"node-ca-8b6d6\" (UID: \"14285dfa-dd38-490e-a486-a1fcfc3d0214\") " pod="openshift-image-registry/node-ca-8b6d6"
Oct 30 00:31:45.625164 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.625143    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6af42f7e-20f7-4b52-9759-0bf1af4d85b4-env-overrides\") pod \"ovnkube-node-lxg8k\" (UID: \"6af42f7e-20f7-4b52-9759-0bf1af4d85b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxg8k"
Oct 30 00:31:45.625164 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.625149    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/94a6f5e3-d553-487e-a6f1-8c2b0fc5c695-crio-socket\") pod \"insights-runtime-extractor-tbjdq\" (UID: \"94a6f5e3-d553-487e-a6f1-8c2b0fc5c695\") " pod="openshift-insights/insights-runtime-extractor-tbjdq"
Oct 30 00:31:45.625164 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.625162    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6af42f7e-20f7-4b52-9759-0bf1af4d85b4-ovnkube-config\") pod \"ovnkube-node-lxg8k\" (UID: \"6af42f7e-20f7-4b52-9759-0bf1af4d85b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxg8k"
Oct 30 00:31:45.625992 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.625183    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvgs7\" (UniqueName: \"kubernetes.io/projected/783c7a98-5ebf-44dd-8da9-797e71c28169-kube-api-access-vvgs7\") pod \"kas-connection-checker-685895c7c9-fbdvl\" (UID: \"783c7a98-5ebf-44dd-8da9-797e71c28169\") " pod="kube-system/kas-connection-checker-685895c7c9-fbdvl"
Oct 30 00:31:45.625992 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.625217    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c5c3a55b-e259-4550-9b7e-f666a28d217e-system-cni-dir\") pod \"multus-additional-cni-plugins-zvcpp\" (UID: \"c5c3a55b-e259-4550-9b7e-f666a28d217e\") " pod="openshift-multus/multus-additional-cni-plugins-zvcpp"
Oct 30 00:31:45.625992 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.625248    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e4ca0ea9-b57d-4d4c-be3a-2da5fc26a65d-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-sklth\" (UID: \"e4ca0ea9-b57d-4d4c-be3a-2da5fc26a65d\") " pod="openshift-insights/insights-operator-7c4986d449-sklth"
Oct 30 00:31:45.625992 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.625288    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6af42f7e-20f7-4b52-9759-0bf1af4d85b4-log-socket\") pod \"ovnkube-node-lxg8k\" (UID: \"6af42f7e-20f7-4b52-9759-0bf1af4d85b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxg8k"
Oct 30 00:31:45.625992 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.625309    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bba787ce-4e61-4a5a-a728-af73c68dfdd3-trusted-ca-bundle\") pod \"periodic-gathering-zzlf6-kvglg\" (UID: \"bba787ce-4e61-4a5a-a728-af73c68dfdd3\") " pod="openshift-insights/periodic-gathering-zzlf6-kvglg"
Oct 30 00:31:45.625992 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.625327    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6af42f7e-20f7-4b52-9759-0bf1af4d85b4-host-run-ovn-kubernetes\") pod \"ovnkube-node-lxg8k\" (UID: \"6af42f7e-20f7-4b52-9759-0bf1af4d85b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxg8k"
Oct 30 00:31:45.625992 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.625396    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a707a5d7-313f-4bef-8baf-a99aeae40d54-multus-cni-dir\") pod \"multus-h4hbh\" (UID: \"a707a5d7-313f-4bef-8baf-a99aeae40d54\") " pod="openshift-multus/multus-h4hbh"
Oct 30 00:31:45.625992 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.625437    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6af42f7e-20f7-4b52-9759-0bf1af4d85b4-host-kubelet\") pod \"ovnkube-node-lxg8k\" (UID: \"6af42f7e-20f7-4b52-9759-0bf1af4d85b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxg8k"
Oct 30 00:31:45.625992 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.625467    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4606f172-d949-471e-842c-b340b50e179b-var-lib-kubelet\") pod \"tuned-5hvvb\" (UID: \"4606f172-d949-471e-842c-b340b50e179b\") " pod="openshift-cluster-node-tuning-operator/tuned-5hvvb"
Oct 30 00:31:45.625992 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.625492    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/88de9f3a-e7d0-400f-9091-46c0b89c1a3c-socket-dir\") pod \"aws-ebs-csi-driver-node-v24xq\" (UID: \"88de9f3a-e7d0-400f-9091-46c0b89c1a3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v24xq"
Oct 30 00:31:45.625992 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.625518    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d7c5e976-596e-4cb5-81a9-b16262b3c85b-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-zqdbk\" (UID: \"d7c5e976-596e-4cb5-81a9-b16262b3c85b\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-zqdbk"
Oct 30 00:31:45.625992 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.625573    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6af42f7e-20f7-4b52-9759-0bf1af4d85b4-host-cni-netd\") pod \"ovnkube-node-lxg8k\" (UID: \"6af42f7e-20f7-4b52-9759-0bf1af4d85b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxg8k"
Oct 30 00:31:45.625992 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.625629    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/80603987-09a1-4ca1-b8b4-3c9ac8d0efc6-dbus\") pod \"global-pull-secret-syncer-tz6g8\" (UID: \"80603987-09a1-4ca1-b8b4-3c9ac8d0efc6\") " pod="kube-system/global-pull-secret-syncer-tz6g8"
Oct 30 00:31:45.625992 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.625665    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dj9wz\" (UniqueName: \"kubernetes.io/projected/c4d423f3-2077-47fb-bf4d-d9991dd5eac1-kube-api-access-dj9wz\") pod \"migrator-66df5d496b-g2dlc\" (UID: \"c4d423f3-2077-47fb-bf4d-d9991dd5eac1\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-g2dlc"
Oct 30 00:31:45.625992 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.625698    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7l585\" (UniqueName: \"kubernetes.io/projected/5ed36f74-5c9e-408e-a7fb-2364144f054e-kube-api-access-7l585\") pod \"image-registry-6c69f88f85-88bkg\" (UID: \"5ed36f74-5c9e-408e-a7fb-2364144f054e\") " pod="openshift-image-registry/image-registry-6c69f88f85-88bkg"
Oct 30 00:31:45.625992 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.625821    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjrlm\" (UniqueName: \"kubernetes.io/projected/5dc56e7e-f3c1-43eb-a21b-8be6bb4706cf-kube-api-access-pjrlm\") pod \"console-operator-7cd56b8dbc-g7c2w\" (UID: \"5dc56e7e-f3c1-43eb-a21b-8be6bb4706cf\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-g7c2w"
Oct 30 00:31:45.625992 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.625905    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c5c3a55b-e259-4550-9b7e-f666a28d217e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zvcpp\" (UID: \"c5c3a55b-e259-4550-9b7e-f666a28d217e\") " pod="openshift-multus/multus-additional-cni-plugins-zvcpp"
Oct 30 00:31:45.625992 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.625969    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqmpd\" (UniqueName: \"kubernetes.io/projected/c5c3a55b-e259-4550-9b7e-f666a28d217e-kube-api-access-mqmpd\") pod \"multus-additional-cni-plugins-zvcpp\" (UID: \"c5c3a55b-e259-4550-9b7e-f666a28d217e\") " pod="openshift-multus/multus-additional-cni-plugins-zvcpp"
Oct 30 00:31:45.626669 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.626009    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e4ca0ea9-b57d-4d4c-be3a-2da5fc26a65d-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-sklth\" (UID: \"e4ca0ea9-b57d-4d4c-be3a-2da5fc26a65d\") " pod="openshift-insights/insights-operator-7c4986d449-sklth"
Oct 30 00:31:45.626669 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.626039    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzg5q\" (UniqueName: \"kubernetes.io/projected/e4ca0ea9-b57d-4d4c-be3a-2da5fc26a65d-kube-api-access-qzg5q\") pod \"insights-operator-7c4986d449-sklth\" (UID: \"e4ca0ea9-b57d-4d4c-be3a-2da5fc26a65d\") " pod="openshift-insights/insights-operator-7c4986d449-sklth"
Oct 30 00:31:45.626669 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.626078    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/078fd0e8-1393-45d0-baee-bbcd88095b2b-tmp-dir\") pod \"node-resolver-6gjtz\" (UID: \"078fd0e8-1393-45d0-baee-bbcd88095b2b\") " pod="openshift-dns/node-resolver-6gjtz"
Oct 30 00:31:45.626669 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.626095    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b874991e-9562-4848-b8f1-4b5a2b7cddc4-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-sxknf\" (UID: \"b874991e-9562-4848-b8f1-4b5a2b7cddc4\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-sxknf"
Oct 30 00:31:45.626669 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.626113    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6af42f7e-20f7-4b52-9759-0bf1af4d85b4-run-systemd\") pod \"ovnkube-node-lxg8k\" (UID: \"6af42f7e-20f7-4b52-9759-0bf1af4d85b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxg8k"
Oct 30 00:31:45.626669 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.626118    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c5c3a55b-e259-4550-9b7e-f666a28d217e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zvcpp\" (UID: \"c5c3a55b-e259-4550-9b7e-f666a28d217e\") " pod="openshift-multus/multus-additional-cni-plugins-zvcpp"
Oct 30 00:31:45.626669 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.626134    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/94a6f5e3-d553-487e-a6f1-8c2b0fc5c695-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tbjdq\" (UID: \"94a6f5e3-d553-487e-a6f1-8c2b0fc5c695\") " pod="openshift-insights/insights-runtime-extractor-tbjdq"
Oct 30 00:31:45.626669 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.626165    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/94a6f5e3-d553-487e-a6f1-8c2b0fc5c695-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tbjdq\" (UID: \"94a6f5e3-d553-487e-a6f1-8c2b0fc5c695\") " pod="openshift-insights/insights-runtime-extractor-tbjdq"
Oct 30 00:31:45.626669 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.626171    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/078fd0e8-1393-45d0-baee-bbcd88095b2b-tmp-dir\") pod \"node-resolver-6gjtz\" (UID: \"078fd0e8-1393-45d0-baee-bbcd88095b2b\") " pod="openshift-dns/node-resolver-6gjtz"
Oct 30 00:31:45.626669 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.626200    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84rlf\" (UniqueName: \"kubernetes.io/projected/2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed-kube-api-access-84rlf\") pod \"image-registry-6d4f74cbd-njpvm\" (UID: \"2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed\") " pod="openshift-image-registry/image-registry-6d4f74cbd-njpvm"
Oct 30 00:31:45.626669 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.626225    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/251cf359-f9f5-44ae-8ba9-42bf0cabb1ab-tmp-dir\") pod \"dns-default-btj6v\" (UID: \"251cf359-f9f5-44ae-8ba9-42bf0cabb1ab\") " pod="openshift-dns/dns-default-btj6v"
Oct 30 00:31:45.626669 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.626240    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a707a5d7-313f-4bef-8baf-a99aeae40d54-host-run-multus-certs\") pod \"multus-h4hbh\" (UID: \"a707a5d7-313f-4bef-8baf-a99aeae40d54\") " pod="openshift-multus/multus-h4hbh"
Oct 30 00:31:45.626669 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.626255    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4606f172-d949-471e-842c-b340b50e179b-tmp\") pod \"tuned-5hvvb\" (UID: \"4606f172-d949-471e-842c-b340b50e179b\") " pod="openshift-cluster-node-tuning-operator/tuned-5hvvb"
Oct 30 00:31:45.626669 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.626258    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b874991e-9562-4848-b8f1-4b5a2b7cddc4-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-sxknf\" (UID: \"b874991e-9562-4848-b8f1-4b5a2b7cddc4\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-sxknf"
Oct 30 00:31:45.626669 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.626275    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed-installation-pull-secrets\") pod \"image-registry-6d4f74cbd-njpvm\" (UID: \"2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed\") " pod="openshift-image-registry/image-registry-6d4f74cbd-njpvm"
Oct 30 00:31:45.626669 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.626328    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jjgr\" (UniqueName: \"kubernetes.io/projected/9aba3099-fe4c-4846-9a34-25db2c2fe593-kube-api-access-2jjgr\") pod \"cluster-monitoring-operator-884576c57-kjsj6\" (UID: \"9aba3099-fe4c-4846-9a34-25db2c2fe593\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-kjsj6"
Oct 30 00:31:45.626669 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.626343    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/251cf359-f9f5-44ae-8ba9-42bf0cabb1ab-tmp-dir\") pod \"dns-default-btj6v\" (UID: \"251cf359-f9f5-44ae-8ba9-42bf0cabb1ab\") " pod="openshift-dns/dns-default-btj6v"
Oct 30 00:31:45.626669 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.626354    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/078fd0e8-1393-45d0-baee-bbcd88095b2b-hosts-file\") pod \"node-resolver-6gjtz\" (UID: \"078fd0e8-1393-45d0-baee-bbcd88095b2b\") " pod="openshift-dns/node-resolver-6gjtz"
Oct 30 00:31:45.626669 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.626395    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvtmg\" (UniqueName: \"kubernetes.io/projected/b874991e-9562-4848-b8f1-4b5a2b7cddc4-kube-api-access-zvtmg\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-sxknf\" (UID: \"b874991e-9562-4848-b8f1-4b5a2b7cddc4\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-sxknf"
Oct 30 00:31:45.626669 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.626423    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4606f172-d949-471e-842c-b340b50e179b-tmp\") pod \"tuned-5hvvb\" (UID: \"4606f172-d949-471e-842c-b340b50e179b\") " pod="openshift-cluster-node-tuning-operator/tuned-5hvvb"
Oct 30 00:31:45.626669 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.626444    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxcbj\" (UniqueName: \"kubernetes.io/projected/88de9f3a-e7d0-400f-9091-46c0b89c1a3c-kube-api-access-nxcbj\") pod \"aws-ebs-csi-driver-node-v24xq\" (UID: \"88de9f3a-e7d0-400f-9091-46c0b89c1a3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v24xq"
Oct 30 00:31:45.626669 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.626497    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/80603987-09a1-4ca1-b8b4-3c9ac8d0efc6-kubelet-config\") pod \"global-pull-secret-syncer-tz6g8\" (UID: \"80603987-09a1-4ca1-b8b4-3c9ac8d0efc6\") " pod="kube-system/global-pull-secret-syncer-tz6g8"
Oct 30 00:31:45.626669 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.626521    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed-installation-pull-secrets\") pod \"image-registry-6d4f74cbd-njpvm\" (UID: \"2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed\") " pod="openshift-image-registry/image-registry-6d4f74cbd-njpvm"
Oct 30 00:31:45.626669 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.626529    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ngcm\" (UniqueName: \"kubernetes.io/projected/cea1b649-9c31-4434-9448-7f6652496830-kube-api-access-2ngcm\") pod \"volume-data-source-validator-58c9f5f46d-246g2\" (UID: \"cea1b649-9c31-4434-9448-7f6652496830\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-246g2"
Oct 30 00:31:45.626669 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.626563    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5dc56e7e-f3c1-43eb-a21b-8be6bb4706cf-serving-cert\") pod \"console-operator-7cd56b8dbc-g7c2w\" (UID: \"5dc56e7e-f3c1-43eb-a21b-8be6bb4706cf\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-g7c2w"
Oct 30 00:31:45.627586 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.626683    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4606f172-d949-471e-842c-b340b50e179b-etc-systemd\") pod \"tuned-5hvvb\" (UID: \"4606f172-d949-471e-842c-b340b50e179b\") " pod="openshift-cluster-node-tuning-operator/tuned-5hvvb"
Oct 30 00:31:45.627586 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.626712    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0d1b347f-c22e-489b-986d-d3b3e01fa36b-agent-certs\") pod \"konnectivity-agent-m5qgr\" (UID: \"0d1b347f-c22e-489b-986d-d3b3e01fa36b\") " pod="kube-system/konnectivity-agent-m5qgr"
Oct 30 00:31:45.627586 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.626745    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e4ca0ea9-b57d-4d4c-be3a-2da5fc26a65d-service-ca-bundle\") pod \"insights-operator-7c4986d449-sklth\" (UID: \"e4ca0ea9-b57d-4d4c-be3a-2da5fc26a65d\") " pod="openshift-insights/insights-operator-7c4986d449-sklth"
Oct 30 00:31:45.627586 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.626851    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/61903cf2-2b92-4f40-ab52-25509de587ef-serving-cert\") pod \"service-ca-operator-544d749c8c-bjzgs\" (UID: \"61903cf2-2b92-4f40-ab52-25509de587ef\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-bjzgs"
Oct 30 00:31:45.627586 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.626873    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e4ca0ea9-b57d-4d4c-be3a-2da5fc26a65d-service-ca-bundle\") pod \"insights-operator-7c4986d449-sklth\" (UID: \"e4ca0ea9-b57d-4d4c-be3a-2da5fc26a65d\") " pod="openshift-insights/insights-operator-7c4986d449-sklth"
Oct 30 00:31:45.627586 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.626885    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a707a5d7-313f-4bef-8baf-a99aeae40d54-cnibin\") pod \"multus-h4hbh\" (UID: \"a707a5d7-313f-4bef-8baf-a99aeae40d54\") " pod="openshift-multus/multus-h4hbh"
Oct 30 00:31:45.627586 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.626913    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/1881dcfe-c2da-4253-bca5-93f3526e14fa-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-2nnk8\" (UID: \"1881dcfe-c2da-4253-bca5-93f3526e14fa\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-2nnk8"
Oct 30 00:31:45.627586 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.626930    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/88de9f3a-e7d0-400f-9091-46c0b89c1a3c-device-dir\") pod \"aws-ebs-csi-driver-node-v24xq\" (UID: \"88de9f3a-e7d0-400f-9091-46c0b89c1a3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v24xq"
Oct 30 00:31:45.627586 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.626945    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a707a5d7-313f-4bef-8baf-a99aeae40d54-cni-binary-copy\") pod \"multus-h4hbh\" (UID: \"a707a5d7-313f-4bef-8baf-a99aeae40d54\") " pod="openshift-multus/multus-h4hbh"
Oct 30 00:31:45.627586 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.626946    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0d1b347f-c22e-489b-986d-d3b3e01fa36b-agent-certs\") pod \"konnectivity-agent-m5qgr\" (UID: \"0d1b347f-c22e-489b-986d-d3b3e01fa36b\") " pod="kube-system/konnectivity-agent-m5qgr"
Oct 30 00:31:45.627586 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.626973    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1881dcfe-c2da-4253-bca5-93f3526e14fa-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-2nnk8\" (UID: \"1881dcfe-c2da-4253-bca5-93f3526e14fa\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-2nnk8"
Oct 30 00:31:45.627586 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.627004    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a707a5d7-313f-4bef-8baf-a99aeae40d54-multus-daemon-config\") pod \"multus-h4hbh\" (UID: \"a707a5d7-313f-4bef-8baf-a99aeae40d54\") " pod="openshift-multus/multus-h4hbh"
Oct 30 00:31:45.627586 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.627020    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6af42f7e-20f7-4b52-9759-0bf1af4d85b4-systemd-units\") pod \"ovnkube-node-lxg8k\" (UID: \"6af42f7e-20f7-4b52-9759-0bf1af4d85b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxg8k"
Oct 30 00:31:45.627586 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.627020    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/61903cf2-2b92-4f40-ab52-25509de587ef-serving-cert\") pod \"service-ca-operator-544d749c8c-bjzgs\" (UID: \"61903cf2-2b92-4f40-ab52-25509de587ef\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-bjzgs"
Oct 30 00:31:45.627586 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.627027    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5dc56e7e-f3c1-43eb-a21b-8be6bb4706cf-serving-cert\") pod \"console-operator-7cd56b8dbc-g7c2w\" (UID: \"5dc56e7e-f3c1-43eb-a21b-8be6bb4706cf\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-g7c2w"
Oct 30 00:31:45.627586 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.627069    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6af42f7e-20f7-4b52-9759-0bf1af4d85b4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lxg8k\" (UID: \"6af42f7e-20f7-4b52-9759-0bf1af4d85b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxg8k"
Oct 30 00:31:45.627586 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.627117    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed-trusted-ca\") pod \"image-registry-6d4f74cbd-njpvm\" (UID: \"2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed\") " pod="openshift-image-registry/image-registry-6d4f74cbd-njpvm"
Oct 30 00:31:45.627586 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.627158    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/1881dcfe-c2da-4253-bca5-93f3526e14fa-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-2nnk8\" (UID: \"1881dcfe-c2da-4253-bca5-93f3526e14fa\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-2nnk8"
Oct 30 00:31:45.627586 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.627180    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a707a5d7-313f-4bef-8baf-a99aeae40d54-cni-binary-copy\") pod \"multus-h4hbh\" (UID: \"a707a5d7-313f-4bef-8baf-a99aeae40d54\") " pod="openshift-multus/multus-h4hbh"
Oct 30 00:31:45.627586 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.627191    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a707a5d7-313f-4bef-8baf-a99aeae40d54-multus-daemon-config\") pod \"multus-h4hbh\" (UID: \"a707a5d7-313f-4bef-8baf-a99aeae40d54\") " pod="openshift-multus/multus-h4hbh"
Oct 30 00:31:45.627586 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.627184    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b874991e-9562-4848-b8f1-4b5a2b7cddc4-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-sxknf\" (UID: \"b874991e-9562-4848-b8f1-4b5a2b7cddc4\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-sxknf"
Oct 30 00:31:45.627586 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.627262    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6af42f7e-20f7-4b52-9759-0bf1af4d85b4-etc-openvswitch\") pod \"ovnkube-node-lxg8k\" (UID: \"6af42f7e-20f7-4b52-9759-0bf1af4d85b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxg8k"
Oct 30 00:31:45.627586 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.627286    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/bba787ce-4e61-4a5a-a728-af73c68dfdd3-archives-path\") pod \"periodic-gathering-zzlf6-kvglg\" (UID: \"bba787ce-4e61-4a5a-a728-af73c68dfdd3\") " pod="openshift-insights/periodic-gathering-zzlf6-kvglg"
Oct 30 00:31:45.627586 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.627302    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/88de9f3a-e7d0-400f-9091-46c0b89c1a3c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-v24xq\" (UID: \"88de9f3a-e7d0-400f-9091-46c0b89c1a3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v24xq"
Oct 30 00:31:45.627586 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.627330    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4d7j7\" (UniqueName: \"kubernetes.io/projected/af2d6f84-aea3-4698-be12-11bd6767caec-kube-api-access-4d7j7\") pod \"network-metrics-daemon-z58bb\" (UID: \"af2d6f84-aea3-4698-be12-11bd6767caec\") " pod="openshift-multus/network-metrics-daemon-z58bb"
Oct 30 00:31:45.627586 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.627354    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/bba787ce-4e61-4a5a-a728-af73c68dfdd3-archives-path\") pod \"periodic-gathering-zzlf6-kvglg\" (UID: \"bba787ce-4e61-4a5a-a728-af73c68dfdd3\") " pod="openshift-insights/periodic-gathering-zzlf6-kvglg"
Oct 30 00:31:45.627586 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.627429    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed-trusted-ca\") pod \"image-registry-6d4f74cbd-njpvm\" (UID: \"2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed\") " pod="openshift-image-registry/image-registry-6d4f74cbd-njpvm"
Oct 30 00:31:45.627586 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.627446    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b874991e-9562-4848-b8f1-4b5a2b7cddc4-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-sxknf\" (UID: \"b874991e-9562-4848-b8f1-4b5a2b7cddc4\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-sxknf"
Oct 30 00:31:45.645832 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.645799    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lbm4\" (UniqueName: \"kubernetes.io/projected/6af42f7e-20f7-4b52-9759-0bf1af4d85b4-kube-api-access-5lbm4\") pod \"ovnkube-node-lxg8k\" (UID: \"6af42f7e-20f7-4b52-9759-0bf1af4d85b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxg8k"
Oct 30 00:31:45.645929 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.645801    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rl4f\" (UniqueName: \"kubernetes.io/projected/4606f172-d949-471e-842c-b340b50e179b-kube-api-access-6rl4f\") pod \"tuned-5hvvb\" (UID: \"4606f172-d949-471e-842c-b340b50e179b\") " pod="openshift-cluster-node-tuning-operator/tuned-5hvvb"
Oct 30 00:31:45.645929 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.645877    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzc86\" (UniqueName: \"kubernetes.io/projected/bba787ce-4e61-4a5a-a728-af73c68dfdd3-kube-api-access-fzc86\") pod \"periodic-gathering-zzlf6-kvglg\" (UID: \"bba787ce-4e61-4a5a-a728-af73c68dfdd3\") " pod="openshift-insights/periodic-gathering-zzlf6-kvglg"
Oct 30 00:31:45.646046 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.646021    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbj4z\" (UniqueName: \"kubernetes.io/projected/9822c484-fd06-4e5e-94b4-ef0f5ade9a13-kube-api-access-xbj4z\") pod \"kas-connection-checker-685895c7c9-cqjlz\" (UID: \"9822c484-fd06-4e5e-94b4-ef0f5ade9a13\") " pod="kube-system/kas-connection-checker-685895c7c9-cqjlz"
Oct 30 00:31:45.646134 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.646117    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmzz7\" (UniqueName: \"kubernetes.io/projected/559377b2-e3fe-4889-995e-55127771d05e-kube-api-access-zmzz7\") pod \"ingress-canary-wqf5x\" (UID: \"559377b2-e3fe-4889-995e-55127771d05e\") " pod="openshift-ingress-canary/ingress-canary-wqf5x"
Oct 30 00:31:45.646640 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.646621    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5ed36f74-5c9e-408e-a7fb-2364144f054e-bound-sa-token\") pod \"image-registry-6c69f88f85-88bkg\" (UID: \"5ed36f74-5c9e-408e-a7fb-2364144f054e\") " pod="openshift-image-registry/image-registry-6c69f88f85-88bkg"
Oct 30 00:31:45.646821 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.646805    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6llvc\" (UniqueName: \"kubernetes.io/projected/14285dfa-dd38-490e-a486-a1fcfc3d0214-kube-api-access-6llvc\") pod \"node-ca-8b6d6\" (UID: \"14285dfa-dd38-490e-a486-a1fcfc3d0214\") " pod="openshift-image-registry/node-ca-8b6d6"
Oct 30 00:31:45.646937 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.646921    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r998s\" (UniqueName: \"kubernetes.io/projected/d27ee5b1-d1aa-418c-9169-12cac37dcc2c-kube-api-access-r998s\") pod \"network-check-target-lngrr\" (UID: \"d27ee5b1-d1aa-418c-9169-12cac37dcc2c\") " pod="openshift-network-diagnostics/network-check-target-lngrr"
Oct 30 00:31:45.647059 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.647031    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhs2g\" (UniqueName: \"kubernetes.io/projected/9b2b3b14-8027-489a-88f0-d056226c2f67-kube-api-access-bhs2g\") pod \"network-check-source-6f78d59f99-jg5h6\" (UID: \"9b2b3b14-8027-489a-88f0-d056226c2f67\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-jg5h6"
Oct 30 00:31:45.650887 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.650872    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jzc7\" (UniqueName: \"kubernetes.io/projected/c1e4fab8-6072-4fcd-9624-a60dee7aed86-kube-api-access-2jzc7\") pod \"iptables-alerter-kph7f\" (UID: \"c1e4fab8-6072-4fcd-9624-a60dee7aed86\") " pod="openshift-network-operator/iptables-alerter-kph7f"
Oct 30 00:31:45.652689 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.652674    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjmbq\" (UniqueName: \"kubernetes.io/projected/a3d234b3-76d4-47d5-98a4-dafd83477b17-kube-api-access-bjmbq\") pod \"router-default-85f7b79cf7-h2tpp\" (UID: \"a3d234b3-76d4-47d5-98a4-dafd83477b17\") " pod="openshift-ingress/router-default-85f7b79cf7-h2tpp"
Oct 30 00:31:45.655766 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.655748    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4d7j7\" (UniqueName: \"kubernetes.io/projected/af2d6f84-aea3-4698-be12-11bd6767caec-kube-api-access-4d7j7\") pod \"network-metrics-daemon-z58bb\" (UID: \"af2d6f84-aea3-4698-be12-11bd6767caec\") " pod="openshift-multus/network-metrics-daemon-z58bb"
Oct 30 00:31:45.656002 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.655982    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-crppr\" (UniqueName: \"kubernetes.io/projected/078fd0e8-1393-45d0-baee-bbcd88095b2b-kube-api-access-crppr\") pod \"node-resolver-6gjtz\" (UID: \"078fd0e8-1393-45d0-baee-bbcd88095b2b\") " pod="openshift-dns/node-resolver-6gjtz"
Oct 30 00:31:45.656227 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.656211    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7l585\" (UniqueName: \"kubernetes.io/projected/5ed36f74-5c9e-408e-a7fb-2364144f054e-kube-api-access-7l585\") pod \"image-registry-6c69f88f85-88bkg\" (UID: \"5ed36f74-5c9e-408e-a7fb-2364144f054e\") " pod="openshift-image-registry/image-registry-6c69f88f85-88bkg"
Oct 30 00:31:45.658032 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.658017    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-566pb\" (UniqueName: \"kubernetes.io/projected/1881dcfe-c2da-4253-bca5-93f3526e14fa-kube-api-access-566pb\") pod \"cluster-samples-operator-6ddb6c8b94-2nnk8\" (UID: \"1881dcfe-c2da-4253-bca5-93f3526e14fa\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-2nnk8"
Oct 30 00:31:45.661536 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.661519    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ngcm\" (UniqueName: \"kubernetes.io/projected/cea1b649-9c31-4434-9448-7f6652496830-kube-api-access-2ngcm\") pod \"volume-data-source-validator-58c9f5f46d-246g2\" (UID: \"cea1b649-9c31-4434-9448-7f6652496830\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-246g2"
Oct 30 00:31:45.661597 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.661569    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4cc6\" (UniqueName: \"kubernetes.io/projected/a707a5d7-313f-4bef-8baf-a99aeae40d54-kube-api-access-h4cc6\") pod \"multus-h4hbh\" (UID: \"a707a5d7-313f-4bef-8baf-a99aeae40d54\") " pod="openshift-multus/multus-h4hbh"
Oct 30 00:31:45.661986 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.661521    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzg5q\" (UniqueName: \"kubernetes.io/projected/e4ca0ea9-b57d-4d4c-be3a-2da5fc26a65d-kube-api-access-qzg5q\") pod \"insights-operator-7c4986d449-sklth\" (UID: \"e4ca0ea9-b57d-4d4c-be3a-2da5fc26a65d\") " pod="openshift-insights/insights-operator-7c4986d449-sklth"
Oct 30 00:31:45.662053 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.662036    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jjgr\" (UniqueName: \"kubernetes.io/projected/9aba3099-fe4c-4846-9a34-25db2c2fe593-kube-api-access-2jjgr\") pod \"cluster-monitoring-operator-884576c57-kjsj6\" (UID: \"9aba3099-fe4c-4846-9a34-25db2c2fe593\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-kjsj6"
Oct 30 00:31:45.663062 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.663047    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvtmg\" (UniqueName: \"kubernetes.io/projected/b874991e-9562-4848-b8f1-4b5a2b7cddc4-kube-api-access-zvtmg\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-sxknf\" (UID: \"b874991e-9562-4848-b8f1-4b5a2b7cddc4\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-sxknf"
Oct 30 00:31:45.666391 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.666360    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dj9wz\" (UniqueName: \"kubernetes.io/projected/c4d423f3-2077-47fb-bf4d-d9991dd5eac1-kube-api-access-dj9wz\") pod \"migrator-66df5d496b-g2dlc\" (UID: \"c4d423f3-2077-47fb-bf4d-d9991dd5eac1\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-g2dlc"
Oct 30 00:31:45.666661 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.666641    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xds6d\" (UniqueName: \"kubernetes.io/projected/139583a1-f622-49ab-8c1e-af168fcdf6c4-kube-api-access-xds6d\") pod \"kas-connection-checker-685895c7c9-f2b4d\" (UID: \"139583a1-f622-49ab-8c1e-af168fcdf6c4\") " pod="kube-system/kas-connection-checker-685895c7c9-f2b4d"
Oct 30 00:31:45.666824 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.666809    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kpm2\" (UniqueName: \"kubernetes.io/projected/251cf359-f9f5-44ae-8ba9-42bf0cabb1ab-kube-api-access-6kpm2\") pod \"dns-default-btj6v\" (UID: \"251cf359-f9f5-44ae-8ba9-42bf0cabb1ab\") " pod="openshift-dns/dns-default-btj6v"
Oct 30 00:31:45.667008 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.666993    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjms2\" (UniqueName: \"kubernetes.io/projected/94a6f5e3-d553-487e-a6f1-8c2b0fc5c695-kube-api-access-wjms2\") pod \"insights-runtime-extractor-tbjdq\" (UID: \"94a6f5e3-d553-487e-a6f1-8c2b0fc5c695\") " pod="openshift-insights/insights-runtime-extractor-tbjdq"
Oct 30 00:31:45.667091 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.667075    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvgs7\" (UniqueName: \"kubernetes.io/projected/783c7a98-5ebf-44dd-8da9-797e71c28169-kube-api-access-vvgs7\") pod \"kas-connection-checker-685895c7c9-fbdvl\" (UID: \"783c7a98-5ebf-44dd-8da9-797e71c28169\") " pod="kube-system/kas-connection-checker-685895c7c9-fbdvl"
Oct 30 00:31:45.671592 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.671570    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed-bound-sa-token\") pod \"image-registry-6d4f74cbd-njpvm\" (UID: \"2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed\") " pod="openshift-image-registry/image-registry-6d4f74cbd-njpvm"
Oct 30 00:31:45.671879 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.671858    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjrlm\" (UniqueName: \"kubernetes.io/projected/5dc56e7e-f3c1-43eb-a21b-8be6bb4706cf-kube-api-access-pjrlm\") pod \"console-operator-7cd56b8dbc-g7c2w\" (UID: \"5dc56e7e-f3c1-43eb-a21b-8be6bb4706cf\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-g7c2w"
Oct 30 00:31:45.672228 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.672204    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgb9w\" (UniqueName: \"kubernetes.io/projected/61903cf2-2b92-4f40-ab52-25509de587ef-kube-api-access-qgb9w\") pod \"service-ca-operator-544d749c8c-bjzgs\" (UID: \"61903cf2-2b92-4f40-ab52-25509de587ef\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-bjzgs"
Oct 30 00:31:45.672732 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.672710    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxcbj\" (UniqueName: \"kubernetes.io/projected/88de9f3a-e7d0-400f-9091-46c0b89c1a3c-kube-api-access-nxcbj\") pod \"aws-ebs-csi-driver-node-v24xq\" (UID: \"88de9f3a-e7d0-400f-9091-46c0b89c1a3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v24xq"
Oct 30 00:31:45.673013 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.672996    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkng8\" (UniqueName: \"kubernetes.io/projected/928e22b3-c3fe-453b-bbc8-a67accd2fcb4-kube-api-access-vkng8\") pod \"migrator-66df5d496b-khqcq\" (UID: \"928e22b3-c3fe-453b-bbc8-a67accd2fcb4\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-khqcq"
Oct 30 00:31:45.673315 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.673289    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqmpd\" (UniqueName: \"kubernetes.io/projected/c5c3a55b-e259-4550-9b7e-f666a28d217e-kube-api-access-mqmpd\") pod \"multus-additional-cni-plugins-zvcpp\" (UID: \"c5c3a55b-e259-4550-9b7e-f666a28d217e\") " pod="openshift-multus/multus-additional-cni-plugins-zvcpp"
Oct 30 00:31:45.673611 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.673590    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-84rlf\" (UniqueName: \"kubernetes.io/projected/2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed-kube-api-access-84rlf\") pod \"image-registry-6d4f74cbd-njpvm\" (UID: \"2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed\") " pod="openshift-image-registry/image-registry-6d4f74cbd-njpvm"
Oct 30 00:31:45.673886 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.673864    5002 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7d4bc45dd6-sxknf_b874991e-9562-4848-b8f1-4b5a2b7cddc4/kube-storage-version-migrator-operator/0.log"
Oct 30 00:31:45.728529 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.728462    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6af42f7e-20f7-4b52-9759-0bf1af4d85b4-run-ovn\") pod \"ovnkube-node-lxg8k\" (UID: \"6af42f7e-20f7-4b52-9759-0bf1af4d85b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxg8k"
Oct 30 00:31:45.728529 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.728496    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9aba3099-fe4c-4846-9a34-25db2c2fe593-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-kjsj6\" (UID: \"9aba3099-fe4c-4846-9a34-25db2c2fe593\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-kjsj6"
Oct 30 00:31:45.728529 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.728512    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a707a5d7-313f-4bef-8baf-a99aeae40d54-host-var-lib-cni-bin\") pod \"multus-h4hbh\" (UID: \"a707a5d7-313f-4bef-8baf-a99aeae40d54\") " pod="openshift-multus/multus-h4hbh"
Oct 30 00:31:45.728529 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.728527    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a707a5d7-313f-4bef-8baf-a99aeae40d54-host-run-k8s-cni-cncf-io\") pod \"multus-h4hbh\" (UID: \"a707a5d7-313f-4bef-8baf-a99aeae40d54\") " pod="openshift-multus/multus-h4hbh"
Oct 30 00:31:45.728795 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.728541    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a707a5d7-313f-4bef-8baf-a99aeae40d54-multus-conf-dir\") pod \"multus-h4hbh\" (UID: \"a707a5d7-313f-4bef-8baf-a99aeae40d54\") " pod="openshift-multus/multus-h4hbh"
Oct 30 00:31:45.728795 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.728557    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a707a5d7-313f-4bef-8baf-a99aeae40d54-etc-kubernetes\") pod \"multus-h4hbh\" (UID: \"a707a5d7-313f-4bef-8baf-a99aeae40d54\") " pod="openshift-multus/multus-h4hbh"
Oct 30 00:31:45.728795 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.728567    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6af42f7e-20f7-4b52-9759-0bf1af4d85b4-run-ovn\") pod \"ovnkube-node-lxg8k\" (UID: \"6af42f7e-20f7-4b52-9759-0bf1af4d85b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxg8k"
Oct 30 00:31:45.728795 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.728571    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5ed36f74-5c9e-408e-a7fb-2364144f054e-registry-tls\") pod \"image-registry-6c69f88f85-88bkg\" (UID: \"5ed36f74-5c9e-408e-a7fb-2364144f054e\") " pod="openshift-image-registry/image-registry-6c69f88f85-88bkg"
Oct 30 00:31:45.728795 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.728605    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a707a5d7-313f-4bef-8baf-a99aeae40d54-host-var-lib-cni-bin\") pod \"multus-h4hbh\" (UID: \"a707a5d7-313f-4bef-8baf-a99aeae40d54\") " pod="openshift-multus/multus-h4hbh"
Oct 30 00:31:45.728795 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.728628    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a707a5d7-313f-4bef-8baf-a99aeae40d54-host-run-k8s-cni-cncf-io\") pod \"multus-h4hbh\" (UID: \"a707a5d7-313f-4bef-8baf-a99aeae40d54\") " pod="openshift-multus/multus-h4hbh"
Oct 30 00:31:45.728795 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.728615    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a707a5d7-313f-4bef-8baf-a99aeae40d54-hostroot\") pod \"multus-h4hbh\" (UID: \"a707a5d7-313f-4bef-8baf-a99aeae40d54\") " pod="openshift-multus/multus-h4hbh"
Oct 30 00:31:45.728795 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.728638    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a707a5d7-313f-4bef-8baf-a99aeae40d54-hostroot\") pod \"multus-h4hbh\" (UID: \"a707a5d7-313f-4bef-8baf-a99aeae40d54\") " pod="openshift-multus/multus-h4hbh"
Oct 30 00:31:45.728795 ip-10-0-128-7 kubenswrapper[5002]: E1030 00:31:45.728644    5002 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 00:31:45.728795 ip-10-0-128-7 kubenswrapper[5002]: E1030 00:31:45.728691    5002 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 00:31:45.728795 ip-10-0-128-7 kubenswrapper[5002]: E1030 00:31:45.728702    5002 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6c69f88f85-88bkg: secret "image-registry-tls" not found
Oct 30 00:31:45.728795 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.728703    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c5c3a55b-e259-4550-9b7e-f666a28d217e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zvcpp\" (UID: \"c5c3a55b-e259-4550-9b7e-f666a28d217e\") " pod="openshift-multus/multus-additional-cni-plugins-zvcpp"
Oct 30 00:31:45.728795 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.728664    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c5c3a55b-e259-4550-9b7e-f666a28d217e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zvcpp\" (UID: \"c5c3a55b-e259-4550-9b7e-f666a28d217e\") " pod="openshift-multus/multus-additional-cni-plugins-zvcpp"
Oct 30 00:31:45.728795 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.728609    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a707a5d7-313f-4bef-8baf-a99aeae40d54-multus-conf-dir\") pod \"multus-h4hbh\" (UID: \"a707a5d7-313f-4bef-8baf-a99aeae40d54\") " pod="openshift-multus/multus-h4hbh"
Oct 30 00:31:45.728795 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.728735    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a707a5d7-313f-4bef-8baf-a99aeae40d54-etc-kubernetes\") pod \"multus-h4hbh\" (UID: \"a707a5d7-313f-4bef-8baf-a99aeae40d54\") " pod="openshift-multus/multus-h4hbh"
Oct 30 00:31:45.728795 ip-10-0-128-7 kubenswrapper[5002]: E1030 00:31:45.728746    5002 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9aba3099-fe4c-4846-9a34-25db2c2fe593-cluster-monitoring-operator-tls podName:9aba3099-fe4c-4846-9a34-25db2c2fe593 nodeName:}" failed. No retries permitted until 2026-10-30 00:31:46.228726105 +0000 UTC m=+1.845159002 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/9aba3099-fe4c-4846-9a34-25db2c2fe593-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-kjsj6" (UID: "9aba3099-fe4c-4846-9a34-25db2c2fe593") : secret "cluster-monitoring-operator-tls" not found
Oct 30 00:31:45.728795 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.728766    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6af42f7e-20f7-4b52-9759-0bf1af4d85b4-host-slash\") pod \"ovnkube-node-lxg8k\" (UID: \"6af42f7e-20f7-4b52-9759-0bf1af4d85b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxg8k"
Oct 30 00:31:45.728795 ip-10-0-128-7 kubenswrapper[5002]: E1030 00:31:45.728795    5002 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5ed36f74-5c9e-408e-a7fb-2364144f054e-registry-tls podName:5ed36f74-5c9e-408e-a7fb-2364144f054e nodeName:}" failed. No retries permitted until 2026-10-30 00:31:46.228786758 +0000 UTC m=+1.845219618 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5ed36f74-5c9e-408e-a7fb-2364144f054e-registry-tls") pod "image-registry-6c69f88f85-88bkg" (UID: "5ed36f74-5c9e-408e-a7fb-2364144f054e") : secret "image-registry-tls" not found
Oct 30 00:31:45.729612 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.728818    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6af42f7e-20f7-4b52-9759-0bf1af4d85b4-host-slash\") pod \"ovnkube-node-lxg8k\" (UID: \"6af42f7e-20f7-4b52-9759-0bf1af4d85b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxg8k"
Oct 30 00:31:45.729612 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.728860    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4606f172-d949-471e-842c-b340b50e179b-etc-sysctl-d\") pod \"tuned-5hvvb\" (UID: \"4606f172-d949-471e-842c-b340b50e179b\") " pod="openshift-cluster-node-tuning-operator/tuned-5hvvb"
Oct 30 00:31:45.729612 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.728889    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/251cf359-f9f5-44ae-8ba9-42bf0cabb1ab-metrics-tls\") pod \"dns-default-btj6v\" (UID: \"251cf359-f9f5-44ae-8ba9-42bf0cabb1ab\") " pod="openshift-dns/dns-default-btj6v"
Oct 30 00:31:45.729612 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.728915    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4606f172-d949-471e-842c-b340b50e179b-etc-modprobe-d\") pod \"tuned-5hvvb\" (UID: \"4606f172-d949-471e-842c-b340b50e179b\") " pod="openshift-cluster-node-tuning-operator/tuned-5hvvb"
Oct 30 00:31:45.729612 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.728938    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4606f172-d949-471e-842c-b340b50e179b-sys\") pod \"tuned-5hvvb\" (UID: \"4606f172-d949-471e-842c-b340b50e179b\") " pod="openshift-cluster-node-tuning-operator/tuned-5hvvb"
Oct 30 00:31:45.729612 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.728962    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed-registry-tls\") pod \"image-registry-6d4f74cbd-njpvm\" (UID: \"2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed\") " pod="openshift-image-registry/image-registry-6d4f74cbd-njpvm"
Oct 30 00:31:45.729612 ip-10-0-128-7 kubenswrapper[5002]: E1030 00:31:45.728974    5002 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 00:31:45.729612 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.728986    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4606f172-d949-471e-842c-b340b50e179b-etc-sysctl-d\") pod \"tuned-5hvvb\" (UID: \"4606f172-d949-471e-842c-b340b50e179b\") " pod="openshift-cluster-node-tuning-operator/tuned-5hvvb"
Oct 30 00:31:45.729612 ip-10-0-128-7 kubenswrapper[5002]: E1030 00:31:45.729023    5002 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/251cf359-f9f5-44ae-8ba9-42bf0cabb1ab-metrics-tls podName:251cf359-f9f5-44ae-8ba9-42bf0cabb1ab nodeName:}" failed. No retries permitted until 2026-10-30 00:31:46.229012211 +0000 UTC m=+1.845445069 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/251cf359-f9f5-44ae-8ba9-42bf0cabb1ab-metrics-tls") pod "dns-default-btj6v" (UID: "251cf359-f9f5-44ae-8ba9-42bf0cabb1ab") : secret "dns-default-metrics-tls" not found
Oct 30 00:31:45.729612 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.729018    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4606f172-d949-471e-842c-b340b50e179b-etc-modprobe-d\") pod \"tuned-5hvvb\" (UID: \"4606f172-d949-471e-842c-b340b50e179b\") " pod="openshift-cluster-node-tuning-operator/tuned-5hvvb"
Oct 30 00:31:45.729612 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.728990    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c1e4fab8-6072-4fcd-9624-a60dee7aed86-host-slash\") pod \"iptables-alerter-kph7f\" (UID: \"c1e4fab8-6072-4fcd-9624-a60dee7aed86\") " pod="openshift-network-operator/iptables-alerter-kph7f"
Oct 30 00:31:45.729612 ip-10-0-128-7 kubenswrapper[5002]: E1030 00:31:45.729068    5002 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 00:31:45.729612 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.729059    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4606f172-d949-471e-842c-b340b50e179b-lib-modules\") pod \"tuned-5hvvb\" (UID: \"4606f172-d949-471e-842c-b340b50e179b\") " pod="openshift-cluster-node-tuning-operator/tuned-5hvvb"
Oct 30 00:31:45.729612 ip-10-0-128-7 kubenswrapper[5002]: E1030 00:31:45.729081    5002 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6d4f74cbd-njpvm: secret "image-registry-tls" not found
Oct 30 00:31:45.729612 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.729094    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/88de9f3a-e7d0-400f-9091-46c0b89c1a3c-etc-selinux\") pod \"aws-ebs-csi-driver-node-v24xq\" (UID: \"88de9f3a-e7d0-400f-9091-46c0b89c1a3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v24xq"
Oct 30 00:31:45.729612 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.729025    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c1e4fab8-6072-4fcd-9624-a60dee7aed86-host-slash\") pod \"iptables-alerter-kph7f\" (UID: \"c1e4fab8-6072-4fcd-9624-a60dee7aed86\") " pod="openshift-network-operator/iptables-alerter-kph7f"
Oct 30 00:31:45.729612 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.729038    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4606f172-d949-471e-842c-b340b50e179b-sys\") pod \"tuned-5hvvb\" (UID: \"4606f172-d949-471e-842c-b340b50e179b\") " pod="openshift-cluster-node-tuning-operator/tuned-5hvvb"
Oct 30 00:31:45.729612 ip-10-0-128-7 kubenswrapper[5002]: E1030 00:31:45.729121    5002 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed-registry-tls podName:2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed nodeName:}" failed. No retries permitted until 2026-10-30 00:31:46.229105364 +0000 UTC m=+1.845538241 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed-registry-tls") pod "image-registry-6d4f74cbd-njpvm" (UID: "2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed") : secret "image-registry-tls" not found
Oct 30 00:31:45.729612 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.729139    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6af42f7e-20f7-4b52-9759-0bf1af4d85b4-host-cni-bin\") pod \"ovnkube-node-lxg8k\" (UID: \"6af42f7e-20f7-4b52-9759-0bf1af4d85b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxg8k"
Oct 30 00:31:45.729612 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.729142    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/88de9f3a-e7d0-400f-9091-46c0b89c1a3c-etc-selinux\") pod \"aws-ebs-csi-driver-node-v24xq\" (UID: \"88de9f3a-e7d0-400f-9091-46c0b89c1a3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v24xq"
Oct 30 00:31:45.729612 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.729162    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4606f172-d949-471e-842c-b340b50e179b-lib-modules\") pod \"tuned-5hvvb\" (UID: \"4606f172-d949-471e-842c-b340b50e179b\") " pod="openshift-cluster-node-tuning-operator/tuned-5hvvb"
Oct 30 00:31:45.729612 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.729180    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6af42f7e-20f7-4b52-9759-0bf1af4d85b4-host-cni-bin\") pod \"ovnkube-node-lxg8k\" (UID: \"6af42f7e-20f7-4b52-9759-0bf1af4d85b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxg8k"
Oct 30 00:31:45.729612 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.729181    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/14285dfa-dd38-490e-a486-a1fcfc3d0214-host\") pod \"node-ca-8b6d6\" (UID: \"14285dfa-dd38-490e-a486-a1fcfc3d0214\") " pod="openshift-image-registry/node-ca-8b6d6"
Oct 30 00:31:45.729612 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.729207    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6af42f7e-20f7-4b52-9759-0bf1af4d85b4-var-lib-openvswitch\") pod \"ovnkube-node-lxg8k\" (UID: \"6af42f7e-20f7-4b52-9759-0bf1af4d85b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxg8k"
Oct 30 00:31:45.729612 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.729225    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4606f172-d949-471e-842c-b340b50e179b-host\") pod \"tuned-5hvvb\" (UID: \"4606f172-d949-471e-842c-b340b50e179b\") " pod="openshift-cluster-node-tuning-operator/tuned-5hvvb"
Oct 30 00:31:45.729612 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.729246    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/14285dfa-dd38-490e-a486-a1fcfc3d0214-host\") pod \"node-ca-8b6d6\" (UID: \"14285dfa-dd38-490e-a486-a1fcfc3d0214\") " pod="openshift-image-registry/node-ca-8b6d6"
Oct 30 00:31:45.729612 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.729248    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c5c3a55b-e259-4550-9b7e-f666a28d217e-cnibin\") pod \"multus-additional-cni-plugins-zvcpp\" (UID: \"c5c3a55b-e259-4550-9b7e-f666a28d217e\") " pod="openshift-multus/multus-additional-cni-plugins-zvcpp"
Oct 30 00:31:45.729612 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.729266    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6af42f7e-20f7-4b52-9759-0bf1af4d85b4-var-lib-openvswitch\") pod \"ovnkube-node-lxg8k\" (UID: \"6af42f7e-20f7-4b52-9759-0bf1af4d85b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxg8k"
Oct 30 00:31:45.729612 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.729283    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a3d234b3-76d4-47d5-98a4-dafd83477b17-service-ca-bundle\") pod \"router-default-85f7b79cf7-h2tpp\" (UID: \"a3d234b3-76d4-47d5-98a4-dafd83477b17\") " pod="openshift-ingress/router-default-85f7b79cf7-h2tpp"
Oct 30 00:31:45.729612 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.729286    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c5c3a55b-e259-4550-9b7e-f666a28d217e-cnibin\") pod \"multus-additional-cni-plugins-zvcpp\" (UID: \"c5c3a55b-e259-4550-9b7e-f666a28d217e\") " pod="openshift-multus/multus-additional-cni-plugins-zvcpp"
Oct 30 00:31:45.729612 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.729283    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4606f172-d949-471e-842c-b340b50e179b-host\") pod \"tuned-5hvvb\" (UID: \"4606f172-d949-471e-842c-b340b50e179b\") " pod="openshift-cluster-node-tuning-operator/tuned-5hvvb"
Oct 30 00:31:45.729612 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.729302    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6af42f7e-20f7-4b52-9759-0bf1af4d85b4-run-openvswitch\") pod \"ovnkube-node-lxg8k\" (UID: \"6af42f7e-20f7-4b52-9759-0bf1af4d85b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxg8k"
Oct 30 00:31:45.729612 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.729329    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6af42f7e-20f7-4b52-9759-0bf1af4d85b4-run-openvswitch\") pod \"ovnkube-node-lxg8k\" (UID: \"6af42f7e-20f7-4b52-9759-0bf1af4d85b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxg8k"
Oct 30 00:31:45.729612 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.729333    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3d234b3-76d4-47d5-98a4-dafd83477b17-metrics-certs\") pod \"router-default-85f7b79cf7-h2tpp\" (UID: \"a3d234b3-76d4-47d5-98a4-dafd83477b17\") " pod="openshift-ingress/router-default-85f7b79cf7-h2tpp"
Oct 30 00:31:45.729612 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.729359    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a707a5d7-313f-4bef-8baf-a99aeae40d54-system-cni-dir\") pod \"multus-h4hbh\" (UID: \"a707a5d7-313f-4bef-8baf-a99aeae40d54\") " pod="openshift-multus/multus-h4hbh"
Oct 30 00:31:45.729612 ip-10-0-128-7 kubenswrapper[5002]: E1030 00:31:45.729373    5002 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a3d234b3-76d4-47d5-98a4-dafd83477b17-service-ca-bundle podName:a3d234b3-76d4-47d5-98a4-dafd83477b17 nodeName:}" failed. No retries permitted until 2026-10-30 00:31:46.229359094 +0000 UTC m=+1.845791973 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/a3d234b3-76d4-47d5-98a4-dafd83477b17-service-ca-bundle") pod "router-default-85f7b79cf7-h2tpp" (UID: "a3d234b3-76d4-47d5-98a4-dafd83477b17") : configmap references non-existent config key: service-ca.crt
Oct 30 00:31:45.729612 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.729424    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a707a5d7-313f-4bef-8baf-a99aeae40d54-system-cni-dir\") pod \"multus-h4hbh\" (UID: \"a707a5d7-313f-4bef-8baf-a99aeae40d54\") " pod="openshift-multus/multus-h4hbh"
Oct 30 00:31:45.729612 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.729426    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a707a5d7-313f-4bef-8baf-a99aeae40d54-multus-socket-dir-parent\") pod \"multus-h4hbh\" (UID: \"a707a5d7-313f-4bef-8baf-a99aeae40d54\") " pod="openshift-multus/multus-h4hbh"
Oct 30 00:31:45.729612 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.729457    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/88de9f3a-e7d0-400f-9091-46c0b89c1a3c-registration-dir\") pod \"aws-ebs-csi-driver-node-v24xq\" (UID: \"88de9f3a-e7d0-400f-9091-46c0b89c1a3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v24xq"
Oct 30 00:31:45.729612 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.729467    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a707a5d7-313f-4bef-8baf-a99aeae40d54-multus-socket-dir-parent\") pod \"multus-h4hbh\" (UID: \"a707a5d7-313f-4bef-8baf-a99aeae40d54\") " pod="openshift-multus/multus-h4hbh"
Oct 30 00:31:45.729612 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.729481    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/88de9f3a-e7d0-400f-9091-46c0b89c1a3c-sys-fs\") pod \"aws-ebs-csi-driver-node-v24xq\" (UID: \"88de9f3a-e7d0-400f-9091-46c0b89c1a3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v24xq"
Oct 30 00:31:45.729612 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.729513    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a707a5d7-313f-4bef-8baf-a99aeae40d54-host-var-lib-cni-multus\") pod \"multus-h4hbh\" (UID: \"a707a5d7-313f-4bef-8baf-a99aeae40d54\") " pod="openshift-multus/multus-h4hbh"
Oct 30 00:31:45.729612 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.729536    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6af42f7e-20f7-4b52-9759-0bf1af4d85b4-node-log\") pod \"ovnkube-node-lxg8k\" (UID: \"6af42f7e-20f7-4b52-9759-0bf1af4d85b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxg8k"
Oct 30 00:31:45.729612 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.729539    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/88de9f3a-e7d0-400f-9091-46c0b89c1a3c-registration-dir\") pod \"aws-ebs-csi-driver-node-v24xq\" (UID: \"88de9f3a-e7d0-400f-9091-46c0b89c1a3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v24xq"
Oct 30 00:31:45.729612 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.729559    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af2d6f84-aea3-4698-be12-11bd6767caec-metrics-certs\") pod \"network-metrics-daemon-z58bb\" (UID: \"af2d6f84-aea3-4698-be12-11bd6767caec\") " pod="openshift-multus/network-metrics-daemon-z58bb"
Oct 30 00:31:45.729612 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.729574    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a707a5d7-313f-4bef-8baf-a99aeae40d54-host-var-lib-cni-multus\") pod \"multus-h4hbh\" (UID: \"a707a5d7-313f-4bef-8baf-a99aeae40d54\") " pod="openshift-multus/multus-h4hbh"
Oct 30 00:31:45.729612 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.729581    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a707a5d7-313f-4bef-8baf-a99aeae40d54-host-run-netns\") pod \"multus-h4hbh\" (UID: \"a707a5d7-313f-4bef-8baf-a99aeae40d54\") " pod="openshift-multus/multus-h4hbh"
Oct 30 00:31:45.729612 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.729595    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/88de9f3a-e7d0-400f-9091-46c0b89c1a3c-sys-fs\") pod \"aws-ebs-csi-driver-node-v24xq\" (UID: \"88de9f3a-e7d0-400f-9091-46c0b89c1a3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v24xq"
Oct 30 00:31:45.729612 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.729603    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4606f172-d949-471e-842c-b340b50e179b-etc-sysctl-conf\") pod \"tuned-5hvvb\" (UID: \"4606f172-d949-471e-842c-b340b50e179b\") " pod="openshift-cluster-node-tuning-operator/tuned-5hvvb"
Oct 30 00:31:45.729612 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.729611    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6af42f7e-20f7-4b52-9759-0bf1af4d85b4-node-log\") pod \"ovnkube-node-lxg8k\" (UID: \"6af42f7e-20f7-4b52-9759-0bf1af4d85b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxg8k"
Oct 30 00:31:45.729612 ip-10-0-128-7 kubenswrapper[5002]: E1030 00:31:45.729632    5002 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 00:31:45.729612 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.729616    5002 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 00:31:45.729612 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.729647    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a707a5d7-313f-4bef-8baf-a99aeae40d54-host-run-netns\") pod \"multus-h4hbh\" (UID: \"a707a5d7-313f-4bef-8baf-a99aeae40d54\") " pod="openshift-multus/multus-h4hbh"
Oct 30 00:31:45.729612 ip-10-0-128-7 kubenswrapper[5002]: E1030 00:31:45.729662    5002 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/af2d6f84-aea3-4698-be12-11bd6767caec-metrics-certs podName:af2d6f84-aea3-4698-be12-11bd6767caec nodeName:}" failed. No retries permitted until 2026-10-30 00:31:46.229654056 +0000 UTC m=+1.846086913 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/af2d6f84-aea3-4698-be12-11bd6767caec-metrics-certs") pod "network-metrics-daemon-z58bb" (UID: "af2d6f84-aea3-4698-be12-11bd6767caec") : secret "metrics-daemon-secret" not found
Oct 30 00:31:45.731988 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.729642    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/94a6f5e3-d553-487e-a6f1-8c2b0fc5c695-crio-socket\") pod \"insights-runtime-extractor-tbjdq\" (UID: \"94a6f5e3-d553-487e-a6f1-8c2b0fc5c695\") " pod="openshift-insights/insights-runtime-extractor-tbjdq"
Oct 30 00:31:45.731988 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.729679    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/94a6f5e3-d553-487e-a6f1-8c2b0fc5c695-crio-socket\") pod \"insights-runtime-extractor-tbjdq\" (UID: \"94a6f5e3-d553-487e-a6f1-8c2b0fc5c695\") " pod="openshift-insights/insights-runtime-extractor-tbjdq"
Oct 30 00:31:45.731988 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.729685    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c5c3a55b-e259-4550-9b7e-f666a28d217e-system-cni-dir\") pod \"multus-additional-cni-plugins-zvcpp\" (UID: \"c5c3a55b-e259-4550-9b7e-f666a28d217e\") " pod="openshift-multus/multus-additional-cni-plugins-zvcpp"
Oct 30 00:31:45.731988 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.729711    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6af42f7e-20f7-4b52-9759-0bf1af4d85b4-log-socket\") pod \"ovnkube-node-lxg8k\" (UID: \"6af42f7e-20f7-4b52-9759-0bf1af4d85b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxg8k"
Oct 30 00:31:45.731988 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.729716    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4606f172-d949-471e-842c-b340b50e179b-etc-sysctl-conf\") pod \"tuned-5hvvb\" (UID: \"4606f172-d949-471e-842c-b340b50e179b\") " pod="openshift-cluster-node-tuning-operator/tuned-5hvvb"
Oct 30 00:31:45.731988 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.729716    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c5c3a55b-e259-4550-9b7e-f666a28d217e-system-cni-dir\") pod \"multus-additional-cni-plugins-zvcpp\" (UID: \"c5c3a55b-e259-4550-9b7e-f666a28d217e\") " pod="openshift-multus/multus-additional-cni-plugins-zvcpp"
Oct 30 00:31:45.731988 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.729729    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6af42f7e-20f7-4b52-9759-0bf1af4d85b4-host-run-ovn-kubernetes\") pod \"ovnkube-node-lxg8k\" (UID: \"6af42f7e-20f7-4b52-9759-0bf1af4d85b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxg8k"
Oct 30 00:31:45.731988 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.729742    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6af42f7e-20f7-4b52-9759-0bf1af4d85b4-log-socket\") pod \"ovnkube-node-lxg8k\" (UID: \"6af42f7e-20f7-4b52-9759-0bf1af4d85b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxg8k"
Oct 30 00:31:45.731988 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.729755    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6af42f7e-20f7-4b52-9759-0bf1af4d85b4-host-run-ovn-kubernetes\") pod \"ovnkube-node-lxg8k\" (UID: \"6af42f7e-20f7-4b52-9759-0bf1af4d85b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxg8k"
Oct 30 00:31:45.731988 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.729766    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a707a5d7-313f-4bef-8baf-a99aeae40d54-multus-cni-dir\") pod \"multus-h4hbh\" (UID: \"a707a5d7-313f-4bef-8baf-a99aeae40d54\") " pod="openshift-multus/multus-h4hbh"
Oct 30 00:31:45.731988 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.729784    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6af42f7e-20f7-4b52-9759-0bf1af4d85b4-host-kubelet\") pod \"ovnkube-node-lxg8k\" (UID: \"6af42f7e-20f7-4b52-9759-0bf1af4d85b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxg8k"
Oct 30 00:31:45.731988 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.729802    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4606f172-d949-471e-842c-b340b50e179b-var-lib-kubelet\") pod \"tuned-5hvvb\" (UID: \"4606f172-d949-471e-842c-b340b50e179b\") " pod="openshift-cluster-node-tuning-operator/tuned-5hvvb"
Oct 30 00:31:45.731988 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.729826    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/88de9f3a-e7d0-400f-9091-46c0b89c1a3c-socket-dir\") pod \"aws-ebs-csi-driver-node-v24xq\" (UID: \"88de9f3a-e7d0-400f-9091-46c0b89c1a3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v24xq"
Oct 30 00:31:45.731988 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.729837    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a707a5d7-313f-4bef-8baf-a99aeae40d54-multus-cni-dir\") pod \"multus-h4hbh\" (UID: \"a707a5d7-313f-4bef-8baf-a99aeae40d54\") " pod="openshift-multus/multus-h4hbh"
Oct 30 00:31:45.731988 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.729845    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6af42f7e-20f7-4b52-9759-0bf1af4d85b4-host-kubelet\") pod \"ovnkube-node-lxg8k\" (UID: \"6af42f7e-20f7-4b52-9759-0bf1af4d85b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxg8k"
Oct 30 00:31:45.731988 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.729852    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d7c5e976-596e-4cb5-81a9-b16262b3c85b-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-zqdbk\" (UID: \"d7c5e976-596e-4cb5-81a9-b16262b3c85b\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-zqdbk"
Oct 30 00:31:45.731988 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.729871    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4606f172-d949-471e-842c-b340b50e179b-var-lib-kubelet\") pod \"tuned-5hvvb\" (UID: \"4606f172-d949-471e-842c-b340b50e179b\") " pod="openshift-cluster-node-tuning-operator/tuned-5hvvb"
Oct 30 00:31:45.731988 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.729898    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6af42f7e-20f7-4b52-9759-0bf1af4d85b4-host-cni-netd\") pod \"ovnkube-node-lxg8k\" (UID: \"6af42f7e-20f7-4b52-9759-0bf1af4d85b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxg8k"
Oct 30 00:31:45.731988 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.729910    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6af42f7e-20f7-4b52-9759-0bf1af4d85b4-host-cni-netd\") pod \"ovnkube-node-lxg8k\" (UID: \"6af42f7e-20f7-4b52-9759-0bf1af4d85b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxg8k"
Oct 30 00:31:45.731988 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.729901    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/88de9f3a-e7d0-400f-9091-46c0b89c1a3c-socket-dir\") pod \"aws-ebs-csi-driver-node-v24xq\" (UID: \"88de9f3a-e7d0-400f-9091-46c0b89c1a3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v24xq"
Oct 30 00:31:45.731988 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.729922    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/80603987-09a1-4ca1-b8b4-3c9ac8d0efc6-dbus\") pod \"global-pull-secret-syncer-tz6g8\" (UID: \"80603987-09a1-4ca1-b8b4-3c9ac8d0efc6\") " pod="kube-system/global-pull-secret-syncer-tz6g8"
Oct 30 00:31:45.731988 ip-10-0-128-7 kubenswrapper[5002]: E1030 00:31:45.729939    5002 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 00:31:45.731988 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.729955    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6af42f7e-20f7-4b52-9759-0bf1af4d85b4-run-systemd\") pod \"ovnkube-node-lxg8k\" (UID: \"6af42f7e-20f7-4b52-9759-0bf1af4d85b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxg8k"
Oct 30 00:31:45.731988 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.729960    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/80603987-09a1-4ca1-b8b4-3c9ac8d0efc6-dbus\") pod \"global-pull-secret-syncer-tz6g8\" (UID: \"80603987-09a1-4ca1-b8b4-3c9ac8d0efc6\") " pod="kube-system/global-pull-secret-syncer-tz6g8"
Oct 30 00:31:45.731988 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.729982    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/94a6f5e3-d553-487e-a6f1-8c2b0fc5c695-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tbjdq\" (UID: \"94a6f5e3-d553-487e-a6f1-8c2b0fc5c695\") " pod="openshift-insights/insights-runtime-extractor-tbjdq"
Oct 30 00:31:45.731988 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.729980    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6af42f7e-20f7-4b52-9759-0bf1af4d85b4-run-systemd\") pod \"ovnkube-node-lxg8k\" (UID: \"6af42f7e-20f7-4b52-9759-0bf1af4d85b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxg8k"
Oct 30 00:31:45.731988 ip-10-0-128-7 kubenswrapper[5002]: E1030 00:31:45.730004    5002 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7c5e976-596e-4cb5-81a9-b16262b3c85b-networking-console-plugin-cert podName:d7c5e976-596e-4cb5-81a9-b16262b3c85b nodeName:}" failed. No retries permitted until 2026-10-30 00:31:46.229989489 +0000 UTC m=+1.846422382 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/d7c5e976-596e-4cb5-81a9-b16262b3c85b-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-zqdbk" (UID: "d7c5e976-596e-4cb5-81a9-b16262b3c85b") : secret "networking-console-plugin-cert" not found
Oct 30 00:31:45.731988 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.730037    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/94a6f5e3-d553-487e-a6f1-8c2b0fc5c695-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tbjdq\" (UID: \"94a6f5e3-d553-487e-a6f1-8c2b0fc5c695\") " pod="openshift-insights/insights-runtime-extractor-tbjdq"
Oct 30 00:31:45.731988 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.730066    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a707a5d7-313f-4bef-8baf-a99aeae40d54-host-run-multus-certs\") pod \"multus-h4hbh\" (UID: \"a707a5d7-313f-4bef-8baf-a99aeae40d54\") " pod="openshift-multus/multus-h4hbh"
Oct 30 00:31:45.731988 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.730091    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/078fd0e8-1393-45d0-baee-bbcd88095b2b-hosts-file\") pod \"node-resolver-6gjtz\" (UID: \"078fd0e8-1393-45d0-baee-bbcd88095b2b\") " pod="openshift-dns/node-resolver-6gjtz"
Oct 30 00:31:45.731988 ip-10-0-128-7 kubenswrapper[5002]: E1030 00:31:45.730101    5002 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 00:31:45.731988 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.730105    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a707a5d7-313f-4bef-8baf-a99aeae40d54-host-run-multus-certs\") pod \"multus-h4hbh\" (UID: \"a707a5d7-313f-4bef-8baf-a99aeae40d54\") " pod="openshift-multus/multus-h4hbh"
Oct 30 00:31:45.731988 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.730116    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/80603987-09a1-4ca1-b8b4-3c9ac8d0efc6-kubelet-config\") pod \"global-pull-secret-syncer-tz6g8\" (UID: \"80603987-09a1-4ca1-b8b4-3c9ac8d0efc6\") " pod="kube-system/global-pull-secret-syncer-tz6g8"
Oct 30 00:31:45.731988 ip-10-0-128-7 kubenswrapper[5002]: E1030 00:31:45.730132    5002 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94a6f5e3-d553-487e-a6f1-8c2b0fc5c695-insights-runtime-extractor-tls podName:94a6f5e3-d553-487e-a6f1-8c2b0fc5c695 nodeName:}" failed. No retries permitted until 2026-10-30 00:31:46.230123344 +0000 UTC m=+1.846556230 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/94a6f5e3-d553-487e-a6f1-8c2b0fc5c695-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tbjdq" (UID: "94a6f5e3-d553-487e-a6f1-8c2b0fc5c695") : secret "insights-runtime-extractor-tls" not found
Oct 30 00:31:45.731988 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.730145    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/078fd0e8-1393-45d0-baee-bbcd88095b2b-hosts-file\") pod \"node-resolver-6gjtz\" (UID: \"078fd0e8-1393-45d0-baee-bbcd88095b2b\") " pod="openshift-dns/node-resolver-6gjtz"
Oct 30 00:31:45.731988 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.730156    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4606f172-d949-471e-842c-b340b50e179b-etc-systemd\") pod \"tuned-5hvvb\" (UID: \"4606f172-d949-471e-842c-b340b50e179b\") " pod="openshift-cluster-node-tuning-operator/tuned-5hvvb"
Oct 30 00:31:45.731988 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.730182    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a707a5d7-313f-4bef-8baf-a99aeae40d54-cnibin\") pod \"multus-h4hbh\" (UID: \"a707a5d7-313f-4bef-8baf-a99aeae40d54\") " pod="openshift-multus/multus-h4hbh"
Oct 30 00:31:45.731988 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.730197    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/88de9f3a-e7d0-400f-9091-46c0b89c1a3c-device-dir\") pod \"aws-ebs-csi-driver-node-v24xq\" (UID: \"88de9f3a-e7d0-400f-9091-46c0b89c1a3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v24xq"
Oct 30 00:31:45.731988 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.730200    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4606f172-d949-471e-842c-b340b50e179b-etc-systemd\") pod \"tuned-5hvvb\" (UID: \"4606f172-d949-471e-842c-b340b50e179b\") " pod="openshift-cluster-node-tuning-operator/tuned-5hvvb"
Oct 30 00:31:45.731988 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.730213    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1881dcfe-c2da-4253-bca5-93f3526e14fa-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-2nnk8\" (UID: \"1881dcfe-c2da-4253-bca5-93f3526e14fa\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-2nnk8"
Oct 30 00:31:45.731988 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.730234    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6af42f7e-20f7-4b52-9759-0bf1af4d85b4-systemd-units\") pod \"ovnkube-node-lxg8k\" (UID: \"6af42f7e-20f7-4b52-9759-0bf1af4d85b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxg8k"
Oct 30 00:31:45.731988 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.730245    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a707a5d7-313f-4bef-8baf-a99aeae40d54-cnibin\") pod \"multus-h4hbh\" (UID: \"a707a5d7-313f-4bef-8baf-a99aeae40d54\") " pod="openshift-multus/multus-h4hbh"
Oct 30 00:31:45.731988 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.730251    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6af42f7e-20f7-4b52-9759-0bf1af4d85b4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lxg8k\" (UID: \"6af42f7e-20f7-4b52-9759-0bf1af4d85b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxg8k"
Oct 30 00:31:45.731988 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.730158    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/80603987-09a1-4ca1-b8b4-3c9ac8d0efc6-kubelet-config\") pod \"global-pull-secret-syncer-tz6g8\" (UID: \"80603987-09a1-4ca1-b8b4-3c9ac8d0efc6\") " pod="kube-system/global-pull-secret-syncer-tz6g8"
Oct 30 00:31:45.731988 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.730258    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/88de9f3a-e7d0-400f-9091-46c0b89c1a3c-device-dir\") pod \"aws-ebs-csi-driver-node-v24xq\" (UID: \"88de9f3a-e7d0-400f-9091-46c0b89c1a3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v24xq"
Oct 30 00:31:45.731988 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.730268    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6af42f7e-20f7-4b52-9759-0bf1af4d85b4-etc-openvswitch\") pod \"ovnkube-node-lxg8k\" (UID: \"6af42f7e-20f7-4b52-9759-0bf1af4d85b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxg8k"
Oct 30 00:31:45.731988 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.730291    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/88de9f3a-e7d0-400f-9091-46c0b89c1a3c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-v24xq\" (UID: \"88de9f3a-e7d0-400f-9091-46c0b89c1a3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v24xq"
Oct 30 00:31:45.731988 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.730298    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6af42f7e-20f7-4b52-9759-0bf1af4d85b4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lxg8k\" (UID: \"6af42f7e-20f7-4b52-9759-0bf1af4d85b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxg8k"
Oct 30 00:31:45.731988 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.730291    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6af42f7e-20f7-4b52-9759-0bf1af4d85b4-systemd-units\") pod \"ovnkube-node-lxg8k\" (UID: \"6af42f7e-20f7-4b52-9759-0bf1af4d85b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxg8k"
Oct 30 00:31:45.731988 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.730318    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4606f172-d949-471e-842c-b340b50e179b-etc-sysconfig\") pod \"tuned-5hvvb\" (UID: \"4606f172-d949-471e-842c-b340b50e179b\") " pod="openshift-cluster-node-tuning-operator/tuned-5hvvb"
Oct 30 00:31:45.731988 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.730330    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/88de9f3a-e7d0-400f-9091-46c0b89c1a3c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-v24xq\" (UID: \"88de9f3a-e7d0-400f-9091-46c0b89c1a3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v24xq"
Oct 30 00:31:45.731988 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.730330    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6af42f7e-20f7-4b52-9759-0bf1af4d85b4-etc-openvswitch\") pod \"ovnkube-node-lxg8k\" (UID: \"6af42f7e-20f7-4b52-9759-0bf1af4d85b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxg8k"
Oct 30 00:31:45.731988 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.730343    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4606f172-d949-471e-842c-b340b50e179b-etc-kubernetes\") pod \"tuned-5hvvb\" (UID: \"4606f172-d949-471e-842c-b340b50e179b\") " pod="openshift-cluster-node-tuning-operator/tuned-5hvvb"
Oct 30 00:31:45.731988 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.730360    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4606f172-d949-471e-842c-b340b50e179b-etc-sysconfig\") pod \"tuned-5hvvb\" (UID: \"4606f172-d949-471e-842c-b340b50e179b\") " pod="openshift-cluster-node-tuning-operator/tuned-5hvvb"
Oct 30 00:31:45.731988 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.730365    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4606f172-d949-471e-842c-b340b50e179b-run\") pod \"tuned-5hvvb\" (UID: \"4606f172-d949-471e-842c-b340b50e179b\") " pod="openshift-cluster-node-tuning-operator/tuned-5hvvb"
Oct 30 00:31:45.731988 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.730412    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4606f172-d949-471e-842c-b340b50e179b-etc-kubernetes\") pod \"tuned-5hvvb\" (UID: \"4606f172-d949-471e-842c-b340b50e179b\") " pod="openshift-cluster-node-tuning-operator/tuned-5hvvb"
Oct 30 00:31:45.731988 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.730417    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a707a5d7-313f-4bef-8baf-a99aeae40d54-host-var-lib-kubelet\") pod \"multus-h4hbh\" (UID: \"a707a5d7-313f-4bef-8baf-a99aeae40d54\") " pod="openshift-multus/multus-h4hbh"
Oct 30 00:31:45.731988 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.730439    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a707a5d7-313f-4bef-8baf-a99aeae40d54-host-var-lib-kubelet\") pod \"multus-h4hbh\" (UID: \"a707a5d7-313f-4bef-8baf-a99aeae40d54\") " pod="openshift-multus/multus-h4hbh"
Oct 30 00:31:45.731988 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.730445    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6af42f7e-20f7-4b52-9759-0bf1af4d85b4-host-run-netns\") pod \"ovnkube-node-lxg8k\" (UID: \"6af42f7e-20f7-4b52-9759-0bf1af4d85b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxg8k"
Oct 30 00:31:45.731988 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.730467    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6af42f7e-20f7-4b52-9759-0bf1af4d85b4-host-run-netns\") pod \"ovnkube-node-lxg8k\" (UID: \"6af42f7e-20f7-4b52-9759-0bf1af4d85b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxg8k"
Oct 30 00:31:45.731988 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.730411    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4606f172-d949-471e-842c-b340b50e179b-run\") pod \"tuned-5hvvb\" (UID: \"4606f172-d949-471e-842c-b340b50e179b\") " pod="openshift-cluster-node-tuning-operator/tuned-5hvvb"
Oct 30 00:31:45.731988 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.730579    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/94a6f5e3-d553-487e-a6f1-8c2b0fc5c695-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tbjdq\" (UID: \"94a6f5e3-d553-487e-a6f1-8c2b0fc5c695\") " pod="openshift-insights/insights-runtime-extractor-tbjdq"
Oct 30 00:31:45.733885 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.732283    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3d234b3-76d4-47d5-98a4-dafd83477b17-metrics-certs\") pod \"router-default-85f7b79cf7-h2tpp\" (UID: \"a3d234b3-76d4-47d5-98a4-dafd83477b17\") " pod="openshift-ingress/router-default-85f7b79cf7-h2tpp"
Oct 30 00:31:45.733885 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.732613    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1881dcfe-c2da-4253-bca5-93f3526e14fa-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-2nnk8\" (UID: \"1881dcfe-c2da-4253-bca5-93f3526e14fa\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-2nnk8"
Oct 30 00:31:45.826534 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.826500    5002 scope.go:122] "RemoveContainer" containerID="b9a214f6a9f96ff4dcd1880bfe7e3bbf691313502185f22d37d7989d8110023a"
Oct 30 00:31:45.827984 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:45.827963    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-2nnk8"
Oct 30 00:31:46.182406 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:46.182365    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-2nnk8"]
Oct 30 00:31:46.220927 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:46.220903    5002 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 00:31:46.235282 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:46.235254    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9aba3099-fe4c-4846-9a34-25db2c2fe593-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-kjsj6\" (UID: \"9aba3099-fe4c-4846-9a34-25db2c2fe593\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-kjsj6"
Oct 30 00:31:46.235400 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:46.235291    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5ed36f74-5c9e-408e-a7fb-2364144f054e-registry-tls\") pod \"image-registry-6c69f88f85-88bkg\" (UID: \"5ed36f74-5c9e-408e-a7fb-2364144f054e\") " pod="openshift-image-registry/image-registry-6c69f88f85-88bkg"
Oct 30 00:31:46.235453 ip-10-0-128-7 kubenswrapper[5002]: E1030 00:31:46.235407    5002 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 00:31:46.235453 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:46.235429    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/251cf359-f9f5-44ae-8ba9-42bf0cabb1ab-metrics-tls\") pod \"dns-default-btj6v\" (UID: \"251cf359-f9f5-44ae-8ba9-42bf0cabb1ab\") " pod="openshift-dns/dns-default-btj6v"
Oct 30 00:31:46.235515 ip-10-0-128-7 kubenswrapper[5002]: E1030 00:31:46.235410    5002 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 00:31:46.235515 ip-10-0-128-7 kubenswrapper[5002]: E1030 00:31:46.235464    5002 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6c69f88f85-88bkg: secret "image-registry-tls" not found
Oct 30 00:31:46.235515 ip-10-0-128-7 kubenswrapper[5002]: E1030 00:31:46.235483    5002 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9aba3099-fe4c-4846-9a34-25db2c2fe593-cluster-monitoring-operator-tls podName:9aba3099-fe4c-4846-9a34-25db2c2fe593 nodeName:}" failed. No retries permitted until 2026-10-30 00:31:47.235466085 +0000 UTC m=+2.851898943 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/9aba3099-fe4c-4846-9a34-25db2c2fe593-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-kjsj6" (UID: "9aba3099-fe4c-4846-9a34-25db2c2fe593") : secret "cluster-monitoring-operator-tls" not found
Oct 30 00:31:46.235515 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:46.235503    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed-registry-tls\") pod \"image-registry-6d4f74cbd-njpvm\" (UID: \"2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed\") " pod="openshift-image-registry/image-registry-6d4f74cbd-njpvm"
Oct 30 00:31:46.235627 ip-10-0-128-7 kubenswrapper[5002]: E1030 00:31:46.235516    5002 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 00:31:46.235627 ip-10-0-128-7 kubenswrapper[5002]: E1030 00:31:46.235540    5002 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 00:31:46.235627 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:46.235541    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a3d234b3-76d4-47d5-98a4-dafd83477b17-service-ca-bundle\") pod \"router-default-85f7b79cf7-h2tpp\" (UID: \"a3d234b3-76d4-47d5-98a4-dafd83477b17\") " pod="openshift-ingress/router-default-85f7b79cf7-h2tpp"
Oct 30 00:31:46.235627 ip-10-0-128-7 kubenswrapper[5002]: E1030 00:31:46.235546    5002 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6d4f74cbd-njpvm: secret "image-registry-tls" not found
Oct 30 00:31:46.235627 ip-10-0-128-7 kubenswrapper[5002]: E1030 00:31:46.235553    5002 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/251cf359-f9f5-44ae-8ba9-42bf0cabb1ab-metrics-tls podName:251cf359-f9f5-44ae-8ba9-42bf0cabb1ab nodeName:}" failed. No retries permitted until 2026-10-30 00:31:47.235540742 +0000 UTC m=+2.851973627 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/251cf359-f9f5-44ae-8ba9-42bf0cabb1ab-metrics-tls") pod "dns-default-btj6v" (UID: "251cf359-f9f5-44ae-8ba9-42bf0cabb1ab") : secret "dns-default-metrics-tls" not found
Oct 30 00:31:46.235627 ip-10-0-128-7 kubenswrapper[5002]: E1030 00:31:46.235623    5002 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5ed36f74-5c9e-408e-a7fb-2364144f054e-registry-tls podName:5ed36f74-5c9e-408e-a7fb-2364144f054e nodeName:}" failed. No retries permitted until 2026-10-30 00:31:47.23561211 +0000 UTC m=+2.852044971 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5ed36f74-5c9e-408e-a7fb-2364144f054e-registry-tls") pod "image-registry-6c69f88f85-88bkg" (UID: "5ed36f74-5c9e-408e-a7fb-2364144f054e") : secret "image-registry-tls" not found
Oct 30 00:31:46.235790 ip-10-0-128-7 kubenswrapper[5002]: E1030 00:31:46.235642    5002 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed-registry-tls podName:2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed nodeName:}" failed. No retries permitted until 2026-10-30 00:31:47.235633256 +0000 UTC m=+2.852066113 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed-registry-tls") pod "image-registry-6d4f74cbd-njpvm" (UID: "2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed") : secret "image-registry-tls" not found
Oct 30 00:31:46.235790 ip-10-0-128-7 kubenswrapper[5002]: E1030 00:31:46.235654    5002 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a3d234b3-76d4-47d5-98a4-dafd83477b17-service-ca-bundle podName:a3d234b3-76d4-47d5-98a4-dafd83477b17 nodeName:}" failed. No retries permitted until 2026-10-30 00:31:47.235649849 +0000 UTC m=+2.852082706 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/a3d234b3-76d4-47d5-98a4-dafd83477b17-service-ca-bundle") pod "router-default-85f7b79cf7-h2tpp" (UID: "a3d234b3-76d4-47d5-98a4-dafd83477b17") : configmap references non-existent config key: service-ca.crt
Oct 30 00:31:46.235790 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:46.235682    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af2d6f84-aea3-4698-be12-11bd6767caec-metrics-certs\") pod \"network-metrics-daemon-z58bb\" (UID: \"af2d6f84-aea3-4698-be12-11bd6767caec\") " pod="openshift-multus/network-metrics-daemon-z58bb"
Oct 30 00:31:46.235790 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:46.235709    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d7c5e976-596e-4cb5-81a9-b16262b3c85b-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-zqdbk\" (UID: \"d7c5e976-596e-4cb5-81a9-b16262b3c85b\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-zqdbk"
Oct 30 00:31:46.235790 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:46.235732    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/94a6f5e3-d553-487e-a6f1-8c2b0fc5c695-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tbjdq\" (UID: \"94a6f5e3-d553-487e-a6f1-8c2b0fc5c695\") " pod="openshift-insights/insights-runtime-extractor-tbjdq"
Oct 30 00:31:46.235790 ip-10-0-128-7 kubenswrapper[5002]: E1030 00:31:46.235789    5002 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 00:31:46.235954 ip-10-0-128-7 kubenswrapper[5002]: E1030 00:31:46.235818    5002 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/af2d6f84-aea3-4698-be12-11bd6767caec-metrics-certs podName:af2d6f84-aea3-4698-be12-11bd6767caec nodeName:}" failed. No retries permitted until 2026-10-30 00:31:47.23580827 +0000 UTC m=+2.852241153 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/af2d6f84-aea3-4698-be12-11bd6767caec-metrics-certs") pod "network-metrics-daemon-z58bb" (UID: "af2d6f84-aea3-4698-be12-11bd6767caec") : secret "metrics-daemon-secret" not found
Oct 30 00:31:46.235954 ip-10-0-128-7 kubenswrapper[5002]: E1030 00:31:46.235821    5002 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 00:31:46.235954 ip-10-0-128-7 kubenswrapper[5002]: E1030 00:31:46.235851    5002 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94a6f5e3-d553-487e-a6f1-8c2b0fc5c695-insights-runtime-extractor-tls podName:94a6f5e3-d553-487e-a6f1-8c2b0fc5c695 nodeName:}" failed. No retries permitted until 2026-10-30 00:31:47.235842383 +0000 UTC m=+2.852275239 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/94a6f5e3-d553-487e-a6f1-8c2b0fc5c695-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tbjdq" (UID: "94a6f5e3-d553-487e-a6f1-8c2b0fc5c695") : secret "insights-runtime-extractor-tls" not found
Oct 30 00:31:46.235954 ip-10-0-128-7 kubenswrapper[5002]: E1030 00:31:46.235821    5002 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 00:31:46.235954 ip-10-0-128-7 kubenswrapper[5002]: E1030 00:31:46.235874    5002 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7c5e976-596e-4cb5-81a9-b16262b3c85b-networking-console-plugin-cert podName:d7c5e976-596e-4cb5-81a9-b16262b3c85b nodeName:}" failed. No retries permitted until 2026-10-30 00:31:47.235869316 +0000 UTC m=+2.852302173 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/d7c5e976-596e-4cb5-81a9-b16262b3c85b-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-zqdbk" (UID: "d7c5e976-596e-4cb5-81a9-b16262b3c85b") : secret "networking-console-plugin-cert" not found
Oct 30 00:31:46.307036 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:46.307005    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-m5qgr"
Oct 30 00:31:46.562069 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:46.562028    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-2nnk8" event={"ID":"1881dcfe-c2da-4253-bca5-93f3526e14fa","Type":"ContainerStarted","Data":"51e271818c7a54e7e6d509f3b8d8fed420c067b103dff93210c4722fe4be8c8c"}
Oct 30 00:31:46.563876 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:46.563855    5002 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-g7c2w_5dc56e7e-f3c1-43eb-a21b-8be6bb4706cf/console-operator/1.log"
Oct 30 00:31:46.563996 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:46.563962    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-g7c2w" event={"ID":"5dc56e7e-f3c1-43eb-a21b-8be6bb4706cf","Type":"ContainerStarted","Data":"d5f267f8f82a968656356f77ce2b7d7c68a24fb815a9251feff8292387ec3e87"}
Oct 30 00:31:47.070247 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:47.070210    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-lngrr"
Oct 30 00:31:47.246119 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:47.246075    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af2d6f84-aea3-4698-be12-11bd6767caec-metrics-certs\") pod \"network-metrics-daemon-z58bb\" (UID: \"af2d6f84-aea3-4698-be12-11bd6767caec\") " pod="openshift-multus/network-metrics-daemon-z58bb"
Oct 30 00:31:47.246305 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:47.246138    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d7c5e976-596e-4cb5-81a9-b16262b3c85b-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-zqdbk\" (UID: \"d7c5e976-596e-4cb5-81a9-b16262b3c85b\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-zqdbk"
Oct 30 00:31:47.246305 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:47.246177    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/94a6f5e3-d553-487e-a6f1-8c2b0fc5c695-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tbjdq\" (UID: \"94a6f5e3-d553-487e-a6f1-8c2b0fc5c695\") " pod="openshift-insights/insights-runtime-extractor-tbjdq"
Oct 30 00:31:47.246305 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:47.246214    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9aba3099-fe4c-4846-9a34-25db2c2fe593-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-kjsj6\" (UID: \"9aba3099-fe4c-4846-9a34-25db2c2fe593\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-kjsj6"
Oct 30 00:31:47.246305 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:47.246241    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5ed36f74-5c9e-408e-a7fb-2364144f054e-registry-tls\") pod \"image-registry-6c69f88f85-88bkg\" (UID: \"5ed36f74-5c9e-408e-a7fb-2364144f054e\") " pod="openshift-image-registry/image-registry-6c69f88f85-88bkg"
Oct 30 00:31:47.246305 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:47.246273    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/251cf359-f9f5-44ae-8ba9-42bf0cabb1ab-metrics-tls\") pod \"dns-default-btj6v\" (UID: \"251cf359-f9f5-44ae-8ba9-42bf0cabb1ab\") " pod="openshift-dns/dns-default-btj6v"
Oct 30 00:31:47.246305 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:47.246290    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed-registry-tls\") pod \"image-registry-6d4f74cbd-njpvm\" (UID: \"2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed\") " pod="openshift-image-registry/image-registry-6d4f74cbd-njpvm"
Oct 30 00:31:47.246665 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:47.246314    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a3d234b3-76d4-47d5-98a4-dafd83477b17-service-ca-bundle\") pod \"router-default-85f7b79cf7-h2tpp\" (UID: \"a3d234b3-76d4-47d5-98a4-dafd83477b17\") " pod="openshift-ingress/router-default-85f7b79cf7-h2tpp"
Oct 30 00:31:47.246665 ip-10-0-128-7 kubenswrapper[5002]: E1030 00:31:47.246428    5002 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a3d234b3-76d4-47d5-98a4-dafd83477b17-service-ca-bundle podName:a3d234b3-76d4-47d5-98a4-dafd83477b17 nodeName:}" failed. No retries permitted until 2026-10-30 00:31:49.24641298 +0000 UTC m=+4.862845850 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/a3d234b3-76d4-47d5-98a4-dafd83477b17-service-ca-bundle") pod "router-default-85f7b79cf7-h2tpp" (UID: "a3d234b3-76d4-47d5-98a4-dafd83477b17") : configmap references non-existent config key: service-ca.crt
Oct 30 00:31:47.246665 ip-10-0-128-7 kubenswrapper[5002]: E1030 00:31:47.246491    5002 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 00:31:47.246665 ip-10-0-128-7 kubenswrapper[5002]: E1030 00:31:47.246516    5002 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94a6f5e3-d553-487e-a6f1-8c2b0fc5c695-insights-runtime-extractor-tls podName:94a6f5e3-d553-487e-a6f1-8c2b0fc5c695 nodeName:}" failed. No retries permitted until 2026-10-30 00:31:49.246509265 +0000 UTC m=+4.862942136 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/94a6f5e3-d553-487e-a6f1-8c2b0fc5c695-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tbjdq" (UID: "94a6f5e3-d553-487e-a6f1-8c2b0fc5c695") : secret "insights-runtime-extractor-tls" not found
Oct 30 00:31:47.246665 ip-10-0-128-7 kubenswrapper[5002]: E1030 00:31:47.246553    5002 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 00:31:47.246665 ip-10-0-128-7 kubenswrapper[5002]: E1030 00:31:47.246573    5002 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/251cf359-f9f5-44ae-8ba9-42bf0cabb1ab-metrics-tls podName:251cf359-f9f5-44ae-8ba9-42bf0cabb1ab nodeName:}" failed. No retries permitted until 2026-10-30 00:31:49.246566518 +0000 UTC m=+4.862999389 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/251cf359-f9f5-44ae-8ba9-42bf0cabb1ab-metrics-tls") pod "dns-default-btj6v" (UID: "251cf359-f9f5-44ae-8ba9-42bf0cabb1ab") : secret "dns-default-metrics-tls" not found
Oct 30 00:31:47.246665 ip-10-0-128-7 kubenswrapper[5002]: E1030 00:31:47.246663    5002 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 00:31:47.247085 ip-10-0-128-7 kubenswrapper[5002]: E1030 00:31:47.246712    5002 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9aba3099-fe4c-4846-9a34-25db2c2fe593-cluster-monitoring-operator-tls podName:9aba3099-fe4c-4846-9a34-25db2c2fe593 nodeName:}" failed. No retries permitted until 2026-10-30 00:31:49.246696459 +0000 UTC m=+4.863129320 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/9aba3099-fe4c-4846-9a34-25db2c2fe593-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-kjsj6" (UID: "9aba3099-fe4c-4846-9a34-25db2c2fe593") : secret "cluster-monitoring-operator-tls" not found
Oct 30 00:31:47.248873 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:47.248849    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af2d6f84-aea3-4698-be12-11bd6767caec-metrics-certs\") pod \"network-metrics-daemon-z58bb\" (UID: \"af2d6f84-aea3-4698-be12-11bd6767caec\") " pod="openshift-multus/network-metrics-daemon-z58bb"
Oct 30 00:31:47.249023 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:47.249000    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d7c5e976-596e-4cb5-81a9-b16262b3c85b-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-zqdbk\" (UID: \"d7c5e976-596e-4cb5-81a9-b16262b3c85b\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-zqdbk"
Oct 30 00:31:47.249130 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:47.249110    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed-registry-tls\") pod \"image-registry-6d4f74cbd-njpvm\" (UID: \"2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed\") " pod="openshift-image-registry/image-registry-6d4f74cbd-njpvm"
Oct 30 00:31:47.250797 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:47.250776    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5ed36f74-5c9e-408e-a7fb-2364144f054e-registry-tls\") pod \"image-registry-6c69f88f85-88bkg\" (UID: \"5ed36f74-5c9e-408e-a7fb-2364144f054e\") " pod="openshift-image-registry/image-registry-6c69f88f85-88bkg"
Oct 30 00:31:47.327331 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:47.327252    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6d4f74cbd-njpvm"
Oct 30 00:31:47.328317 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:47.328295    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-zqdbk"
Oct 30 00:31:47.331435 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:47.331250    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z58bb"
Oct 30 00:31:47.335086 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:47.335065    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6c69f88f85-88bkg"
Oct 30 00:31:47.659307 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:47.659281    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-z58bb"]
Oct 30 00:31:47.662116 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:47.662073    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf2d6f84_aea3_4698_be12_11bd6767caec.slice/crio-ed94161edcc2e7150c126acdf8005fd0f54b7315a4a6f97c88153ea64c869c9d WatchSource:0}: Error finding container ed94161edcc2e7150c126acdf8005fd0f54b7315a4a6f97c88153ea64c869c9d: Status 404 returned error can't find the container with id ed94161edcc2e7150c126acdf8005fd0f54b7315a4a6f97c88153ea64c869c9d
Oct 30 00:31:47.669470 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:47.669443    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6d4f74cbd-njpvm"]
Oct 30 00:31:47.678281 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:47.678250    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a1658ed_d4f5_4a0a_9ee9_df362b2a13ed.slice/crio-63d1b3fcd388225474e9c44cad41a566a60e42560c7dcb0f21d1f42df2c87524 WatchSource:0}: Error finding container 63d1b3fcd388225474e9c44cad41a566a60e42560c7dcb0f21d1f42df2c87524: Status 404 returned error can't find the container with id 63d1b3fcd388225474e9c44cad41a566a60e42560c7dcb0f21d1f42df2c87524
Oct 30 00:31:47.700666 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:47.700636    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-zqdbk"]
Oct 30 00:31:47.705502 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:47.705369    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7c5e976_596e_4cb5_81a9_b16262b3c85b.slice/crio-9900949d6e959bbf126bf23890a76438c7971ed6af4ec844fa2156ac21335595 WatchSource:0}: Error finding container 9900949d6e959bbf126bf23890a76438c7971ed6af4ec844fa2156ac21335595: Status 404 returned error can't find the container with id 9900949d6e959bbf126bf23890a76438c7971ed6af4ec844fa2156ac21335595
Oct 30 00:31:47.705988 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:47.705967    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6c69f88f85-88bkg"]
Oct 30 00:31:47.707567 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:47.707542    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ed36f74_5c9e_408e_a7fb_2364144f054e.slice/crio-910db8ae6b6a9af3f1879820859bd9e54ee5da41548aba5d0414e4cdab3c4b65 WatchSource:0}: Error finding container 910db8ae6b6a9af3f1879820859bd9e54ee5da41548aba5d0414e4cdab3c4b65: Status 404 returned error can't find the container with id 910db8ae6b6a9af3f1879820859bd9e54ee5da41548aba5d0414e4cdab3c4b65
Oct 30 00:31:48.573640 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:48.573554    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-z58bb" event={"ID":"af2d6f84-aea3-4698-be12-11bd6767caec","Type":"ContainerStarted","Data":"ed94161edcc2e7150c126acdf8005fd0f54b7315a4a6f97c88153ea64c869c9d"}
Oct 30 00:31:48.575070 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:48.575038    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-zqdbk" event={"ID":"d7c5e976-596e-4cb5-81a9-b16262b3c85b","Type":"ContainerStarted","Data":"9900949d6e959bbf126bf23890a76438c7971ed6af4ec844fa2156ac21335595"}
Oct 30 00:31:48.577100 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:48.577073    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6d4f74cbd-njpvm" event={"ID":"2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed","Type":"ContainerStarted","Data":"8705722d28440e3097208f046fbfcdd85cde2371a15e8579786616cb49af6de1"}
Oct 30 00:31:48.577208 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:48.577119    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6d4f74cbd-njpvm" event={"ID":"2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed","Type":"ContainerStarted","Data":"63d1b3fcd388225474e9c44cad41a566a60e42560c7dcb0f21d1f42df2c87524"}
Oct 30 00:31:48.579194 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:48.579168    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-2nnk8" event={"ID":"1881dcfe-c2da-4253-bca5-93f3526e14fa","Type":"ContainerStarted","Data":"f852f5ffddd68149d6ee2b419ef0a3ba959476a9f499551fc61ce37a2d58c6f3"}
Oct 30 00:31:48.579301 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:48.579215    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-2nnk8" event={"ID":"1881dcfe-c2da-4253-bca5-93f3526e14fa","Type":"ContainerStarted","Data":"826043b331d80b1c2c1f5d8c2b6c6a964322f219967799622888235fbd0df6c8"}
Oct 30 00:31:48.580862 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:48.580837    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6c69f88f85-88bkg" event={"ID":"5ed36f74-5c9e-408e-a7fb-2364144f054e","Type":"ContainerStarted","Data":"0d54e2296511e2f2f11e01e551bf9b779b9137f44e51c2662257bd19446606fe"}
Oct 30 00:31:48.580941 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:48.580882    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6c69f88f85-88bkg" event={"ID":"5ed36f74-5c9e-408e-a7fb-2364144f054e","Type":"ContainerStarted","Data":"910db8ae6b6a9af3f1879820859bd9e54ee5da41548aba5d0414e4cdab3c4b65"}
Oct 30 00:31:48.581000 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:48.580959    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6c69f88f85-88bkg"
Oct 30 00:31:48.849268 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:48.849239    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6d4f74cbd-njpvm"
Oct 30 00:31:49.268043 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:49.268011    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9aba3099-fe4c-4846-9a34-25db2c2fe593-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-kjsj6\" (UID: \"9aba3099-fe4c-4846-9a34-25db2c2fe593\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-kjsj6"
Oct 30 00:31:49.268198 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:49.268057    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/251cf359-f9f5-44ae-8ba9-42bf0cabb1ab-metrics-tls\") pod \"dns-default-btj6v\" (UID: \"251cf359-f9f5-44ae-8ba9-42bf0cabb1ab\") " pod="openshift-dns/dns-default-btj6v"
Oct 30 00:31:49.268198 ip-10-0-128-7 kubenswrapper[5002]: E1030 00:31:49.268163    5002 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 00:31:49.268198 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:49.268184    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a3d234b3-76d4-47d5-98a4-dafd83477b17-service-ca-bundle\") pod \"router-default-85f7b79cf7-h2tpp\" (UID: \"a3d234b3-76d4-47d5-98a4-dafd83477b17\") " pod="openshift-ingress/router-default-85f7b79cf7-h2tpp"
Oct 30 00:31:49.268335 ip-10-0-128-7 kubenswrapper[5002]: E1030 00:31:49.268239    5002 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9aba3099-fe4c-4846-9a34-25db2c2fe593-cluster-monitoring-operator-tls podName:9aba3099-fe4c-4846-9a34-25db2c2fe593 nodeName:}" failed. No retries permitted until 2026-10-30 00:31:53.268220999 +0000 UTC m=+8.884653858 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/9aba3099-fe4c-4846-9a34-25db2c2fe593-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-kjsj6" (UID: "9aba3099-fe4c-4846-9a34-25db2c2fe593") : secret "cluster-monitoring-operator-tls" not found
Oct 30 00:31:49.268335 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:49.268293    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/94a6f5e3-d553-487e-a6f1-8c2b0fc5c695-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tbjdq\" (UID: \"94a6f5e3-d553-487e-a6f1-8c2b0fc5c695\") " pod="openshift-insights/insights-runtime-extractor-tbjdq"
Oct 30 00:31:49.268465 ip-10-0-128-7 kubenswrapper[5002]: E1030 00:31:49.268451    5002 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 00:31:49.268526 ip-10-0-128-7 kubenswrapper[5002]: E1030 00:31:49.268514    5002 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94a6f5e3-d553-487e-a6f1-8c2b0fc5c695-insights-runtime-extractor-tls podName:94a6f5e3-d553-487e-a6f1-8c2b0fc5c695 nodeName:}" failed. No retries permitted until 2026-10-30 00:31:53.268499448 +0000 UTC m=+8.884932305 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/94a6f5e3-d553-487e-a6f1-8c2b0fc5c695-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tbjdq" (UID: "94a6f5e3-d553-487e-a6f1-8c2b0fc5c695") : secret "insights-runtime-extractor-tls" not found
Oct 30 00:31:49.268845 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:49.268826    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a3d234b3-76d4-47d5-98a4-dafd83477b17-service-ca-bundle\") pod \"router-default-85f7b79cf7-h2tpp\" (UID: \"a3d234b3-76d4-47d5-98a4-dafd83477b17\") " pod="openshift-ingress/router-default-85f7b79cf7-h2tpp"
Oct 30 00:31:49.270289 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:49.270271    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/251cf359-f9f5-44ae-8ba9-42bf0cabb1ab-metrics-tls\") pod \"dns-default-btj6v\" (UID: \"251cf359-f9f5-44ae-8ba9-42bf0cabb1ab\") " pod="openshift-dns/dns-default-btj6v"
Oct 30 00:31:49.428218 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:49.428139    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-85f7b79cf7-h2tpp"
Oct 30 00:31:49.435498 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:49.435472    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-btj6v"
Oct 30 00:31:49.586496 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:49.586450    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-z58bb" event={"ID":"af2d6f84-aea3-4698-be12-11bd6767caec","Type":"ContainerStarted","Data":"6c3141d860472ca1d147f7d886d1f920b3945ef05c9d627252cf9da364d69b97"}
Oct 30 00:31:49.586699 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:49.586507    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-z58bb" event={"ID":"af2d6f84-aea3-4698-be12-11bd6767caec","Type":"ContainerStarted","Data":"db6e894250171152ff18cd2178148a615f6ed5900f2eeaf0bec4cdfeb7360caa"}
Oct 30 00:31:49.588069 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:49.588041    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-zqdbk" event={"ID":"d7c5e976-596e-4cb5-81a9-b16262b3c85b","Type":"ContainerStarted","Data":"74a94be8afee76214c16508fb3b9608db2bc1041c1e7c73b0ec31a3a1a3f03d5"}
Oct 30 00:31:49.634653 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:49.634621    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lxg8k"
Oct 30 00:31:49.739643 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:49.739607    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-85f7b79cf7-h2tpp"]
Oct 30 00:31:49.744174 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:49.744139    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3d234b3_76d4_47d5_98a4_dafd83477b17.slice/crio-a66a091de941854797df987606b003052b4628f1795b74316760b22ff576e86d WatchSource:0}: Error finding container a66a091de941854797df987606b003052b4628f1795b74316760b22ff576e86d: Status 404 returned error can't find the container with id a66a091de941854797df987606b003052b4628f1795b74316760b22ff576e86d
Oct 30 00:31:49.745972 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:49.745953    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-btj6v"]
Oct 30 00:31:49.747348 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:49.747293    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod251cf359_f9f5_44ae_8ba9_42bf0cabb1ab.slice/crio-9930b6e6993fd1f5cce665647c456371b9dfc4d7aaafc9d7235e2fdc28292603 WatchSource:0}: Error finding container 9930b6e6993fd1f5cce665647c456371b9dfc4d7aaafc9d7235e2fdc28292603: Status 404 returned error can't find the container with id 9930b6e6993fd1f5cce665647c456371b9dfc4d7aaafc9d7235e2fdc28292603
Oct 30 00:31:50.594028 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:50.593986    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-85f7b79cf7-h2tpp" event={"ID":"a3d234b3-76d4-47d5-98a4-dafd83477b17","Type":"ContainerStarted","Data":"e5a97f4c705ec1e24326ae7eec9069b4fb29a3631c5318649fcd89efe044b75a"}
Oct 30 00:31:50.594526 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:50.594051    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-85f7b79cf7-h2tpp" event={"ID":"a3d234b3-76d4-47d5-98a4-dafd83477b17","Type":"ContainerStarted","Data":"a66a091de941854797df987606b003052b4628f1795b74316760b22ff576e86d"}
Oct 30 00:31:50.595577 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:50.595545    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-btj6v" event={"ID":"251cf359-f9f5-44ae-8ba9-42bf0cabb1ab","Type":"ContainerStarted","Data":"9930b6e6993fd1f5cce665647c456371b9dfc4d7aaafc9d7235e2fdc28292603"}
Oct 30 00:31:51.428651 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:51.428617    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-85f7b79cf7-h2tpp"
Oct 30 00:31:51.431690 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:51.431661    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-85f7b79cf7-h2tpp"
Oct 30 00:31:51.600658 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:51.600618    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-btj6v" event={"ID":"251cf359-f9f5-44ae-8ba9-42bf0cabb1ab","Type":"ContainerStarted","Data":"318dee82049a5ba863374bfbd930352546b3e29ee702a3def44456f96b07b01f"}
Oct 30 00:31:51.601067 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:51.600676    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-btj6v" event={"ID":"251cf359-f9f5-44ae-8ba9-42bf0cabb1ab","Type":"ContainerStarted","Data":"1106592f22379551697f2afab2e02f462ce4287a77a4fe678c4c7b55b518a43f"}
Oct 30 00:31:51.916806 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:51.916761    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-g7c2w"
Oct 30 00:31:51.917008 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:51.916897    5002 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 00:31:51.920527 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:51.920506    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-g7c2w"
Oct 30 00:31:52.387257 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:52.387229    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lxg8k"
Oct 30 00:31:52.387514 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:52.387498    5002 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 00:31:52.436881 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:52.436856    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lxg8k"
Oct 30 00:31:52.604424 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:52.604392    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-btj6v"
Oct 30 00:31:52.605023 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:52.605004    5002 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 00:31:53.310279 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:53.310236    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/94a6f5e3-d553-487e-a6f1-8c2b0fc5c695-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tbjdq\" (UID: \"94a6f5e3-d553-487e-a6f1-8c2b0fc5c695\") " pod="openshift-insights/insights-runtime-extractor-tbjdq"
Oct 30 00:31:53.310444 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:53.310296    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9aba3099-fe4c-4846-9a34-25db2c2fe593-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-kjsj6\" (UID: \"9aba3099-fe4c-4846-9a34-25db2c2fe593\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-kjsj6"
Oct 30 00:31:53.310444 ip-10-0-128-7 kubenswrapper[5002]: E1030 00:31:53.310416    5002 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 00:31:53.310531 ip-10-0-128-7 kubenswrapper[5002]: E1030 00:31:53.310498    5002 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94a6f5e3-d553-487e-a6f1-8c2b0fc5c695-insights-runtime-extractor-tls podName:94a6f5e3-d553-487e-a6f1-8c2b0fc5c695 nodeName:}" failed. No retries permitted until 2026-10-30 00:32:01.310481517 +0000 UTC m=+16.926914377 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/94a6f5e3-d553-487e-a6f1-8c2b0fc5c695-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tbjdq" (UID: "94a6f5e3-d553-487e-a6f1-8c2b0fc5c695") : secret "insights-runtime-extractor-tls" not found
Oct 30 00:31:53.312682 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:53.312657    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9aba3099-fe4c-4846-9a34-25db2c2fe593-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-kjsj6\" (UID: \"9aba3099-fe4c-4846-9a34-25db2c2fe593\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-kjsj6"
Oct 30 00:31:53.326734 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:53.326712    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-kjsj6"
Oct 30 00:31:53.619751 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:53.619729    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-kjsj6"]
Oct 30 00:31:53.621289 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:31:53.621259    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9aba3099_fe4c_4846_9a34_25db2c2fe593.slice/crio-7019a186a3fcce2cd59a29b2515684551460a4e4d68664e4e3bff5a65f2e0fee WatchSource:0}: Error finding container 7019a186a3fcce2cd59a29b2515684551460a4e4d68664e4e3bff5a65f2e0fee: Status 404 returned error can't find the container with id 7019a186a3fcce2cd59a29b2515684551460a4e4d68664e4e3bff5a65f2e0fee
Oct 30 00:31:54.613219 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:54.613173    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-kjsj6" event={"ID":"9aba3099-fe4c-4846-9a34-25db2c2fe593","Type":"ContainerStarted","Data":"7019a186a3fcce2cd59a29b2515684551460a4e4d68664e4e3bff5a65f2e0fee"}
Oct 30 00:31:55.618859 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:55.618778    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-kjsj6" event={"ID":"9aba3099-fe4c-4846-9a34-25db2c2fe593","Type":"ContainerStarted","Data":"f3f69576374cc7975676fdb887cc830c8fcb70b961114e9c20ea6529bc8e4305"}
Oct 30 00:31:55.686140 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:55.686106    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-85f7b79cf7-h2tpp"
Oct 30 00:31:55.686321 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:55.686228    5002 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 00:31:55.687573 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:31:55.687550    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-85f7b79cf7-h2tpp"
Oct 30 00:32:00.057312 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:32:00.057276    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6d4f74cbd-njpvm"]
Oct 30 00:32:01.374880 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:32:01.374834    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/94a6f5e3-d553-487e-a6f1-8c2b0fc5c695-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tbjdq\" (UID: \"94a6f5e3-d553-487e-a6f1-8c2b0fc5c695\") " pod="openshift-insights/insights-runtime-extractor-tbjdq"
Oct 30 00:32:01.377237 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:32:01.377190    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/94a6f5e3-d553-487e-a6f1-8c2b0fc5c695-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tbjdq\" (UID: \"94a6f5e3-d553-487e-a6f1-8c2b0fc5c695\") " pod="openshift-insights/insights-runtime-extractor-tbjdq"
Oct 30 00:32:01.437566 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:32:01.437532    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-tbjdq"
Oct 30 00:32:01.734147 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:32:01.734118    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-tbjdq"]
Oct 30 00:32:01.735544 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:32:01.735519    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94a6f5e3_d553_487e_a6f1_8c2b0fc5c695.slice/crio-5eada04eaea0cabb6bed1cf30e894c603422ecbb5f2723373a55376a7ac711ef WatchSource:0}: Error finding container 5eada04eaea0cabb6bed1cf30e894c603422ecbb5f2723373a55376a7ac711ef: Status 404 returned error can't find the container with id 5eada04eaea0cabb6bed1cf30e894c603422ecbb5f2723373a55376a7ac711ef
Oct 30 00:32:02.609770 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:32:02.609742    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-btj6v"
Oct 30 00:32:02.646534 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:32:02.646451    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-tbjdq" event={"ID":"94a6f5e3-d553-487e-a6f1-8c2b0fc5c695","Type":"ContainerStarted","Data":"011ea613ba99db437ed8a302e2968155a67aedb9bfd99b104b40976b74541b6b"}
Oct 30 00:32:02.646534 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:32:02.646516    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-tbjdq" event={"ID":"94a6f5e3-d553-487e-a6f1-8c2b0fc5c695","Type":"ContainerStarted","Data":"eeb1a784a0b056a3de6a2b69206435808fecf031c19987f8b2e9fe92997afab2"}
Oct 30 00:32:02.646534 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:32:02.646536    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-tbjdq" event={"ID":"94a6f5e3-d553-487e-a6f1-8c2b0fc5c695","Type":"ContainerStarted","Data":"5eada04eaea0cabb6bed1cf30e894c603422ecbb5f2723373a55376a7ac711ef"}
Oct 30 00:32:04.656362 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:32:04.656326    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-tbjdq" event={"ID":"94a6f5e3-d553-487e-a6f1-8c2b0fc5c695","Type":"ContainerStarted","Data":"7baff727d0db1ddf24032cd3e64cf69d68e6f0d831459ce96875d127afe24718"}
Oct 30 00:32:07.387571 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:32:07.387531    5002 patch_prober.go:29] interesting pod/image-registry-6c69f88f85-88bkg container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Oct 30 00:32:07.387974 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:32:07.387604    5002 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-6c69f88f85-88bkg" podUID="5ed36f74-5c9e-408e-a7fb-2364144f054e" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 00:32:08.183267 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:32:08.183236    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-hxbpg"]
Oct 30 00:32:08.203316 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:32:08.203289    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-hxbpg"
Oct 30 00:32:08.230051 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:32:08.230020    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4320a582-7ff0-456a-aa56-800a11ea6333-sys\") pod \"node-exporter-hxbpg\" (UID: \"4320a582-7ff0-456a-aa56-800a11ea6333\") " pod="openshift-monitoring/node-exporter-hxbpg"
Oct 30 00:32:08.230188 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:32:08.230067    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4320a582-7ff0-456a-aa56-800a11ea6333-root\") pod \"node-exporter-hxbpg\" (UID: \"4320a582-7ff0-456a-aa56-800a11ea6333\") " pod="openshift-monitoring/node-exporter-hxbpg"
Oct 30 00:32:08.230232 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:32:08.230190    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4320a582-7ff0-456a-aa56-800a11ea6333-node-exporter-textfile\") pod \"node-exporter-hxbpg\" (UID: \"4320a582-7ff0-456a-aa56-800a11ea6333\") " pod="openshift-monitoring/node-exporter-hxbpg"
Oct 30 00:32:08.230232 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:32:08.230219    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4320a582-7ff0-456a-aa56-800a11ea6333-metrics-client-ca\") pod \"node-exporter-hxbpg\" (UID: \"4320a582-7ff0-456a-aa56-800a11ea6333\") " pod="openshift-monitoring/node-exporter-hxbpg"
Oct 30 00:32:08.230294 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:32:08.230278    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4320a582-7ff0-456a-aa56-800a11ea6333-node-exporter-tls\") pod \"node-exporter-hxbpg\" (UID: \"4320a582-7ff0-456a-aa56-800a11ea6333\") " pod="openshift-monitoring/node-exporter-hxbpg"
Oct 30 00:32:08.230342 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:32:08.230329    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4320a582-7ff0-456a-aa56-800a11ea6333-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-hxbpg\" (UID: \"4320a582-7ff0-456a-aa56-800a11ea6333\") " pod="openshift-monitoring/node-exporter-hxbpg"
Oct 30 00:32:08.230390 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:32:08.230354    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4320a582-7ff0-456a-aa56-800a11ea6333-node-exporter-wtmp\") pod \"node-exporter-hxbpg\" (UID: \"4320a582-7ff0-456a-aa56-800a11ea6333\") " pod="openshift-monitoring/node-exporter-hxbpg"
Oct 30 00:32:08.230429 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:32:08.230373    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lph8r\" (UniqueName: \"kubernetes.io/projected/4320a582-7ff0-456a-aa56-800a11ea6333-kube-api-access-lph8r\") pod \"node-exporter-hxbpg\" (UID: \"4320a582-7ff0-456a-aa56-800a11ea6333\") " pod="openshift-monitoring/node-exporter-hxbpg"
Oct 30 00:32:08.230461 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:32:08.230443    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4320a582-7ff0-456a-aa56-800a11ea6333-node-exporter-accelerators-collector-config\") pod \"node-exporter-hxbpg\" (UID: \"4320a582-7ff0-456a-aa56-800a11ea6333\") " pod="openshift-monitoring/node-exporter-hxbpg"
Oct 30 00:32:08.235682 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:32:08.235661    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Oct 30 00:32:08.235682 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:32:08.235665    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Oct 30 00:32:08.235826 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:32:08.235683    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-w25sk\""
Oct 30 00:32:08.248958 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:32:08.248939    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Oct 30 00:32:08.249590 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:32:08.249572    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Oct 30 00:32:08.331341 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:32:08.331306    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4320a582-7ff0-456a-aa56-800a11ea6333-node-exporter-textfile\") pod \"node-exporter-hxbpg\" (UID: \"4320a582-7ff0-456a-aa56-800a11ea6333\") " pod="openshift-monitoring/node-exporter-hxbpg"
Oct 30 00:32:08.331341 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:32:08.331346    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4320a582-7ff0-456a-aa56-800a11ea6333-metrics-client-ca\") pod \"node-exporter-hxbpg\" (UID: \"4320a582-7ff0-456a-aa56-800a11ea6333\") " pod="openshift-monitoring/node-exporter-hxbpg"
Oct 30 00:32:08.331604 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:32:08.331367    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4320a582-7ff0-456a-aa56-800a11ea6333-node-exporter-tls\") pod \"node-exporter-hxbpg\" (UID: \"4320a582-7ff0-456a-aa56-800a11ea6333\") " pod="openshift-monitoring/node-exporter-hxbpg"
Oct 30 00:32:08.331604 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:32:08.331417    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4320a582-7ff0-456a-aa56-800a11ea6333-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-hxbpg\" (UID: \"4320a582-7ff0-456a-aa56-800a11ea6333\") " pod="openshift-monitoring/node-exporter-hxbpg"
Oct 30 00:32:08.331604 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:32:08.331551    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4320a582-7ff0-456a-aa56-800a11ea6333-node-exporter-wtmp\") pod \"node-exporter-hxbpg\" (UID: \"4320a582-7ff0-456a-aa56-800a11ea6333\") " pod="openshift-monitoring/node-exporter-hxbpg"
Oct 30 00:32:08.331604 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:32:08.331590    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lph8r\" (UniqueName: \"kubernetes.io/projected/4320a582-7ff0-456a-aa56-800a11ea6333-kube-api-access-lph8r\") pod \"node-exporter-hxbpg\" (UID: \"4320a582-7ff0-456a-aa56-800a11ea6333\") " pod="openshift-monitoring/node-exporter-hxbpg"
Oct 30 00:32:08.331814 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:32:08.331620    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4320a582-7ff0-456a-aa56-800a11ea6333-node-exporter-accelerators-collector-config\") pod \"node-exporter-hxbpg\" (UID: \"4320a582-7ff0-456a-aa56-800a11ea6333\") " pod="openshift-monitoring/node-exporter-hxbpg"
Oct 30 00:32:08.331814 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:32:08.331655    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4320a582-7ff0-456a-aa56-800a11ea6333-sys\") pod \"node-exporter-hxbpg\" (UID: \"4320a582-7ff0-456a-aa56-800a11ea6333\") " pod="openshift-monitoring/node-exporter-hxbpg"
Oct 30 00:32:08.331814 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:32:08.331701    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4320a582-7ff0-456a-aa56-800a11ea6333-sys\") pod \"node-exporter-hxbpg\" (UID: \"4320a582-7ff0-456a-aa56-800a11ea6333\") " pod="openshift-monitoring/node-exporter-hxbpg"
Oct 30 00:32:08.331814 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:32:08.331744    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4320a582-7ff0-456a-aa56-800a11ea6333-node-exporter-wtmp\") pod \"node-exporter-hxbpg\" (UID: \"4320a582-7ff0-456a-aa56-800a11ea6333\") " pod="openshift-monitoring/node-exporter-hxbpg"
Oct 30 00:32:08.331814 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:32:08.331754    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4320a582-7ff0-456a-aa56-800a11ea6333-root\") pod \"node-exporter-hxbpg\" (UID: \"4320a582-7ff0-456a-aa56-800a11ea6333\") " pod="openshift-monitoring/node-exporter-hxbpg"
Oct 30 00:32:08.331814 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:32:08.331703    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4320a582-7ff0-456a-aa56-800a11ea6333-root\") pod \"node-exporter-hxbpg\" (UID: \"4320a582-7ff0-456a-aa56-800a11ea6333\") " pod="openshift-monitoring/node-exporter-hxbpg"
Oct 30 00:32:08.332213 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:32:08.332001    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4320a582-7ff0-456a-aa56-800a11ea6333-node-exporter-textfile\") pod \"node-exporter-hxbpg\" (UID: \"4320a582-7ff0-456a-aa56-800a11ea6333\") " pod="openshift-monitoring/node-exporter-hxbpg"
Oct 30 00:32:08.332213 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:32:08.332074    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4320a582-7ff0-456a-aa56-800a11ea6333-metrics-client-ca\") pod \"node-exporter-hxbpg\" (UID: \"4320a582-7ff0-456a-aa56-800a11ea6333\") " pod="openshift-monitoring/node-exporter-hxbpg"
Oct 30 00:32:08.332308 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:32:08.332285    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4320a582-7ff0-456a-aa56-800a11ea6333-node-exporter-accelerators-collector-config\") pod \"node-exporter-hxbpg\" (UID: \"4320a582-7ff0-456a-aa56-800a11ea6333\") " pod="openshift-monitoring/node-exporter-hxbpg"
Oct 30 00:32:08.333937 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:32:08.333917    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4320a582-7ff0-456a-aa56-800a11ea6333-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-hxbpg\" (UID: \"4320a582-7ff0-456a-aa56-800a11ea6333\") " pod="openshift-monitoring/node-exporter-hxbpg"
Oct 30 00:32:08.333994 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:32:08.333948    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4320a582-7ff0-456a-aa56-800a11ea6333-node-exporter-tls\") pod \"node-exporter-hxbpg\" (UID: \"4320a582-7ff0-456a-aa56-800a11ea6333\") " pod="openshift-monitoring/node-exporter-hxbpg"
Oct 30 00:32:08.356856 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:32:08.356825    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lph8r\" (UniqueName: \"kubernetes.io/projected/4320a582-7ff0-456a-aa56-800a11ea6333-kube-api-access-lph8r\") pod \"node-exporter-hxbpg\" (UID: \"4320a582-7ff0-456a-aa56-800a11ea6333\") " pod="openshift-monitoring/node-exporter-hxbpg"
Oct 30 00:32:08.545782 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:32:08.545703    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-hxbpg"
Oct 30 00:32:08.581751 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:32:08.581720    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4320a582_7ff0_456a_aa56_800a11ea6333.slice/crio-09558b04136eb52f265be781f08533d17949c9862f4c0431d5c1c5636316316f WatchSource:0}: Error finding container 09558b04136eb52f265be781f08533d17949c9862f4c0431d5c1c5636316316f: Status 404 returned error can't find the container with id 09558b04136eb52f265be781f08533d17949c9862f4c0431d5c1c5636316316f
Oct 30 00:32:08.673214 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:32:08.673170    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hxbpg" event={"ID":"4320a582-7ff0-456a-aa56-800a11ea6333","Type":"ContainerStarted","Data":"09558b04136eb52f265be781f08533d17949c9862f4c0431d5c1c5636316316f"}
Oct 30 00:32:09.592574 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:32:09.592547    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6c69f88f85-88bkg"
Oct 30 00:32:10.063463 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:32:10.063435    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6d4f74cbd-njpvm"
Oct 30 00:32:10.683322 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:32:10.683284    5002 generic.go:356] "Generic (PLEG): container finished" podID="4320a582-7ff0-456a-aa56-800a11ea6333" containerID="44b6ef92ea61a64c02fc5e946946c7077d9b632fb324a9a22a075d2b1f9acee3" exitCode=0
Oct 30 00:32:10.683798 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:32:10.683394    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hxbpg" event={"ID":"4320a582-7ff0-456a-aa56-800a11ea6333","Type":"ContainerDied","Data":"44b6ef92ea61a64c02fc5e946946c7077d9b632fb324a9a22a075d2b1f9acee3"}
Oct 30 00:32:11.689945 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:32:11.689912    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hxbpg" event={"ID":"4320a582-7ff0-456a-aa56-800a11ea6333","Type":"ContainerStarted","Data":"1b2043fa845ccf7e6a92d697e2febdb667fa0ccd286d003056f161a016621950"}
Oct 30 00:32:11.690339 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:32:11.689975    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hxbpg" event={"ID":"4320a582-7ff0-456a-aa56-800a11ea6333","Type":"ContainerStarted","Data":"f441859f38b0c17d845d5bab5e9c1bf43e74d78ce5f4170bbe4badd2e423ede9"}
Oct 30 00:32:11.764685 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:32:11.764634    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-hxbpg" podStartSLOduration=2.410094441 podStartE2EDuration="3.764621069s" totalImagesPullingTime="1.27458697s" totalInitContainerRuntime="79.939658ms" isStatefulPod=true podCreationTimestamp="2026-10-30 00:32:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:32:11.764123923 +0000 UTC m=+27.380556805" watchObservedRunningTime="2026-10-30 00:32:11.764621069 +0000 UTC m=+27.381053983"
Oct 30 00:32:12.860460 ip-10-0-128-7 kubenswrapper[5002]: E1030 00:32:12.860426    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbjdq" containerName="extractor"
Oct 30 00:32:17.073206 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:32:17.073175    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-lngrr"
Oct 30 00:32:23.931792 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:32:23.931757    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6c69f88f85-88bkg"]
Oct 30 00:32:25.080053 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:32:25.079990    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-6d4f74cbd-njpvm" podUID="2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed" containerName="registry" containerID="cri-o://8705722d28440e3097208f046fbfcdd85cde2371a15e8579786616cb49af6de1" gracePeriod=30
Oct 30 00:32:25.332303 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:32:25.332246    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6d4f74cbd-njpvm"
Oct 30 00:32:25.475489 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:32:25.475454    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed-image-registry-private-configuration\") pod \"2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed\" (UID: \"2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed\") "
Oct 30 00:32:25.475690 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:32:25.475501    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed-trusted-ca\") pod \"2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed\" (UID: \"2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed\") "
Oct 30 00:32:25.475690 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:32:25.475521    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed-registry-certificates\") pod \"2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed\" (UID: \"2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed\") "
Oct 30 00:32:25.475811 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:32:25.475668    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed-bound-sa-token\") pod \"2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed\" (UID: \"2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed\") "
Oct 30 00:32:25.475811 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:32:25.475774    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed-registry-tls\" (UniqueName: \"kubernetes.io/projected/2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed-registry-tls\") pod \"2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed\" (UID: \"2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed\") "
Oct 30 00:32:25.475924 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:32:25.475812    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed-ca-trust-extracted\") pod \"2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed\" (UID: \"2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed\") "
Oct 30 00:32:25.475924 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:32:25.475857    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed-installation-pull-secrets\") pod \"2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed\" (UID: \"2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed\") "
Oct 30 00:32:25.475924 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:32:25.475902    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed-kube-api-access-84rlf\" (UniqueName: \"kubernetes.io/projected/2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed-kube-api-access-84rlf\") pod \"2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed\" (UID: \"2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed\") "
Oct 30 00:32:25.476065 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:32:25.475983    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed-trusted-ca" pod "2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed" (UID: "2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:32:25.476139 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:32:25.476110    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed-registry-certificates" pod "2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed" (UID: "2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:32:25.476348 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:32:25.476325    5002 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed-trusted-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:32:25.476545 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:32:25.476521    5002 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed-registry-certificates\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:32:25.478108 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:32:25.478063    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed-image-registry-private-configuration" pod "2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed" (UID: "2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:32:25.478201 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:32:25.478178    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed-registry-tls" pod "2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed" (UID: "2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:32:25.478302 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:32:25.478274    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed-bound-sa-token" pod "2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed" (UID: "2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:32:25.478506 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:32:25.478484    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed-kube-api-access-84rlf" pod "2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed" (UID: "2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed"). InnerVolumeSpecName "kube-api-access-84rlf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:32:25.478569 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:32:25.478512    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed-installation-pull-secrets" pod "2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed" (UID: "2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:32:25.482725 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:32:25.482620    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed-ca-trust-extracted" pod "2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed" (UID: "2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:32:25.577121 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:32:25.577086    5002 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed-image-registry-private-configuration\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:32:25.577121 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:32:25.577115    5002 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed-bound-sa-token\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:32:25.577121 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:32:25.577126    5002 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed-registry-tls\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:32:25.577341 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:32:25.577135    5002 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed-ca-trust-extracted\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:32:25.577341 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:32:25.577143    5002 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed-installation-pull-secrets\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:32:25.577341 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:32:25.577152    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-84rlf\" (UniqueName: \"kubernetes.io/projected/2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed-kube-api-access-84rlf\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:32:25.747291 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:32:25.747213    5002 generic.go:356] "Generic (PLEG): container finished" podID="2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed" containerID="8705722d28440e3097208f046fbfcdd85cde2371a15e8579786616cb49af6de1" exitCode=0
Oct 30 00:32:25.747291 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:32:25.747262    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6d4f74cbd-njpvm"
Oct 30 00:32:25.747513 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:32:25.747299    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6d4f74cbd-njpvm" event={"ID":"2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed","Type":"ContainerDied","Data":"8705722d28440e3097208f046fbfcdd85cde2371a15e8579786616cb49af6de1"}
Oct 30 00:32:25.747513 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:32:25.747355    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6d4f74cbd-njpvm" event={"ID":"2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed","Type":"ContainerDied","Data":"63d1b3fcd388225474e9c44cad41a566a60e42560c7dcb0f21d1f42df2c87524"}
Oct 30 00:32:25.747513 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:32:25.747395    5002 scope.go:122] "RemoveContainer" containerID="8705722d28440e3097208f046fbfcdd85cde2371a15e8579786616cb49af6de1"
Oct 30 00:32:25.760408 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:32:25.760365    5002 scope.go:122] "RemoveContainer" containerID="8705722d28440e3097208f046fbfcdd85cde2371a15e8579786616cb49af6de1"
Oct 30 00:32:25.760702 ip-10-0-128-7 kubenswrapper[5002]: E1030 00:32:25.760680    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8705722d28440e3097208f046fbfcdd85cde2371a15e8579786616cb49af6de1\": container with ID starting with 8705722d28440e3097208f046fbfcdd85cde2371a15e8579786616cb49af6de1 not found: ID does not exist" containerID="8705722d28440e3097208f046fbfcdd85cde2371a15e8579786616cb49af6de1"
Oct 30 00:32:25.760774 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:32:25.760715    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8705722d28440e3097208f046fbfcdd85cde2371a15e8579786616cb49af6de1"} err="failed to get container status \"8705722d28440e3097208f046fbfcdd85cde2371a15e8579786616cb49af6de1\": rpc error: code = NotFound desc = could not find container \"8705722d28440e3097208f046fbfcdd85cde2371a15e8579786616cb49af6de1\": container with ID starting with 8705722d28440e3097208f046fbfcdd85cde2371a15e8579786616cb49af6de1 not found: ID does not exist"
Oct 30 00:32:25.804173 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:32:25.804141    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6d4f74cbd-njpvm"]
Oct 30 00:32:25.808652 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:32:25.808632    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-6d4f74cbd-njpvm"]
Oct 30 00:32:26.492036 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:32:26.491999    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed" path="/var/lib/kubelet/pods/2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed/volumes"
Oct 30 00:32:42.859180 ip-10-0-128-7 kubenswrapper[5002]: E1030 00:32:42.859130    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbjdq" containerName="extractor"
Oct 30 00:32:48.951609 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:32:48.951548    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-6c69f88f85-88bkg" podUID="5ed36f74-5c9e-408e-a7fb-2364144f054e" containerName="registry" containerID="cri-o://0d54e2296511e2f2f11e01e551bf9b779b9137f44e51c2662257bd19446606fe" gracePeriod=30
Oct 30 00:32:49.199047 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:32:49.199023    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6c69f88f85-88bkg"
Oct 30 00:32:49.277159 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:32:49.277079    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ed36f74-5c9e-408e-a7fb-2364144f054e-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5ed36f74-5c9e-408e-a7fb-2364144f054e-registry-certificates\") pod \"5ed36f74-5c9e-408e-a7fb-2364144f054e\" (UID: \"5ed36f74-5c9e-408e-a7fb-2364144f054e\") "
Oct 30 00:32:49.277159 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:32:49.277114    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ed36f74-5c9e-408e-a7fb-2364144f054e-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5ed36f74-5c9e-408e-a7fb-2364144f054e-bound-sa-token\") pod \"5ed36f74-5c9e-408e-a7fb-2364144f054e\" (UID: \"5ed36f74-5c9e-408e-a7fb-2364144f054e\") "
Oct 30 00:32:49.277159 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:32:49.277154    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5ed36f74-5c9e-408e-a7fb-2364144f054e-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5ed36f74-5c9e-408e-a7fb-2364144f054e-installation-pull-secrets\") pod \"5ed36f74-5c9e-408e-a7fb-2364144f054e\" (UID: \"5ed36f74-5c9e-408e-a7fb-2364144f054e\") "
Oct 30 00:32:49.277456 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:32:49.277179    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ed36f74-5c9e-408e-a7fb-2364144f054e-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5ed36f74-5c9e-408e-a7fb-2364144f054e-ca-trust-extracted\") pod \"5ed36f74-5c9e-408e-a7fb-2364144f054e\" (UID: \"5ed36f74-5c9e-408e-a7fb-2364144f054e\") "
Oct 30 00:32:49.277456 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:32:49.277198    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ed36f74-5c9e-408e-a7fb-2364144f054e-kube-api-access-7l585\" (UniqueName: \"kubernetes.io/projected/5ed36f74-5c9e-408e-a7fb-2364144f054e-kube-api-access-7l585\") pod \"5ed36f74-5c9e-408e-a7fb-2364144f054e\" (UID: \"5ed36f74-5c9e-408e-a7fb-2364144f054e\") "
Oct 30 00:32:49.277456 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:32:49.277225    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ed36f74-5c9e-408e-a7fb-2364144f054e-registry-tls\" (UniqueName: \"kubernetes.io/projected/5ed36f74-5c9e-408e-a7fb-2364144f054e-registry-tls\") pod \"5ed36f74-5c9e-408e-a7fb-2364144f054e\" (UID: \"5ed36f74-5c9e-408e-a7fb-2364144f054e\") "
Oct 30 00:32:49.277456 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:32:49.277260    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ed36f74-5c9e-408e-a7fb-2364144f054e-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5ed36f74-5c9e-408e-a7fb-2364144f054e-trusted-ca\") pod \"5ed36f74-5c9e-408e-a7fb-2364144f054e\" (UID: \"5ed36f74-5c9e-408e-a7fb-2364144f054e\") "
Oct 30 00:32:49.277456 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:32:49.277294    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5ed36f74-5c9e-408e-a7fb-2364144f054e-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5ed36f74-5c9e-408e-a7fb-2364144f054e-image-registry-private-configuration\") pod \"5ed36f74-5c9e-408e-a7fb-2364144f054e\" (UID: \"5ed36f74-5c9e-408e-a7fb-2364144f054e\") "
Oct 30 00:32:49.277701 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:32:49.277526    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ed36f74-5c9e-408e-a7fb-2364144f054e-registry-certificates" pod "5ed36f74-5c9e-408e-a7fb-2364144f054e" (UID: "5ed36f74-5c9e-408e-a7fb-2364144f054e"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:32:49.278210 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:32:49.278175    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ed36f74-5c9e-408e-a7fb-2364144f054e-trusted-ca" pod "5ed36f74-5c9e-408e-a7fb-2364144f054e" (UID: "5ed36f74-5c9e-408e-a7fb-2364144f054e"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:32:49.280606 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:32:49.280565    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ed36f74-5c9e-408e-a7fb-2364144f054e-kube-api-access-7l585" pod "5ed36f74-5c9e-408e-a7fb-2364144f054e" (UID: "5ed36f74-5c9e-408e-a7fb-2364144f054e"). InnerVolumeSpecName "kube-api-access-7l585". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:32:49.280606 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:32:49.280573    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ed36f74-5c9e-408e-a7fb-2364144f054e-image-registry-private-configuration" pod "5ed36f74-5c9e-408e-a7fb-2364144f054e" (UID: "5ed36f74-5c9e-408e-a7fb-2364144f054e"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:32:49.280778 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:32:49.280625    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ed36f74-5c9e-408e-a7fb-2364144f054e-bound-sa-token" pod "5ed36f74-5c9e-408e-a7fb-2364144f054e" (UID: "5ed36f74-5c9e-408e-a7fb-2364144f054e"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:32:49.280778 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:32:49.280673    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ed36f74-5c9e-408e-a7fb-2364144f054e-installation-pull-secrets" pod "5ed36f74-5c9e-408e-a7fb-2364144f054e" (UID: "5ed36f74-5c9e-408e-a7fb-2364144f054e"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:32:49.281924 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:32:49.281898    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ed36f74-5c9e-408e-a7fb-2364144f054e-registry-tls" pod "5ed36f74-5c9e-408e-a7fb-2364144f054e" (UID: "5ed36f74-5c9e-408e-a7fb-2364144f054e"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:32:49.286902 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:32:49.286872    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ed36f74-5c9e-408e-a7fb-2364144f054e-ca-trust-extracted" pod "5ed36f74-5c9e-408e-a7fb-2364144f054e" (UID: "5ed36f74-5c9e-408e-a7fb-2364144f054e"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:32:49.378010 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:32:49.377970    5002 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5ed36f74-5c9e-408e-a7fb-2364144f054e-registry-certificates\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:32:49.378010 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:32:49.378008    5002 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5ed36f74-5c9e-408e-a7fb-2364144f054e-bound-sa-token\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:32:49.378010 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:32:49.378018    5002 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5ed36f74-5c9e-408e-a7fb-2364144f054e-installation-pull-secrets\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:32:49.378010 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:32:49.378028    5002 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5ed36f74-5c9e-408e-a7fb-2364144f054e-ca-trust-extracted\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:32:49.378271 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:32:49.378037    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7l585\" (UniqueName: \"kubernetes.io/projected/5ed36f74-5c9e-408e-a7fb-2364144f054e-kube-api-access-7l585\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:32:49.378271 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:32:49.378046    5002 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5ed36f74-5c9e-408e-a7fb-2364144f054e-registry-tls\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:32:49.378271 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:32:49.378055    5002 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5ed36f74-5c9e-408e-a7fb-2364144f054e-trusted-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:32:49.378271 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:32:49.378063    5002 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5ed36f74-5c9e-408e-a7fb-2364144f054e-image-registry-private-configuration\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:32:49.841245 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:32:49.841215    5002 generic.go:356] "Generic (PLEG): container finished" podID="b874991e-9562-4848-b8f1-4b5a2b7cddc4" containerID="6480afc5a4cd6b87539908cc8f161f96c3724a81bfd3e829a511c95f9e005bf8" exitCode=0
Oct 30 00:32:49.841444 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:32:49.841296    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-sxknf" event={"ID":"b874991e-9562-4848-b8f1-4b5a2b7cddc4","Type":"ContainerDied","Data":"6480afc5a4cd6b87539908cc8f161f96c3724a81bfd3e829a511c95f9e005bf8"}
Oct 30 00:32:49.841642 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:32:49.841627    5002 scope.go:122] "RemoveContainer" containerID="6480afc5a4cd6b87539908cc8f161f96c3724a81bfd3e829a511c95f9e005bf8"
Oct 30 00:32:49.842672 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:32:49.842642    5002 generic.go:356] "Generic (PLEG): container finished" podID="5ed36f74-5c9e-408e-a7fb-2364144f054e" containerID="0d54e2296511e2f2f11e01e551bf9b779b9137f44e51c2662257bd19446606fe" exitCode=0
Oct 30 00:32:49.842775 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:32:49.842692    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6c69f88f85-88bkg"
Oct 30 00:32:49.842775 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:32:49.842695    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6c69f88f85-88bkg" event={"ID":"5ed36f74-5c9e-408e-a7fb-2364144f054e","Type":"ContainerDied","Data":"0d54e2296511e2f2f11e01e551bf9b779b9137f44e51c2662257bd19446606fe"}
Oct 30 00:32:49.842775 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:32:49.842742    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6c69f88f85-88bkg" event={"ID":"5ed36f74-5c9e-408e-a7fb-2364144f054e","Type":"ContainerDied","Data":"910db8ae6b6a9af3f1879820859bd9e54ee5da41548aba5d0414e4cdab3c4b65"}
Oct 30 00:32:49.842775 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:32:49.842763    5002 scope.go:122] "RemoveContainer" containerID="0d54e2296511e2f2f11e01e551bf9b779b9137f44e51c2662257bd19446606fe"
Oct 30 00:32:49.853562 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:32:49.853539    5002 scope.go:122] "RemoveContainer" containerID="0d54e2296511e2f2f11e01e551bf9b779b9137f44e51c2662257bd19446606fe"
Oct 30 00:32:49.853810 ip-10-0-128-7 kubenswrapper[5002]: E1030 00:32:49.853792    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d54e2296511e2f2f11e01e551bf9b779b9137f44e51c2662257bd19446606fe\": container with ID starting with 0d54e2296511e2f2f11e01e551bf9b779b9137f44e51c2662257bd19446606fe not found: ID does not exist" containerID="0d54e2296511e2f2f11e01e551bf9b779b9137f44e51c2662257bd19446606fe"
Oct 30 00:32:49.853865 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:32:49.853818    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d54e2296511e2f2f11e01e551bf9b779b9137f44e51c2662257bd19446606fe"} err="failed to get container status \"0d54e2296511e2f2f11e01e551bf9b779b9137f44e51c2662257bd19446606fe\": rpc error: code = NotFound desc = could not find container \"0d54e2296511e2f2f11e01e551bf9b779b9137f44e51c2662257bd19446606fe\": container with ID starting with 0d54e2296511e2f2f11e01e551bf9b779b9137f44e51c2662257bd19446606fe not found: ID does not exist"
Oct 30 00:32:49.927926 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:32:49.927894    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6c69f88f85-88bkg"]
Oct 30 00:32:49.940910 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:32:49.940880    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-6c69f88f85-88bkg"]
Oct 30 00:32:50.492088 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:32:50.492055    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ed36f74-5c9e-408e-a7fb-2364144f054e" path="/var/lib/kubelet/pods/5ed36f74-5c9e-408e-a7fb-2364144f054e/volumes"
Oct 30 00:32:50.848526 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:32:50.848427    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-sxknf" event={"ID":"b874991e-9562-4848-b8f1-4b5a2b7cddc4","Type":"ContainerStarted","Data":"d0722508d262e5cb4d98933f4aacdddb686dcb7f6371dee034db7b97bad59533"}
Oct 30 00:32:59.883165 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:32:59.883129    5002 generic.go:356] "Generic (PLEG): container finished" podID="61903cf2-2b92-4f40-ab52-25509de587ef" containerID="e60031b407f1559d4973b6c9c56cd518f0aa584a7f73932563c0129f202e36f4" exitCode=0
Oct 30 00:32:59.883572 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:32:59.883171    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-bjzgs" event={"ID":"61903cf2-2b92-4f40-ab52-25509de587ef","Type":"ContainerDied","Data":"e60031b407f1559d4973b6c9c56cd518f0aa584a7f73932563c0129f202e36f4"}
Oct 30 00:32:59.883572 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:32:59.883488    5002 scope.go:122] "RemoveContainer" containerID="e60031b407f1559d4973b6c9c56cd518f0aa584a7f73932563c0129f202e36f4"
Oct 30 00:33:00.888309 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:33:00.888275    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-bjzgs" event={"ID":"61903cf2-2b92-4f40-ab52-25509de587ef","Type":"ContainerStarted","Data":"78679d013740d90bccf98dea2cbc83f6c72701f9c09430b2f0e9134cebdf46bb"}
Oct 30 00:33:04.905872 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:33:04.905836    5002 generic.go:356] "Generic (PLEG): container finished" podID="e4ca0ea9-b57d-4d4c-be3a-2da5fc26a65d" containerID="696239c269899c960abd90dacba42f6bd67abbd33c8406dc29a4900955093ae4" exitCode=0
Oct 30 00:33:04.906366 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:33:04.905912    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-sklth" event={"ID":"e4ca0ea9-b57d-4d4c-be3a-2da5fc26a65d","Type":"ContainerDied","Data":"696239c269899c960abd90dacba42f6bd67abbd33c8406dc29a4900955093ae4"}
Oct 30 00:33:04.906366 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:33:04.906301    5002 scope.go:122] "RemoveContainer" containerID="696239c269899c960abd90dacba42f6bd67abbd33c8406dc29a4900955093ae4"
Oct 30 00:33:05.388759 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:33:05.388722    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-zzlf6-kvglg"]
Oct 30 00:33:05.388953 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:33:05.388933    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-insights/periodic-gathering-zzlf6-kvglg" podUID="bba787ce-4e61-4a5a-a728-af73c68dfdd3" containerName="insights-gathering" containerID="cri-o://c46ed1ec8ce6982a6892e8f614588ea94fa7e225818413fd0196bced66dec9f3" gracePeriod=30
Oct 30 00:33:05.569563 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:33:05.569533    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-v82fj-f6t49"]
Oct 30 00:33:05.569890 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:33:05.569874    5002 cpu_manager.go:414] "containerMap: removing container" podUID="5ed36f74-5c9e-408e-a7fb-2364144f054e" containerName="registry"
Oct 30 00:33:05.569965 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:33:05.569893    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ed36f74-5c9e-408e-a7fb-2364144f054e" containerName="registry"
Oct 30 00:33:05.569965 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:33:05.569927    5002 cpu_manager.go:414] "containerMap: removing container" podUID="2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed" containerName="registry"
Oct 30 00:33:05.569965 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:33:05.569936    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed" containerName="registry"
Oct 30 00:33:05.570109 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:33:05.570024    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ed36f74-5c9e-408e-a7fb-2364144f054e" containerName="registry"
Oct 30 00:33:05.570109 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:33:05.570042    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a1658ed-d4f5-4a0a-9ee9-df362b2a13ed" containerName="registry"
Oct 30 00:33:05.572859 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:33:05.572837    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-v82fj-f6t49"
Oct 30 00:33:05.592537 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:33:05.592508    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-v82fj-f6t49"]
Oct 30 00:33:05.597937 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:33:05.597911    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/18026bd1-33b6-48cd-bffc-7d087abf2640-service-ca-bundle\") pod \"periodic-gathering-v82fj-f6t49\" (UID: \"18026bd1-33b6-48cd-bffc-7d087abf2640\") " pod="openshift-insights/periodic-gathering-v82fj-f6t49"
Oct 30 00:33:05.598068 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:33:05.597957    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/18026bd1-33b6-48cd-bffc-7d087abf2640-trusted-ca-bundle\") pod \"periodic-gathering-v82fj-f6t49\" (UID: \"18026bd1-33b6-48cd-bffc-7d087abf2640\") " pod="openshift-insights/periodic-gathering-v82fj-f6t49"
Oct 30 00:33:05.598068 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:33:05.597982    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4j8fc\" (UniqueName: \"kubernetes.io/projected/18026bd1-33b6-48cd-bffc-7d087abf2640-kube-api-access-4j8fc\") pod \"periodic-gathering-v82fj-f6t49\" (UID: \"18026bd1-33b6-48cd-bffc-7d087abf2640\") " pod="openshift-insights/periodic-gathering-v82fj-f6t49"
Oct 30 00:33:05.598155 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:33:05.598077    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/18026bd1-33b6-48cd-bffc-7d087abf2640-archives-path\") pod \"periodic-gathering-v82fj-f6t49\" (UID: \"18026bd1-33b6-48cd-bffc-7d087abf2640\") " pod="openshift-insights/periodic-gathering-v82fj-f6t49"
Oct 30 00:33:05.645414 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:33:05.645327    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-zzlf6-kvglg"
Oct 30 00:33:05.699087 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:33:05.699024    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bba787ce-4e61-4a5a-a728-af73c68dfdd3-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bba787ce-4e61-4a5a-a728-af73c68dfdd3-service-ca-bundle\") pod \"bba787ce-4e61-4a5a-a728-af73c68dfdd3\" (UID: \"bba787ce-4e61-4a5a-a728-af73c68dfdd3\") "
Oct 30 00:33:05.699266 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:33:05.699117    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bba787ce-4e61-4a5a-a728-af73c68dfdd3-kube-api-access-fzc86\" (UniqueName: \"kubernetes.io/projected/bba787ce-4e61-4a5a-a728-af73c68dfdd3-kube-api-access-fzc86\") pod \"bba787ce-4e61-4a5a-a728-af73c68dfdd3\" (UID: \"bba787ce-4e61-4a5a-a728-af73c68dfdd3\") "
Oct 30 00:33:05.699266 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:33:05.699139    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bba787ce-4e61-4a5a-a728-af73c68dfdd3-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bba787ce-4e61-4a5a-a728-af73c68dfdd3-trusted-ca-bundle\") pod \"bba787ce-4e61-4a5a-a728-af73c68dfdd3\" (UID: \"bba787ce-4e61-4a5a-a728-af73c68dfdd3\") "
Oct 30 00:33:05.699266 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:33:05.699187    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bba787ce-4e61-4a5a-a728-af73c68dfdd3-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/bba787ce-4e61-4a5a-a728-af73c68dfdd3-archives-path\") pod \"bba787ce-4e61-4a5a-a728-af73c68dfdd3\" (UID: \"bba787ce-4e61-4a5a-a728-af73c68dfdd3\") "
Oct 30 00:33:05.699665 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:33:05.699321    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/18026bd1-33b6-48cd-bffc-7d087abf2640-trusted-ca-bundle\") pod \"periodic-gathering-v82fj-f6t49\" (UID: \"18026bd1-33b6-48cd-bffc-7d087abf2640\") " pod="openshift-insights/periodic-gathering-v82fj-f6t49"
Oct 30 00:33:05.699665 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:33:05.699410    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4j8fc\" (UniqueName: \"kubernetes.io/projected/18026bd1-33b6-48cd-bffc-7d087abf2640-kube-api-access-4j8fc\") pod \"periodic-gathering-v82fj-f6t49\" (UID: \"18026bd1-33b6-48cd-bffc-7d087abf2640\") " pod="openshift-insights/periodic-gathering-v82fj-f6t49"
Oct 30 00:33:05.699665 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:33:05.699453    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/18026bd1-33b6-48cd-bffc-7d087abf2640-archives-path\") pod \"periodic-gathering-v82fj-f6t49\" (UID: \"18026bd1-33b6-48cd-bffc-7d087abf2640\") " pod="openshift-insights/periodic-gathering-v82fj-f6t49"
Oct 30 00:33:05.699665 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:33:05.699493    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bba787ce-4e61-4a5a-a728-af73c68dfdd3-service-ca-bundle" pod "bba787ce-4e61-4a5a-a728-af73c68dfdd3" (UID: "bba787ce-4e61-4a5a-a728-af73c68dfdd3"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:33:05.699665 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:33:05.699554    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/18026bd1-33b6-48cd-bffc-7d087abf2640-service-ca-bundle\") pod \"periodic-gathering-v82fj-f6t49\" (UID: \"18026bd1-33b6-48cd-bffc-7d087abf2640\") " pod="openshift-insights/periodic-gathering-v82fj-f6t49"
Oct 30 00:33:05.699665 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:33:05.699601    5002 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bba787ce-4e61-4a5a-a728-af73c68dfdd3-service-ca-bundle\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:33:05.699665 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:33:05.699604    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bba787ce-4e61-4a5a-a728-af73c68dfdd3-archives-path" pod "bba787ce-4e61-4a5a-a728-af73c68dfdd3" (UID: "bba787ce-4e61-4a5a-a728-af73c68dfdd3"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:33:05.699665 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:33:05.699607    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bba787ce-4e61-4a5a-a728-af73c68dfdd3-trusted-ca-bundle" pod "bba787ce-4e61-4a5a-a728-af73c68dfdd3" (UID: "bba787ce-4e61-4a5a-a728-af73c68dfdd3"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:33:05.700035 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:33:05.699821    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/18026bd1-33b6-48cd-bffc-7d087abf2640-archives-path\") pod \"periodic-gathering-v82fj-f6t49\" (UID: \"18026bd1-33b6-48cd-bffc-7d087abf2640\") " pod="openshift-insights/periodic-gathering-v82fj-f6t49"
Oct 30 00:33:05.700129 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:33:05.700111    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/18026bd1-33b6-48cd-bffc-7d087abf2640-service-ca-bundle\") pod \"periodic-gathering-v82fj-f6t49\" (UID: \"18026bd1-33b6-48cd-bffc-7d087abf2640\") " pod="openshift-insights/periodic-gathering-v82fj-f6t49"
Oct 30 00:33:05.700291 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:33:05.700274    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/18026bd1-33b6-48cd-bffc-7d087abf2640-trusted-ca-bundle\") pod \"periodic-gathering-v82fj-f6t49\" (UID: \"18026bd1-33b6-48cd-bffc-7d087abf2640\") " pod="openshift-insights/periodic-gathering-v82fj-f6t49"
Oct 30 00:33:05.702052 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:33:05.702034    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bba787ce-4e61-4a5a-a728-af73c68dfdd3-kube-api-access-fzc86" pod "bba787ce-4e61-4a5a-a728-af73c68dfdd3" (UID: "bba787ce-4e61-4a5a-a728-af73c68dfdd3"). InnerVolumeSpecName "kube-api-access-fzc86". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:33:05.724632 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:33:05.724604    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4j8fc\" (UniqueName: \"kubernetes.io/projected/18026bd1-33b6-48cd-bffc-7d087abf2640-kube-api-access-4j8fc\") pod \"periodic-gathering-v82fj-f6t49\" (UID: \"18026bd1-33b6-48cd-bffc-7d087abf2640\") " pod="openshift-insights/periodic-gathering-v82fj-f6t49"
Oct 30 00:33:05.800787 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:33:05.800741    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fzc86\" (UniqueName: \"kubernetes.io/projected/bba787ce-4e61-4a5a-a728-af73c68dfdd3-kube-api-access-fzc86\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:33:05.800787 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:33:05.800783    5002 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bba787ce-4e61-4a5a-a728-af73c68dfdd3-trusted-ca-bundle\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:33:05.800787 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:33:05.800793    5002 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/bba787ce-4e61-4a5a-a728-af73c68dfdd3-archives-path\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:33:05.891066 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:33:05.891036    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-v82fj-f6t49"
Oct 30 00:33:05.911041 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:33:05.911012    5002 generic.go:356] "Generic (PLEG): container finished" podID="bba787ce-4e61-4a5a-a728-af73c68dfdd3" containerID="c46ed1ec8ce6982a6892e8f614588ea94fa7e225818413fd0196bced66dec9f3" exitCode=2
Oct 30 00:33:05.911420 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:33:05.911081    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-zzlf6-kvglg"
Oct 30 00:33:05.911420 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:33:05.911091    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-zzlf6-kvglg" event={"ID":"bba787ce-4e61-4a5a-a728-af73c68dfdd3","Type":"ContainerDied","Data":"c46ed1ec8ce6982a6892e8f614588ea94fa7e225818413fd0196bced66dec9f3"}
Oct 30 00:33:05.911420 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:33:05.911145    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-zzlf6-kvglg" event={"ID":"bba787ce-4e61-4a5a-a728-af73c68dfdd3","Type":"ContainerDied","Data":"b984324d5c7f9d7f10d6b87625aafb838e43218eaa1db56a88123a6411f19142"}
Oct 30 00:33:05.911420 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:33:05.911173    5002 scope.go:122] "RemoveContainer" containerID="c46ed1ec8ce6982a6892e8f614588ea94fa7e225818413fd0196bced66dec9f3"
Oct 30 00:33:05.913041 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:33:05.913014    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-sklth" event={"ID":"e4ca0ea9-b57d-4d4c-be3a-2da5fc26a65d","Type":"ContainerStarted","Data":"bb518b9c60460c2a3e78d8bb5e2baddc4db216f283ec546e4dd4de5b49586914"}
Oct 30 00:33:05.920185 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:33:05.920166    5002 scope.go:122] "RemoveContainer" containerID="c46ed1ec8ce6982a6892e8f614588ea94fa7e225818413fd0196bced66dec9f3"
Oct 30 00:33:05.920549 ip-10-0-128-7 kubenswrapper[5002]: E1030 00:33:05.920525    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c46ed1ec8ce6982a6892e8f614588ea94fa7e225818413fd0196bced66dec9f3\": container with ID starting with c46ed1ec8ce6982a6892e8f614588ea94fa7e225818413fd0196bced66dec9f3 not found: ID does not exist" containerID="c46ed1ec8ce6982a6892e8f614588ea94fa7e225818413fd0196bced66dec9f3"
Oct 30 00:33:05.920642 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:33:05.920564    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c46ed1ec8ce6982a6892e8f614588ea94fa7e225818413fd0196bced66dec9f3"} err="failed to get container status \"c46ed1ec8ce6982a6892e8f614588ea94fa7e225818413fd0196bced66dec9f3\": rpc error: code = NotFound desc = could not find container \"c46ed1ec8ce6982a6892e8f614588ea94fa7e225818413fd0196bced66dec9f3\": container with ID starting with c46ed1ec8ce6982a6892e8f614588ea94fa7e225818413fd0196bced66dec9f3 not found: ID does not exist"
Oct 30 00:33:05.967123 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:33:05.967094    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-zzlf6-kvglg"]
Oct 30 00:33:05.973257 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:33:05.973231    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-insights/periodic-gathering-zzlf6-kvglg"]
Oct 30 00:33:06.186642 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:33:06.186610    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-v82fj-f6t49"]
Oct 30 00:33:06.187638 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:33:06.187613    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18026bd1_33b6_48cd_bffc_7d087abf2640.slice/crio-8e610c202fb8a5c234061a3b905cafd2d414e98346a85d5ae98381468d6608f5 WatchSource:0}: Error finding container 8e610c202fb8a5c234061a3b905cafd2d414e98346a85d5ae98381468d6608f5: Status 404 returned error can't find the container with id 8e610c202fb8a5c234061a3b905cafd2d414e98346a85d5ae98381468d6608f5
Oct 30 00:33:06.491865 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:33:06.491794    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bba787ce-4e61-4a5a-a728-af73c68dfdd3" path="/var/lib/kubelet/pods/bba787ce-4e61-4a5a-a728-af73c68dfdd3/volumes"
Oct 30 00:33:06.918615 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:33:06.918527    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-v82fj-f6t49" event={"ID":"18026bd1-33b6-48cd-bffc-7d087abf2640","Type":"ContainerStarted","Data":"1efac999fa3c679c3ed9b58ffecc714f20ab38a0b0b81aa7490d9716017f06bf"}
Oct 30 00:33:06.918615 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:33:06.918592    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-v82fj-f6t49" event={"ID":"18026bd1-33b6-48cd-bffc-7d087abf2640","Type":"ContainerStarted","Data":"8e610c202fb8a5c234061a3b905cafd2d414e98346a85d5ae98381468d6608f5"}
Oct 30 00:33:06.962793 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:33:06.962741    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-v82fj-f6t49" podStartSLOduration=1.962729004 podStartE2EDuration="1.962729004s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:33:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:33:06.962186379 +0000 UTC m=+82.578619258" watchObservedRunningTime="2026-10-30 00:33:06.962729004 +0000 UTC m=+82.579161862"
Oct 30 00:33:12.859426 ip-10-0-128-7 kubenswrapper[5002]: E1030 00:33:12.859373    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbjdq" containerName="extractor"
Oct 30 00:33:42.860315 ip-10-0-128-7 kubenswrapper[5002]: E1030 00:33:42.860270    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbjdq" containerName="extractor"
Oct 30 00:34:12.859687 ip-10-0-128-7 kubenswrapper[5002]: E1030 00:34:12.859591    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbjdq" containerName="extractor"
Oct 30 00:34:32.239626 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:34:32.239592    5002 generic.go:356] "Generic (PLEG): container finished" podID="18026bd1-33b6-48cd-bffc-7d087abf2640" containerID="1efac999fa3c679c3ed9b58ffecc714f20ab38a0b0b81aa7490d9716017f06bf" exitCode=0
Oct 30 00:34:32.240038 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:34:32.239662    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-v82fj-f6t49" event={"ID":"18026bd1-33b6-48cd-bffc-7d087abf2640","Type":"ContainerDied","Data":"1efac999fa3c679c3ed9b58ffecc714f20ab38a0b0b81aa7490d9716017f06bf"}
Oct 30 00:34:33.365618 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:34:33.365592    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-v82fj-f6t49"
Oct 30 00:34:33.446884 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:34:33.446848    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18026bd1-33b6-48cd-bffc-7d087abf2640-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/18026bd1-33b6-48cd-bffc-7d087abf2640-trusted-ca-bundle\") pod \"18026bd1-33b6-48cd-bffc-7d087abf2640\" (UID: \"18026bd1-33b6-48cd-bffc-7d087abf2640\") "
Oct 30 00:34:33.447058 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:34:33.446899    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18026bd1-33b6-48cd-bffc-7d087abf2640-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/18026bd1-33b6-48cd-bffc-7d087abf2640-archives-path\") pod \"18026bd1-33b6-48cd-bffc-7d087abf2640\" (UID: \"18026bd1-33b6-48cd-bffc-7d087abf2640\") "
Oct 30 00:34:33.447058 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:34:33.446925    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18026bd1-33b6-48cd-bffc-7d087abf2640-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/18026bd1-33b6-48cd-bffc-7d087abf2640-service-ca-bundle\") pod \"18026bd1-33b6-48cd-bffc-7d087abf2640\" (UID: \"18026bd1-33b6-48cd-bffc-7d087abf2640\") "
Oct 30 00:34:33.447058 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:34:33.446951    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18026bd1-33b6-48cd-bffc-7d087abf2640-kube-api-access-4j8fc\" (UniqueName: \"kubernetes.io/projected/18026bd1-33b6-48cd-bffc-7d087abf2640-kube-api-access-4j8fc\") pod \"18026bd1-33b6-48cd-bffc-7d087abf2640\" (UID: \"18026bd1-33b6-48cd-bffc-7d087abf2640\") "
Oct 30 00:34:33.447319 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:34:33.447292    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18026bd1-33b6-48cd-bffc-7d087abf2640-archives-path" pod "18026bd1-33b6-48cd-bffc-7d087abf2640" (UID: "18026bd1-33b6-48cd-bffc-7d087abf2640"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:34:33.447412 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:34:33.447315    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18026bd1-33b6-48cd-bffc-7d087abf2640-trusted-ca-bundle" pod "18026bd1-33b6-48cd-bffc-7d087abf2640" (UID: "18026bd1-33b6-48cd-bffc-7d087abf2640"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:34:33.447412 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:34:33.447321    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18026bd1-33b6-48cd-bffc-7d087abf2640-service-ca-bundle" pod "18026bd1-33b6-48cd-bffc-7d087abf2640" (UID: "18026bd1-33b6-48cd-bffc-7d087abf2640"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:34:33.449044 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:34:33.449019    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18026bd1-33b6-48cd-bffc-7d087abf2640-kube-api-access-4j8fc" pod "18026bd1-33b6-48cd-bffc-7d087abf2640" (UID: "18026bd1-33b6-48cd-bffc-7d087abf2640"). InnerVolumeSpecName "kube-api-access-4j8fc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:34:33.548365 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:34:33.548288    5002 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/18026bd1-33b6-48cd-bffc-7d087abf2640-trusted-ca-bundle\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:34:33.548365 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:34:33.548316    5002 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/18026bd1-33b6-48cd-bffc-7d087abf2640-archives-path\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:34:33.548365 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:34:33.548326    5002 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/18026bd1-33b6-48cd-bffc-7d087abf2640-service-ca-bundle\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:34:33.548365 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:34:33.548336    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4j8fc\" (UniqueName: \"kubernetes.io/projected/18026bd1-33b6-48cd-bffc-7d087abf2640-kube-api-access-4j8fc\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:34:34.252934 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:34:34.252902    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-v82fj-f6t49"
Oct 30 00:34:34.253120 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:34:34.252900    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-v82fj-f6t49" event={"ID":"18026bd1-33b6-48cd-bffc-7d087abf2640","Type":"ContainerDied","Data":"8e610c202fb8a5c234061a3b905cafd2d414e98346a85d5ae98381468d6608f5"}
Oct 30 00:34:34.253120 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:34:34.253048    5002 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8e610c202fb8a5c234061a3b905cafd2d414e98346a85d5ae98381468d6608f5"
Oct 30 00:34:42.860281 ip-10-0-128-7 kubenswrapper[5002]: E1030 00:34:42.860208    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbjdq" containerName="extractor"
Oct 30 00:35:12.859596 ip-10-0-128-7 kubenswrapper[5002]: E1030 00:35:12.859557    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbjdq" containerName="extractor"
Oct 30 00:35:24.783291 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:35:24.783256    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-597b95cb84-7fzkl"]
Oct 30 00:35:24.783904 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:35:24.783704    5002 cpu_manager.go:414] "containerMap: removing container" podUID="bba787ce-4e61-4a5a-a728-af73c68dfdd3" containerName="insights-gathering"
Oct 30 00:35:24.783904 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:35:24.783724    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bba787ce-4e61-4a5a-a728-af73c68dfdd3" containerName="insights-gathering"
Oct 30 00:35:24.783904 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:35:24.783744    5002 cpu_manager.go:414] "containerMap: removing container" podUID="18026bd1-33b6-48cd-bffc-7d087abf2640" containerName="insights-gathering"
Oct 30 00:35:24.783904 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:35:24.783752    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18026bd1-33b6-48cd-bffc-7d087abf2640" containerName="insights-gathering"
Oct 30 00:35:24.783904 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:35:24.783852    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="bba787ce-4e61-4a5a-a728-af73c68dfdd3" containerName="insights-gathering"
Oct 30 00:35:24.783904 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:35:24.783864    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="18026bd1-33b6-48cd-bffc-7d087abf2640" containerName="insights-gathering"
Oct 30 00:35:24.786830 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:35:24.786810    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-7fzkl"
Oct 30 00:35:24.805270 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:35:24.805247    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-7ngfc\""
Oct 30 00:35:24.811067 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:35:24.807499    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-7fzkl"]
Oct 30 00:35:24.819636 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:35:24.819616    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Oct 30 00:35:24.819636 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:35:24.819620    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Oct 30 00:35:24.938066 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:35:24.938032    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-298ds\" (UniqueName: \"kubernetes.io/projected/07961f00-bf56-4948-b7ea-5ade3b6921b2-kube-api-access-298ds\") pod \"cert-manager-597b95cb84-7fzkl\" (UID: \"07961f00-bf56-4948-b7ea-5ade3b6921b2\") " pod="cert-manager/cert-manager-597b95cb84-7fzkl"
Oct 30 00:35:24.938225 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:35:24.938097    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/07961f00-bf56-4948-b7ea-5ade3b6921b2-bound-sa-token\") pod \"cert-manager-597b95cb84-7fzkl\" (UID: \"07961f00-bf56-4948-b7ea-5ade3b6921b2\") " pod="cert-manager/cert-manager-597b95cb84-7fzkl"
Oct 30 00:35:25.038956 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:35:25.038867    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/07961f00-bf56-4948-b7ea-5ade3b6921b2-bound-sa-token\") pod \"cert-manager-597b95cb84-7fzkl\" (UID: \"07961f00-bf56-4948-b7ea-5ade3b6921b2\") " pod="cert-manager/cert-manager-597b95cb84-7fzkl"
Oct 30 00:35:25.038956 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:35:25.038919    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-298ds\" (UniqueName: \"kubernetes.io/projected/07961f00-bf56-4948-b7ea-5ade3b6921b2-kube-api-access-298ds\") pod \"cert-manager-597b95cb84-7fzkl\" (UID: \"07961f00-bf56-4948-b7ea-5ade3b6921b2\") " pod="cert-manager/cert-manager-597b95cb84-7fzkl"
Oct 30 00:35:25.064247 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:35:25.064205    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/07961f00-bf56-4948-b7ea-5ade3b6921b2-bound-sa-token\") pod \"cert-manager-597b95cb84-7fzkl\" (UID: \"07961f00-bf56-4948-b7ea-5ade3b6921b2\") " pod="cert-manager/cert-manager-597b95cb84-7fzkl"
Oct 30 00:35:25.064247 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:35:25.064225    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-298ds\" (UniqueName: \"kubernetes.io/projected/07961f00-bf56-4948-b7ea-5ade3b6921b2-kube-api-access-298ds\") pod \"cert-manager-597b95cb84-7fzkl\" (UID: \"07961f00-bf56-4948-b7ea-5ade3b6921b2\") " pod="cert-manager/cert-manager-597b95cb84-7fzkl"
Oct 30 00:35:25.114755 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:35:25.114715    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-7fzkl"
Oct 30 00:35:25.409183 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:35:25.409160    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-7fzkl"]
Oct 30 00:35:25.410398 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:35:25.410360    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod07961f00_bf56_4948_b7ea_5ade3b6921b2.slice/crio-60f640f4b6e7105c89f04c2f682be7e71fcc4ca97af896f1c2212cef7e99c388 WatchSource:0}: Error finding container 60f640f4b6e7105c89f04c2f682be7e71fcc4ca97af896f1c2212cef7e99c388: Status 404 returned error can't find the container with id 60f640f4b6e7105c89f04c2f682be7e71fcc4ca97af896f1c2212cef7e99c388
Oct 30 00:35:25.449606 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:35:25.449571    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-7fzkl" event={"ID":"07961f00-bf56-4948-b7ea-5ade3b6921b2","Type":"ContainerStarted","Data":"60f640f4b6e7105c89f04c2f682be7e71fcc4ca97af896f1c2212cef7e99c388"}
Oct 30 00:35:29.468340 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:35:29.468299    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-7fzkl" event={"ID":"07961f00-bf56-4948-b7ea-5ade3b6921b2","Type":"ContainerStarted","Data":"f2dc58756772d809139ab24b7a50c63c1f80d47e6334f0f0d5247797ef46e4d8"}
Oct 30 00:35:29.512558 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:35:29.512506    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-597b95cb84-7fzkl" podStartSLOduration=2.346766149 podStartE2EDuration="5.512493576s" totalImagesPullingTime="3.165727427s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:35:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:35:29.512178917 +0000 UTC m=+225.128611796" watchObservedRunningTime="2026-10-30 00:35:29.512493576 +0000 UTC m=+225.128926449"
Oct 30 00:35:42.860035 ip-10-0-128-7 kubenswrapper[5002]: E1030 00:35:42.859998    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbjdq" containerName="extractor"
Oct 30 00:35:48.691597 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:35:48.691561    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6bcdb85766-9j9s2"]
Oct 30 00:35:48.695156 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:35:48.695138    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6bcdb85766-9j9s2"
Oct 30 00:35:48.715598 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:35:48.715569    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Oct 30 00:35:48.715598 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:35:48.715589    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Oct 30 00:35:48.739218 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:35:48.739191    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-gqpv5\""
Oct 30 00:35:48.739218 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:35:48.739195    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Oct 30 00:35:48.740985 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:35:48.740965    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-metrics-tls\""
Oct 30 00:35:48.748134 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:35:48.748118    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Oct 30 00:35:48.759218 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:35:48.759186    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6bcdb85766-9j9s2"]
Oct 30 00:35:48.817483 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:35:48.817450    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gh7pq\" (UniqueName: \"kubernetes.io/projected/e7944126-c185-43db-b8ba-04cbefec39fb-kube-api-access-gh7pq\") pod \"opendatahub-operator-controller-manager-6bcdb85766-9j9s2\" (UID: \"e7944126-c185-43db-b8ba-04cbefec39fb\") " pod="opendatahub/opendatahub-operator-controller-manager-6bcdb85766-9j9s2"
Oct 30 00:35:48.817642 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:35:48.817489    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e7944126-c185-43db-b8ba-04cbefec39fb-manifests\") pod \"opendatahub-operator-controller-manager-6bcdb85766-9j9s2\" (UID: \"e7944126-c185-43db-b8ba-04cbefec39fb\") " pod="opendatahub/opendatahub-operator-controller-manager-6bcdb85766-9j9s2"
Oct 30 00:35:48.817642 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:35:48.817510    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e7944126-c185-43db-b8ba-04cbefec39fb-webhook-cert\") pod \"opendatahub-operator-controller-manager-6bcdb85766-9j9s2\" (UID: \"e7944126-c185-43db-b8ba-04cbefec39fb\") " pod="opendatahub/opendatahub-operator-controller-manager-6bcdb85766-9j9s2"
Oct 30 00:35:48.817711 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:35:48.817656    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e7944126-c185-43db-b8ba-04cbefec39fb-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6bcdb85766-9j9s2\" (UID: \"e7944126-c185-43db-b8ba-04cbefec39fb\") " pod="opendatahub/opendatahub-operator-controller-manager-6bcdb85766-9j9s2"
Oct 30 00:35:48.817711 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:35:48.817699    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/e7944126-c185-43db-b8ba-04cbefec39fb-metrics-cert\") pod \"opendatahub-operator-controller-manager-6bcdb85766-9j9s2\" (UID: \"e7944126-c185-43db-b8ba-04cbefec39fb\") " pod="opendatahub/opendatahub-operator-controller-manager-6bcdb85766-9j9s2"
Oct 30 00:35:48.817796 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:35:48.817779    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e7944126-c185-43db-b8ba-04cbefec39fb-tmp\") pod \"opendatahub-operator-controller-manager-6bcdb85766-9j9s2\" (UID: \"e7944126-c185-43db-b8ba-04cbefec39fb\") " pod="opendatahub/opendatahub-operator-controller-manager-6bcdb85766-9j9s2"
Oct 30 00:35:48.918194 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:35:48.918154    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gh7pq\" (UniqueName: \"kubernetes.io/projected/e7944126-c185-43db-b8ba-04cbefec39fb-kube-api-access-gh7pq\") pod \"opendatahub-operator-controller-manager-6bcdb85766-9j9s2\" (UID: \"e7944126-c185-43db-b8ba-04cbefec39fb\") " pod="opendatahub/opendatahub-operator-controller-manager-6bcdb85766-9j9s2"
Oct 30 00:35:48.918194 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:35:48.918200    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e7944126-c185-43db-b8ba-04cbefec39fb-manifests\") pod \"opendatahub-operator-controller-manager-6bcdb85766-9j9s2\" (UID: \"e7944126-c185-43db-b8ba-04cbefec39fb\") " pod="opendatahub/opendatahub-operator-controller-manager-6bcdb85766-9j9s2"
Oct 30 00:35:48.918432 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:35:48.918222    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e7944126-c185-43db-b8ba-04cbefec39fb-webhook-cert\") pod \"opendatahub-operator-controller-manager-6bcdb85766-9j9s2\" (UID: \"e7944126-c185-43db-b8ba-04cbefec39fb\") " pod="opendatahub/opendatahub-operator-controller-manager-6bcdb85766-9j9s2"
Oct 30 00:35:48.918432 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:35:48.918273    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e7944126-c185-43db-b8ba-04cbefec39fb-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6bcdb85766-9j9s2\" (UID: \"e7944126-c185-43db-b8ba-04cbefec39fb\") " pod="opendatahub/opendatahub-operator-controller-manager-6bcdb85766-9j9s2"
Oct 30 00:35:48.918432 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:35:48.918347    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/e7944126-c185-43db-b8ba-04cbefec39fb-metrics-cert\") pod \"opendatahub-operator-controller-manager-6bcdb85766-9j9s2\" (UID: \"e7944126-c185-43db-b8ba-04cbefec39fb\") " pod="opendatahub/opendatahub-operator-controller-manager-6bcdb85766-9j9s2"
Oct 30 00:35:48.918432 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:35:48.918425    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e7944126-c185-43db-b8ba-04cbefec39fb-tmp\") pod \"opendatahub-operator-controller-manager-6bcdb85766-9j9s2\" (UID: \"e7944126-c185-43db-b8ba-04cbefec39fb\") " pod="opendatahub/opendatahub-operator-controller-manager-6bcdb85766-9j9s2"
Oct 30 00:35:48.918718 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:35:48.918687    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e7944126-c185-43db-b8ba-04cbefec39fb-manifests\") pod \"opendatahub-operator-controller-manager-6bcdb85766-9j9s2\" (UID: \"e7944126-c185-43db-b8ba-04cbefec39fb\") " pod="opendatahub/opendatahub-operator-controller-manager-6bcdb85766-9j9s2"
Oct 30 00:35:48.920736 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:35:48.920708    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e7944126-c185-43db-b8ba-04cbefec39fb-tmp\") pod \"opendatahub-operator-controller-manager-6bcdb85766-9j9s2\" (UID: \"e7944126-c185-43db-b8ba-04cbefec39fb\") " pod="opendatahub/opendatahub-operator-controller-manager-6bcdb85766-9j9s2"
Oct 30 00:35:48.920856 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:35:48.920715    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e7944126-c185-43db-b8ba-04cbefec39fb-webhook-cert\") pod \"opendatahub-operator-controller-manager-6bcdb85766-9j9s2\" (UID: \"e7944126-c185-43db-b8ba-04cbefec39fb\") " pod="opendatahub/opendatahub-operator-controller-manager-6bcdb85766-9j9s2"
Oct 30 00:35:48.920914 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:35:48.920854    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e7944126-c185-43db-b8ba-04cbefec39fb-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6bcdb85766-9j9s2\" (UID: \"e7944126-c185-43db-b8ba-04cbefec39fb\") " pod="opendatahub/opendatahub-operator-controller-manager-6bcdb85766-9j9s2"
Oct 30 00:35:48.920955 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:35:48.920922    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/e7944126-c185-43db-b8ba-04cbefec39fb-metrics-cert\") pod \"opendatahub-operator-controller-manager-6bcdb85766-9j9s2\" (UID: \"e7944126-c185-43db-b8ba-04cbefec39fb\") " pod="opendatahub/opendatahub-operator-controller-manager-6bcdb85766-9j9s2"
Oct 30 00:35:48.947468 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:35:48.947415    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gh7pq\" (UniqueName: \"kubernetes.io/projected/e7944126-c185-43db-b8ba-04cbefec39fb-kube-api-access-gh7pq\") pod \"opendatahub-operator-controller-manager-6bcdb85766-9j9s2\" (UID: \"e7944126-c185-43db-b8ba-04cbefec39fb\") " pod="opendatahub/opendatahub-operator-controller-manager-6bcdb85766-9j9s2"
Oct 30 00:35:49.009227 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:35:49.009194    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6bcdb85766-9j9s2"
Oct 30 00:35:49.412638 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:35:49.412554    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6bcdb85766-9j9s2"]
Oct 30 00:35:49.424335 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:35:49.424307    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7944126_c185_43db_b8ba_04cbefec39fb.slice/crio-9a5ecfd74d1e3a56597f6d161db7a92054d3ca319c39e3bd93cd415d10e63af5 WatchSource:0}: Error finding container 9a5ecfd74d1e3a56597f6d161db7a92054d3ca319c39e3bd93cd415d10e63af5: Status 404 returned error can't find the container with id 9a5ecfd74d1e3a56597f6d161db7a92054d3ca319c39e3bd93cd415d10e63af5
Oct 30 00:35:49.542591 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:35:49.542559    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6bcdb85766-9j9s2" event={"ID":"e7944126-c185-43db-b8ba-04cbefec39fb","Type":"ContainerStarted","Data":"9a5ecfd74d1e3a56597f6d161db7a92054d3ca319c39e3bd93cd415d10e63af5"}
Oct 30 00:35:52.557054 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:35:52.557008    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6bcdb85766-9j9s2" event={"ID":"e7944126-c185-43db-b8ba-04cbefec39fb","Type":"ContainerStarted","Data":"e994cd1814593110ff84824068f3e87161031000d6ce9ee9c599df793eb05e25"}
Oct 30 00:35:53.561696 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:35:53.561661    5002 generic.go:356] "Generic (PLEG): container finished" podID="e7944126-c185-43db-b8ba-04cbefec39fb" containerID="e994cd1814593110ff84824068f3e87161031000d6ce9ee9c599df793eb05e25" exitCode=0
Oct 30 00:35:53.562095 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:35:53.561709    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6bcdb85766-9j9s2" event={"ID":"e7944126-c185-43db-b8ba-04cbefec39fb","Type":"ContainerDied","Data":"e994cd1814593110ff84824068f3e87161031000d6ce9ee9c599df793eb05e25"}
Oct 30 00:35:54.567897 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:35:54.567858    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6bcdb85766-9j9s2" event={"ID":"e7944126-c185-43db-b8ba-04cbefec39fb","Type":"ContainerStarted","Data":"cd6ec839543e4ac1ed90e0779e85e7d9322214b723325f6ebb51140d8d8bfc5c"}
Oct 30 00:35:54.568409 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:35:54.567977    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-6bcdb85766-9j9s2"
Oct 30 00:35:54.664856 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:35:54.664803    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-6bcdb85766-9j9s2" podStartSLOduration=3.162817698 podStartE2EDuration="6.66479037s" totalImagesPullingTime="3.014728295s" totalInitContainerRuntime="487.244377ms" isStatefulPod=false podCreationTimestamp="2026-10-30 00:35:48 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:35:54.649804467 +0000 UTC m=+250.266237366" watchObservedRunningTime="2026-10-30 00:35:54.66479037 +0000 UTC m=+250.281223243"
Oct 30 00:36:05.574405 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:36:05.574344    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-6bcdb85766-9j9s2"
Oct 30 00:36:06.608031 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:36:06.607995    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-7cf8f946d-hmhtc"]
Oct 30 00:36:06.610557 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:36:06.610534    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7cf8f946d-hmhtc"
Oct 30 00:36:06.635586 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:36:06.635559    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7cf8f946d-hmhtc"]
Oct 30 00:36:06.721859 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:36:06.721830    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Oct 30 00:36:06.721859 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:36:06.721830    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Oct 30 00:36:06.722069 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:36:06.721898    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Oct 30 00:36:06.722363 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:36:06.722346    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Oct 30 00:36:06.722424 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:36:06.722413    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-d66jw\""
Oct 30 00:36:06.722524 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:36:06.722509    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Oct 30 00:36:06.767093 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:36:06.767059    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/52c38cfd-f05e-49d1-bcdb-06196733beb2-manager-config\") pod \"lws-controller-manager-7cf8f946d-hmhtc\" (UID: \"52c38cfd-f05e-49d1-bcdb-06196733beb2\") " pod="openshift-lws-operator/lws-controller-manager-7cf8f946d-hmhtc"
Oct 30 00:36:06.767093 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:36:06.767096    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcbwl\" (UniqueName: \"kubernetes.io/projected/52c38cfd-f05e-49d1-bcdb-06196733beb2-kube-api-access-lcbwl\") pod \"lws-controller-manager-7cf8f946d-hmhtc\" (UID: \"52c38cfd-f05e-49d1-bcdb-06196733beb2\") " pod="openshift-lws-operator/lws-controller-manager-7cf8f946d-hmhtc"
Oct 30 00:36:06.767326 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:36:06.767119    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/52c38cfd-f05e-49d1-bcdb-06196733beb2-cert\") pod \"lws-controller-manager-7cf8f946d-hmhtc\" (UID: \"52c38cfd-f05e-49d1-bcdb-06196733beb2\") " pod="openshift-lws-operator/lws-controller-manager-7cf8f946d-hmhtc"
Oct 30 00:36:06.767326 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:36:06.767135    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/52c38cfd-f05e-49d1-bcdb-06196733beb2-metrics-cert\") pod \"lws-controller-manager-7cf8f946d-hmhtc\" (UID: \"52c38cfd-f05e-49d1-bcdb-06196733beb2\") " pod="openshift-lws-operator/lws-controller-manager-7cf8f946d-hmhtc"
Oct 30 00:36:06.868180 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:36:06.868082    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/52c38cfd-f05e-49d1-bcdb-06196733beb2-cert\") pod \"lws-controller-manager-7cf8f946d-hmhtc\" (UID: \"52c38cfd-f05e-49d1-bcdb-06196733beb2\") " pod="openshift-lws-operator/lws-controller-manager-7cf8f946d-hmhtc"
Oct 30 00:36:06.868180 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:36:06.868126    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/52c38cfd-f05e-49d1-bcdb-06196733beb2-metrics-cert\") pod \"lws-controller-manager-7cf8f946d-hmhtc\" (UID: \"52c38cfd-f05e-49d1-bcdb-06196733beb2\") " pod="openshift-lws-operator/lws-controller-manager-7cf8f946d-hmhtc"
Oct 30 00:36:06.868463 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:36:06.868219    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/52c38cfd-f05e-49d1-bcdb-06196733beb2-manager-config\") pod \"lws-controller-manager-7cf8f946d-hmhtc\" (UID: \"52c38cfd-f05e-49d1-bcdb-06196733beb2\") " pod="openshift-lws-operator/lws-controller-manager-7cf8f946d-hmhtc"
Oct 30 00:36:06.868463 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:36:06.868245    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcbwl\" (UniqueName: \"kubernetes.io/projected/52c38cfd-f05e-49d1-bcdb-06196733beb2-kube-api-access-lcbwl\") pod \"lws-controller-manager-7cf8f946d-hmhtc\" (UID: \"52c38cfd-f05e-49d1-bcdb-06196733beb2\") " pod="openshift-lws-operator/lws-controller-manager-7cf8f946d-hmhtc"
Oct 30 00:36:06.868890 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:36:06.868864    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/52c38cfd-f05e-49d1-bcdb-06196733beb2-manager-config\") pod \"lws-controller-manager-7cf8f946d-hmhtc\" (UID: \"52c38cfd-f05e-49d1-bcdb-06196733beb2\") " pod="openshift-lws-operator/lws-controller-manager-7cf8f946d-hmhtc"
Oct 30 00:36:06.870638 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:36:06.870613    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/52c38cfd-f05e-49d1-bcdb-06196733beb2-metrics-cert\") pod \"lws-controller-manager-7cf8f946d-hmhtc\" (UID: \"52c38cfd-f05e-49d1-bcdb-06196733beb2\") " pod="openshift-lws-operator/lws-controller-manager-7cf8f946d-hmhtc"
Oct 30 00:36:06.870735 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:36:06.870667    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/52c38cfd-f05e-49d1-bcdb-06196733beb2-cert\") pod \"lws-controller-manager-7cf8f946d-hmhtc\" (UID: \"52c38cfd-f05e-49d1-bcdb-06196733beb2\") " pod="openshift-lws-operator/lws-controller-manager-7cf8f946d-hmhtc"
Oct 30 00:36:06.898162 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:36:06.898129    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcbwl\" (UniqueName: \"kubernetes.io/projected/52c38cfd-f05e-49d1-bcdb-06196733beb2-kube-api-access-lcbwl\") pod \"lws-controller-manager-7cf8f946d-hmhtc\" (UID: \"52c38cfd-f05e-49d1-bcdb-06196733beb2\") " pod="openshift-lws-operator/lws-controller-manager-7cf8f946d-hmhtc"
Oct 30 00:36:06.920304 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:36:06.920278    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7cf8f946d-hmhtc"
Oct 30 00:36:07.217292 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:36:07.217252    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7cf8f946d-hmhtc"]
Oct 30 00:36:07.218559 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:36:07.218530    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52c38cfd_f05e_49d1_bcdb_06196733beb2.slice/crio-c09d67a7c38b38c9cfaba809b57b344d6787fb73d7c6c00e9aa6fe6037511a09 WatchSource:0}: Error finding container c09d67a7c38b38c9cfaba809b57b344d6787fb73d7c6c00e9aa6fe6037511a09: Status 404 returned error can't find the container with id c09d67a7c38b38c9cfaba809b57b344d6787fb73d7c6c00e9aa6fe6037511a09
Oct 30 00:36:07.617353 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:36:07.617263    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7cf8f946d-hmhtc" event={"ID":"52c38cfd-f05e-49d1-bcdb-06196733beb2","Type":"ContainerStarted","Data":"c09d67a7c38b38c9cfaba809b57b344d6787fb73d7c6c00e9aa6fe6037511a09"}
Oct 30 00:36:11.636828 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:36:11.636747    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7cf8f946d-hmhtc" event={"ID":"52c38cfd-f05e-49d1-bcdb-06196733beb2","Type":"ContainerStarted","Data":"0dba692ebe226a441f181fbb61784ff689ffa3f5b460565308d975085ccf00cf"}
Oct 30 00:36:11.637218 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:36:11.636892    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-7cf8f946d-hmhtc"
Oct 30 00:36:11.685337 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:36:11.685284    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-7cf8f946d-hmhtc" podStartSLOduration=1.528381682 podStartE2EDuration="5.68526778s" totalImagesPullingTime="4.156886098s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:36:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:36:11.684390927 +0000 UTC m=+267.300823797" watchObservedRunningTime="2026-10-30 00:36:11.68526778 +0000 UTC m=+267.301700661"
Oct 30 00:36:12.798145 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:36:12.798112    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-observability-584f99c788-h48gx"]
Oct 30 00:36:12.800375 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:36:12.800357    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-observability-584f99c788-h48gx"
Oct 30 00:36:12.819788 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:36:12.819764    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-observability-webhook-cert\""
Oct 30 00:36:12.821493 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:36:12.821467    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-observability-584f99c788-h48gx"]
Oct 30 00:36:12.833748 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:36:12.833729    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-observability-dockercfg-wc8sv\""
Oct 30 00:36:12.859713 ip-10-0-128-7 kubenswrapper[5002]: E1030 00:36:12.859679    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbjdq" containerName="extractor"
Oct 30 00:36:12.921202 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:36:12.921166    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-certs\" (UniqueName: \"kubernetes.io/secret/a2a28b50-085d-43ec-bcd1-22d8d5a94cf9-webhook-certs\") pod \"odh-observability-584f99c788-h48gx\" (UID: \"a2a28b50-085d-43ec-bcd1-22d8d5a94cf9\") " pod="opendatahub/odh-observability-584f99c788-h48gx"
Oct 30 00:36:12.921202 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:36:12.921211    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lwr8\" (UniqueName: \"kubernetes.io/projected/a2a28b50-085d-43ec-bcd1-22d8d5a94cf9-kube-api-access-4lwr8\") pod \"odh-observability-584f99c788-h48gx\" (UID: \"a2a28b50-085d-43ec-bcd1-22d8d5a94cf9\") " pod="opendatahub/odh-observability-584f99c788-h48gx"
Oct 30 00:36:13.021781 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:36:13.021745    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lwr8\" (UniqueName: \"kubernetes.io/projected/a2a28b50-085d-43ec-bcd1-22d8d5a94cf9-kube-api-access-4lwr8\") pod \"odh-observability-584f99c788-h48gx\" (UID: \"a2a28b50-085d-43ec-bcd1-22d8d5a94cf9\") " pod="opendatahub/odh-observability-584f99c788-h48gx"
Oct 30 00:36:13.021946 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:36:13.021821    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-certs\" (UniqueName: \"kubernetes.io/secret/a2a28b50-085d-43ec-bcd1-22d8d5a94cf9-webhook-certs\") pod \"odh-observability-584f99c788-h48gx\" (UID: \"a2a28b50-085d-43ec-bcd1-22d8d5a94cf9\") " pod="opendatahub/odh-observability-584f99c788-h48gx"
Oct 30 00:36:13.021946 ip-10-0-128-7 kubenswrapper[5002]: E1030 00:36:13.021910    5002 secret.go:189] Couldn't get secret opendatahub/odh-observability-webhook-cert: secret "odh-observability-webhook-cert" not found
Oct 30 00:36:13.022072 ip-10-0-128-7 kubenswrapper[5002]: E1030 00:36:13.021962    5002 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a2a28b50-085d-43ec-bcd1-22d8d5a94cf9-webhook-certs podName:a2a28b50-085d-43ec-bcd1-22d8d5a94cf9 nodeName:}" failed. No retries permitted until 2026-10-30 00:36:13.521948066 +0000 UTC m=+269.138380936 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-certs" (UniqueName: "kubernetes.io/secret/a2a28b50-085d-43ec-bcd1-22d8d5a94cf9-webhook-certs") pod "odh-observability-584f99c788-h48gx" (UID: "a2a28b50-085d-43ec-bcd1-22d8d5a94cf9") : secret "odh-observability-webhook-cert" not found
Oct 30 00:36:13.049604 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:36:13.049537    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lwr8\" (UniqueName: \"kubernetes.io/projected/a2a28b50-085d-43ec-bcd1-22d8d5a94cf9-kube-api-access-4lwr8\") pod \"odh-observability-584f99c788-h48gx\" (UID: \"a2a28b50-085d-43ec-bcd1-22d8d5a94cf9\") " pod="opendatahub/odh-observability-584f99c788-h48gx"
Oct 30 00:36:13.525831 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:36:13.525784    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-certs\" (UniqueName: \"kubernetes.io/secret/a2a28b50-085d-43ec-bcd1-22d8d5a94cf9-webhook-certs\") pod \"odh-observability-584f99c788-h48gx\" (UID: \"a2a28b50-085d-43ec-bcd1-22d8d5a94cf9\") " pod="opendatahub/odh-observability-584f99c788-h48gx"
Oct 30 00:36:13.528307 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:36:13.528287    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-certs\" (UniqueName: \"kubernetes.io/secret/a2a28b50-085d-43ec-bcd1-22d8d5a94cf9-webhook-certs\") pod \"odh-observability-584f99c788-h48gx\" (UID: \"a2a28b50-085d-43ec-bcd1-22d8d5a94cf9\") " pod="opendatahub/odh-observability-584f99c788-h48gx"
Oct 30 00:36:13.711436 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:36:13.711397    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-observability-584f99c788-h48gx"
Oct 30 00:36:14.007482 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:36:14.007455    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-observability-584f99c788-h48gx"]
Oct 30 00:36:14.008992 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:36:14.008967    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2a28b50_085d_43ec_bcd1_22d8d5a94cf9.slice/crio-7ba83681951bdc13231dd88604e6d974b1822fe4911e39cf9fd64ac62de44b0e WatchSource:0}: Error finding container 7ba83681951bdc13231dd88604e6d974b1822fe4911e39cf9fd64ac62de44b0e: Status 404 returned error can't find the container with id 7ba83681951bdc13231dd88604e6d974b1822fe4911e39cf9fd64ac62de44b0e
Oct 30 00:36:14.650505 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:36:14.650466    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-observability-584f99c788-h48gx" event={"ID":"a2a28b50-085d-43ec-bcd1-22d8d5a94cf9","Type":"ContainerStarted","Data":"7ba83681951bdc13231dd88604e6d974b1822fe4911e39cf9fd64ac62de44b0e"}
Oct 30 00:36:16.662225 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:36:16.662136    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-observability-584f99c788-h48gx" event={"ID":"a2a28b50-085d-43ec-bcd1-22d8d5a94cf9","Type":"ContainerStarted","Data":"65a403609c884d7a48c6f19fb94e585ac09a56507cba3bf9084d2b9a1cf089e5"}
Oct 30 00:36:16.662653 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:36:16.662251    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-observability-584f99c788-h48gx"
Oct 30 00:36:16.706313 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:36:16.706259    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-observability-584f99c788-h48gx" podStartSLOduration=2.311752343 podStartE2EDuration="4.7062458s" totalImagesPullingTime="2.394493457s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:36:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:36:16.705758763 +0000 UTC m=+272.322191641" watchObservedRunningTime="2026-10-30 00:36:16.7062458 +0000 UTC m=+272.322678678"
Oct 30 00:36:22.643203 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:36:22.643170    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-7cf8f946d-hmhtc"
Oct 30 00:36:26.748985 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:36:26.748951    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-module-controller-manager-5cfb77bf8c-lm8qn"]
Oct 30 00:36:26.752106 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:36:26.752088    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-lm8qn"
Oct 30 00:36:26.782594 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:36:26.782560    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-module-controller-manager-dockercfg-tkctv\""
Oct 30 00:36:26.811058 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:36:26.811021    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"odh-trusted-ca-bundle\""
Oct 30 00:36:26.812388 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:36:26.812362    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-5cfb77bf8c-lm8qn"]
Oct 30 00:36:26.937816 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:36:26.937781    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a9373505-073b-4c82-92ee-ca6365ea95dc-trusted-ca\") pod \"kserve-module-controller-manager-5cfb77bf8c-lm8qn\" (UID: \"a9373505-073b-4c82-92ee-ca6365ea95dc\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-lm8qn"
Oct 30 00:36:26.937961 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:36:26.937827    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdhmg\" (UniqueName: \"kubernetes.io/projected/a9373505-073b-4c82-92ee-ca6365ea95dc-kube-api-access-wdhmg\") pod \"kserve-module-controller-manager-5cfb77bf8c-lm8qn\" (UID: \"a9373505-073b-4c82-92ee-ca6365ea95dc\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-lm8qn"
Oct 30 00:36:26.937961 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:36:26.937851    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a9373505-073b-4c82-92ee-ca6365ea95dc-tmp\") pod \"kserve-module-controller-manager-5cfb77bf8c-lm8qn\" (UID: \"a9373505-073b-4c82-92ee-ca6365ea95dc\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-lm8qn"
Oct 30 00:36:26.938053 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:36:26.937975    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/a9373505-073b-4c82-92ee-ca6365ea95dc-manifests\") pod \"kserve-module-controller-manager-5cfb77bf8c-lm8qn\" (UID: \"a9373505-073b-4c82-92ee-ca6365ea95dc\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-lm8qn"
Oct 30 00:36:27.038777 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:36:27.038700    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/a9373505-073b-4c82-92ee-ca6365ea95dc-manifests\") pod \"kserve-module-controller-manager-5cfb77bf8c-lm8qn\" (UID: \"a9373505-073b-4c82-92ee-ca6365ea95dc\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-lm8qn"
Oct 30 00:36:27.038777 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:36:27.038756    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a9373505-073b-4c82-92ee-ca6365ea95dc-trusted-ca\") pod \"kserve-module-controller-manager-5cfb77bf8c-lm8qn\" (UID: \"a9373505-073b-4c82-92ee-ca6365ea95dc\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-lm8qn"
Oct 30 00:36:27.038928 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:36:27.038785    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdhmg\" (UniqueName: \"kubernetes.io/projected/a9373505-073b-4c82-92ee-ca6365ea95dc-kube-api-access-wdhmg\") pod \"kserve-module-controller-manager-5cfb77bf8c-lm8qn\" (UID: \"a9373505-073b-4c82-92ee-ca6365ea95dc\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-lm8qn"
Oct 30 00:36:27.038928 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:36:27.038804    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a9373505-073b-4c82-92ee-ca6365ea95dc-tmp\") pod \"kserve-module-controller-manager-5cfb77bf8c-lm8qn\" (UID: \"a9373505-073b-4c82-92ee-ca6365ea95dc\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-lm8qn"
Oct 30 00:36:27.039158 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:36:27.039138    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a9373505-073b-4c82-92ee-ca6365ea95dc-tmp\") pod \"kserve-module-controller-manager-5cfb77bf8c-lm8qn\" (UID: \"a9373505-073b-4c82-92ee-ca6365ea95dc\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-lm8qn"
Oct 30 00:36:27.039289 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:36:27.039264    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/a9373505-073b-4c82-92ee-ca6365ea95dc-manifests\") pod \"kserve-module-controller-manager-5cfb77bf8c-lm8qn\" (UID: \"a9373505-073b-4c82-92ee-ca6365ea95dc\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-lm8qn"
Oct 30 00:36:27.039675 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:36:27.039655    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a9373505-073b-4c82-92ee-ca6365ea95dc-trusted-ca\") pod \"kserve-module-controller-manager-5cfb77bf8c-lm8qn\" (UID: \"a9373505-073b-4c82-92ee-ca6365ea95dc\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-lm8qn"
Oct 30 00:36:27.064347 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:36:27.064316    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdhmg\" (UniqueName: \"kubernetes.io/projected/a9373505-073b-4c82-92ee-ca6365ea95dc-kube-api-access-wdhmg\") pod \"kserve-module-controller-manager-5cfb77bf8c-lm8qn\" (UID: \"a9373505-073b-4c82-92ee-ca6365ea95dc\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-lm8qn"
Oct 30 00:36:27.364164 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:36:27.364065    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-lm8qn"
Oct 30 00:36:27.668665 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:36:27.668426    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-observability-584f99c788-h48gx"
Oct 30 00:36:27.671925 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:36:27.671893    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-5cfb77bf8c-lm8qn"]
Oct 30 00:36:27.672550 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:36:27.672523    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9373505_073b_4c82_92ee_ca6365ea95dc.slice/crio-cc0f5d965c668a3c19854edec1480e040a3328819d991eb637089f1d64737f92 WatchSource:0}: Error finding container cc0f5d965c668a3c19854edec1480e040a3328819d991eb637089f1d64737f92: Status 404 returned error can't find the container with id cc0f5d965c668a3c19854edec1480e040a3328819d991eb637089f1d64737f92
Oct 30 00:36:27.710683 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:36:27.710653    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-lm8qn" event={"ID":"a9373505-073b-4c82-92ee-ca6365ea95dc","Type":"ContainerStarted","Data":"cc0f5d965c668a3c19854edec1480e040a3328819d991eb637089f1d64737f92"}
Oct 30 00:36:30.725203 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:36:30.725164    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-lm8qn" event={"ID":"a9373505-073b-4c82-92ee-ca6365ea95dc","Type":"ContainerStarted","Data":"ed5441bd1e9f78b7eed4eaee19570a76be9903c3eaa242ad03b95f08f3088a0b"}
Oct 30 00:36:30.725700 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:36:30.725314    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-lm8qn"
Oct 30 00:36:30.787615 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:36:30.787560    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-lm8qn" podStartSLOduration=2.509662933 podStartE2EDuration="4.787545709s" totalImagesPullingTime="2.277882776s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:36:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:36:30.786289081 +0000 UTC m=+286.402721987" watchObservedRunningTime="2026-10-30 00:36:30.787545709 +0000 UTC m=+286.403978625"
Oct 30 00:36:42.860781 ip-10-0-128-7 kubenswrapper[5002]: E1030 00:36:42.860744    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbjdq" containerName="extractor"
Oct 30 00:36:44.520142 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:36:44.520113    5002 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-g7c2w_5dc56e7e-f3c1-43eb-a21b-8be6bb4706cf/console-operator/1.log"
Oct 30 00:36:44.520653 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:36:44.520113    5002 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-g7c2w_5dc56e7e-f3c1-43eb-a21b-8be6bb4706cf/console-operator/1.log"
Oct 30 00:36:51.749820 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:36:51.749785    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-lm8qn"
Oct 30 00:37:12.860876 ip-10-0-128-7 kubenswrapper[5002]: E1030 00:37:12.860794    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbjdq" containerName="extractor"
Oct 30 00:37:15.216770 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:37:15.216734    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-k2k4j"]
Oct 30 00:37:15.226944 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:37:15.226920    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-k2k4j"
Oct 30 00:37:15.243836 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:37:15.243801    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-k2k4j"]
Oct 30 00:37:15.256320 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:37:15.256298    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-p4s8z\""
Oct 30 00:37:15.261905 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:37:15.261880    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Oct 30 00:37:15.263345 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:37:15.263319    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-n9mwb"]
Oct 30 00:37:15.266701 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:37:15.266685    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-n9mwb"
Oct 30 00:37:15.283192 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:37:15.283172    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-n9mwb"]
Oct 30 00:37:15.297920 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:37:15.297899    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-4gc22"]
Oct 30 00:37:15.301173 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:37:15.301157    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-4gc22"
Oct 30 00:37:15.323190 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:37:15.323163    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-4gc22"]
Oct 30 00:37:15.347507 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:37:15.347474    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49t87\" (UniqueName: \"kubernetes.io/projected/3f63d6f0-ed39-4ac0-90e7-74fd716b58f8-kube-api-access-49t87\") pod \"kserve-controller-manager-776cc66d47-k2k4j\" (UID: \"3f63d6f0-ed39-4ac0-90e7-74fd716b58f8\") " pod="opendatahub/kserve-controller-manager-776cc66d47-k2k4j"
Oct 30 00:37:15.347685 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:37:15.347530    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3f63d6f0-ed39-4ac0-90e7-74fd716b58f8-cert\") pod \"kserve-controller-manager-776cc66d47-k2k4j\" (UID: \"3f63d6f0-ed39-4ac0-90e7-74fd716b58f8\") " pod="opendatahub/kserve-controller-manager-776cc66d47-k2k4j"
Oct 30 00:37:15.348481 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:37:15.348459    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Oct 30 00:37:15.348481 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:37:15.348472    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-m9v8t\""
Oct 30 00:37:15.348624 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:37:15.348499    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-xg4wj\""
Oct 30 00:37:15.348624 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:37:15.348566    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Oct 30 00:37:15.448325 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:37:15.448290    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3f63d6f0-ed39-4ac0-90e7-74fd716b58f8-cert\") pod \"kserve-controller-manager-776cc66d47-k2k4j\" (UID: \"3f63d6f0-ed39-4ac0-90e7-74fd716b58f8\") " pod="opendatahub/kserve-controller-manager-776cc66d47-k2k4j"
Oct 30 00:37:15.448541 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:37:15.448419    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19afed17-6147-4955-86b1-3440be3f780a-tls-certs\") pod \"model-serving-api-75bd89fdfc-4gc22\" (UID: \"19afed17-6147-4955-86b1-3440be3f780a\") " pod="opendatahub/model-serving-api-75bd89fdfc-4gc22"
Oct 30 00:37:15.448620 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:37:15.448565    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49t87\" (UniqueName: \"kubernetes.io/projected/3f63d6f0-ed39-4ac0-90e7-74fd716b58f8-kube-api-access-49t87\") pod \"kserve-controller-manager-776cc66d47-k2k4j\" (UID: \"3f63d6f0-ed39-4ac0-90e7-74fd716b58f8\") " pod="opendatahub/kserve-controller-manager-776cc66d47-k2k4j"
Oct 30 00:37:15.448620 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:37:15.448601    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdszt\" (UniqueName: \"kubernetes.io/projected/19afed17-6147-4955-86b1-3440be3f780a-kube-api-access-jdszt\") pod \"model-serving-api-75bd89fdfc-4gc22\" (UID: \"19afed17-6147-4955-86b1-3440be3f780a\") " pod="opendatahub/model-serving-api-75bd89fdfc-4gc22"
Oct 30 00:37:15.448730 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:37:15.448623    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6920db4e-c843-4524-a14e-e9e1064b0b9d-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-n9mwb\" (UID: \"6920db4e-c843-4524-a14e-e9e1064b0b9d\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-n9mwb"
Oct 30 00:37:15.448730 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:37:15.448642    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tv82\" (UniqueName: \"kubernetes.io/projected/6920db4e-c843-4524-a14e-e9e1064b0b9d-kube-api-access-6tv82\") pod \"llmisvc-controller-manager-869d5dcc5d-n9mwb\" (UID: \"6920db4e-c843-4524-a14e-e9e1064b0b9d\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-n9mwb"
Oct 30 00:37:15.450802 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:37:15.450780    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3f63d6f0-ed39-4ac0-90e7-74fd716b58f8-cert\") pod \"kserve-controller-manager-776cc66d47-k2k4j\" (UID: \"3f63d6f0-ed39-4ac0-90e7-74fd716b58f8\") " pod="opendatahub/kserve-controller-manager-776cc66d47-k2k4j"
Oct 30 00:37:15.475247 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:37:15.475186    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-49t87\" (UniqueName: \"kubernetes.io/projected/3f63d6f0-ed39-4ac0-90e7-74fd716b58f8-kube-api-access-49t87\") pod \"kserve-controller-manager-776cc66d47-k2k4j\" (UID: \"3f63d6f0-ed39-4ac0-90e7-74fd716b58f8\") " pod="opendatahub/kserve-controller-manager-776cc66d47-k2k4j"
Oct 30 00:37:15.539206 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:37:15.539167    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-k2k4j"
Oct 30 00:37:15.549352 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:37:15.549296    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19afed17-6147-4955-86b1-3440be3f780a-tls-certs\") pod \"model-serving-api-75bd89fdfc-4gc22\" (UID: \"19afed17-6147-4955-86b1-3440be3f780a\") " pod="opendatahub/model-serving-api-75bd89fdfc-4gc22"
Oct 30 00:37:15.549516 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:37:15.549435    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdszt\" (UniqueName: \"kubernetes.io/projected/19afed17-6147-4955-86b1-3440be3f780a-kube-api-access-jdszt\") pod \"model-serving-api-75bd89fdfc-4gc22\" (UID: \"19afed17-6147-4955-86b1-3440be3f780a\") " pod="opendatahub/model-serving-api-75bd89fdfc-4gc22"
Oct 30 00:37:15.549516 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:37:15.549468    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6920db4e-c843-4524-a14e-e9e1064b0b9d-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-n9mwb\" (UID: \"6920db4e-c843-4524-a14e-e9e1064b0b9d\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-n9mwb"
Oct 30 00:37:15.549516 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:37:15.549497    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tv82\" (UniqueName: \"kubernetes.io/projected/6920db4e-c843-4524-a14e-e9e1064b0b9d-kube-api-access-6tv82\") pod \"llmisvc-controller-manager-869d5dcc5d-n9mwb\" (UID: \"6920db4e-c843-4524-a14e-e9e1064b0b9d\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-n9mwb"
Oct 30 00:37:15.551936 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:37:15.551912    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19afed17-6147-4955-86b1-3440be3f780a-tls-certs\") pod \"model-serving-api-75bd89fdfc-4gc22\" (UID: \"19afed17-6147-4955-86b1-3440be3f780a\") " pod="opendatahub/model-serving-api-75bd89fdfc-4gc22"
Oct 30 00:37:15.552352 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:37:15.552330    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6920db4e-c843-4524-a14e-e9e1064b0b9d-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-n9mwb\" (UID: \"6920db4e-c843-4524-a14e-e9e1064b0b9d\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-n9mwb"
Oct 30 00:37:15.574637 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:37:15.574606    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdszt\" (UniqueName: \"kubernetes.io/projected/19afed17-6147-4955-86b1-3440be3f780a-kube-api-access-jdszt\") pod \"model-serving-api-75bd89fdfc-4gc22\" (UID: \"19afed17-6147-4955-86b1-3440be3f780a\") " pod="opendatahub/model-serving-api-75bd89fdfc-4gc22"
Oct 30 00:37:15.574781 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:37:15.574660    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tv82\" (UniqueName: \"kubernetes.io/projected/6920db4e-c843-4524-a14e-e9e1064b0b9d-kube-api-access-6tv82\") pod \"llmisvc-controller-manager-869d5dcc5d-n9mwb\" (UID: \"6920db4e-c843-4524-a14e-e9e1064b0b9d\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-n9mwb"
Oct 30 00:37:15.579458 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:37:15.579432    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-n9mwb"
Oct 30 00:37:15.613636 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:37:15.613604    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-4gc22"
Oct 30 00:37:15.837921 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:37:15.837894    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-k2k4j"]
Oct 30 00:37:15.839024 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:37:15.838985    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f63d6f0_ed39_4ac0_90e7_74fd716b58f8.slice/crio-1cfa44cde994a6068c9b6ec754298333549a1f059550412fa682447c4ac64725 WatchSource:0}: Error finding container 1cfa44cde994a6068c9b6ec754298333549a1f059550412fa682447c4ac64725: Status 404 returned error can't find the container with id 1cfa44cde994a6068c9b6ec754298333549a1f059550412fa682447c4ac64725
Oct 30 00:37:15.840429 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:37:15.840411    5002 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 00:37:15.882195 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:37:15.882166    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-n9mwb"]
Oct 30 00:37:15.882827 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:37:15.882805    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod6920db4e_c843_4524_a14e_e9e1064b0b9d.slice/crio-05a2b3567f6507c4b1cde01b40781d0d5e27afd6239219f27b13b22d783b3307 WatchSource:0}: Error finding container 05a2b3567f6507c4b1cde01b40781d0d5e27afd6239219f27b13b22d783b3307: Status 404 returned error can't find the container with id 05a2b3567f6507c4b1cde01b40781d0d5e27afd6239219f27b13b22d783b3307
Oct 30 00:37:15.907872 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:37:15.907839    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-n9mwb" event={"ID":"6920db4e-c843-4524-a14e-e9e1064b0b9d","Type":"ContainerStarted","Data":"05a2b3567f6507c4b1cde01b40781d0d5e27afd6239219f27b13b22d783b3307"}
Oct 30 00:37:15.908789 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:37:15.908767    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-k2k4j" event={"ID":"3f63d6f0-ed39-4ac0-90e7-74fd716b58f8","Type":"ContainerStarted","Data":"1cfa44cde994a6068c9b6ec754298333549a1f059550412fa682447c4ac64725"}
Oct 30 00:37:15.918406 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:37:15.918370    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-4gc22"]
Oct 30 00:37:15.919754 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:37:15.919728    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19afed17_6147_4955_86b1_3440be3f780a.slice/crio-059c969d08594c7d2ce69a0e513e13a589f9825006631ace28f1539445e38379 WatchSource:0}: Error finding container 059c969d08594c7d2ce69a0e513e13a589f9825006631ace28f1539445e38379: Status 404 returned error can't find the container with id 059c969d08594c7d2ce69a0e513e13a589f9825006631ace28f1539445e38379
Oct 30 00:37:16.916806 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:37:16.916771    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-4gc22" event={"ID":"19afed17-6147-4955-86b1-3440be3f780a","Type":"ContainerStarted","Data":"059c969d08594c7d2ce69a0e513e13a589f9825006631ace28f1539445e38379"}
Oct 30 00:37:18.927549 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:37:18.927508    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-k2k4j" event={"ID":"3f63d6f0-ed39-4ac0-90e7-74fd716b58f8","Type":"ContainerStarted","Data":"236a4b73ff170a52ef23fa1171f6e6381b350a0efb84469360369b6371a2d0e2"}
Oct 30 00:37:18.928016 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:37:18.927578    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-776cc66d47-k2k4j"
Oct 30 00:37:18.929174 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:37:18.929144    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-4gc22" event={"ID":"19afed17-6147-4955-86b1-3440be3f780a","Type":"ContainerStarted","Data":"5d7b9fabd2c6fd8494aef92af438461c27b87321ed6ed429ad6e4e7bd62291a3"}
Oct 30 00:37:18.929297 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:37:18.929244    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-75bd89fdfc-4gc22"
Oct 30 00:37:18.930642 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:37:18.930618    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-n9mwb" event={"ID":"6920db4e-c843-4524-a14e-e9e1064b0b9d","Type":"ContainerStarted","Data":"5fe971621f83ace6d08d7971437dca58fa2e3987f5643e1643f4c6f42691f02a"}
Oct 30 00:37:18.930734 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:37:18.930671    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-n9mwb"
Oct 30 00:37:18.972061 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:37:18.972011    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-776cc66d47-k2k4j" podStartSLOduration=1.595122325 podStartE2EDuration="3.971997555s" totalImagesPullingTime="2.37687523s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:37:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:37:18.971010488 +0000 UTC m=+334.587443367" watchObservedRunningTime="2026-10-30 00:37:18.971997555 +0000 UTC m=+334.588430432"
Oct 30 00:37:19.015097 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:37:19.015039    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/model-serving-api-75bd89fdfc-4gc22" podStartSLOduration=2.026022767 podStartE2EDuration="4.015025887s" totalImagesPullingTime="1.98900312s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:37:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:37:19.014171318 +0000 UTC m=+334.630604197" watchObservedRunningTime="2026-10-30 00:37:19.015025887 +0000 UTC m=+334.631458744"
Oct 30 00:37:19.059237 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:37:19.059183    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-n9mwb" podStartSLOduration=1.719163094 podStartE2EDuration="4.059168439s" totalImagesPullingTime="2.340005345s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:37:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:37:19.058519435 +0000 UTC m=+334.674952325" watchObservedRunningTime="2026-10-30 00:37:19.059168439 +0000 UTC m=+334.675601317"
Oct 30 00:37:29.938395 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:37:29.938357    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-75bd89fdfc-4gc22"
Oct 30 00:37:42.859506 ip-10-0-128-7 kubenswrapper[5002]: E1030 00:37:42.859468    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbjdq" containerName="extractor"
Oct 30 00:37:49.936972 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:37:49.936937    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-n9mwb"
Oct 30 00:37:49.940486 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:37:49.940458    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-776cc66d47-k2k4j"
Oct 30 00:38:12.859893 ip-10-0-128-7 kubenswrapper[5002]: E1030 00:38:12.859846    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbjdq" containerName="extractor"
Oct 30 00:38:30.790301 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:38:30.790186    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-jrv82"]
Oct 30 00:38:30.793790 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:38:30.793766    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-jrv82"
Oct 30 00:38:30.822963 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:38:30.822938    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Oct 30 00:38:30.823176 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:38:30.823155    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-jrv82"]
Oct 30 00:38:30.835972 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:38:30.835947    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-tt5qh\""
Oct 30 00:38:30.835972 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:38:30.835949    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Oct 30 00:38:30.836160 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:38:30.835983    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Oct 30 00:38:30.836160 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:38:30.835947    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Oct 30 00:38:30.874256 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:38:30.874229    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/d21d8c35-34a7-42ba-89a9-b9754b6b339b-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-jrv82\" (UID: \"d21d8c35-34a7-42ba-89a9-b9754b6b339b\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-jrv82"
Oct 30 00:38:30.874428 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:38:30.874261    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwcbc\" (UniqueName: \"kubernetes.io/projected/d21d8c35-34a7-42ba-89a9-b9754b6b339b-kube-api-access-xwcbc\") pod \"kuadrant-console-plugin-66c4fb575d-jrv82\" (UID: \"d21d8c35-34a7-42ba-89a9-b9754b6b339b\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-jrv82"
Oct 30 00:38:30.874428 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:38:30.874303    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d21d8c35-34a7-42ba-89a9-b9754b6b339b-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-jrv82\" (UID: \"d21d8c35-34a7-42ba-89a9-b9754b6b339b\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-jrv82"
Oct 30 00:38:30.975227 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:38:30.975193    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d21d8c35-34a7-42ba-89a9-b9754b6b339b-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-jrv82\" (UID: \"d21d8c35-34a7-42ba-89a9-b9754b6b339b\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-jrv82"
Oct 30 00:38:30.975444 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:38:30.975263    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/d21d8c35-34a7-42ba-89a9-b9754b6b339b-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-jrv82\" (UID: \"d21d8c35-34a7-42ba-89a9-b9754b6b339b\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-jrv82"
Oct 30 00:38:30.975444 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:38:30.975293    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwcbc\" (UniqueName: \"kubernetes.io/projected/d21d8c35-34a7-42ba-89a9-b9754b6b339b-kube-api-access-xwcbc\") pod \"kuadrant-console-plugin-66c4fb575d-jrv82\" (UID: \"d21d8c35-34a7-42ba-89a9-b9754b6b339b\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-jrv82"
Oct 30 00:38:30.975958 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:38:30.975932    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d21d8c35-34a7-42ba-89a9-b9754b6b339b-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-jrv82\" (UID: \"d21d8c35-34a7-42ba-89a9-b9754b6b339b\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-jrv82"
Oct 30 00:38:30.977722 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:38:30.977700    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/d21d8c35-34a7-42ba-89a9-b9754b6b339b-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-jrv82\" (UID: \"d21d8c35-34a7-42ba-89a9-b9754b6b339b\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-jrv82"
Oct 30 00:38:31.008832 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:38:31.008810    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwcbc\" (UniqueName: \"kubernetes.io/projected/d21d8c35-34a7-42ba-89a9-b9754b6b339b-kube-api-access-xwcbc\") pod \"kuadrant-console-plugin-66c4fb575d-jrv82\" (UID: \"d21d8c35-34a7-42ba-89a9-b9754b6b339b\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-jrv82"
Oct 30 00:38:31.105706 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:38:31.105623    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-jrv82"
Oct 30 00:38:31.417799 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:38:31.417768    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-jrv82"]
Oct 30 00:38:31.418362 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:38:31.418331    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd21d8c35_34a7_42ba_89a9_b9754b6b339b.slice/crio-9cc2f9d91c7dce795a5cb9ec2ecc04a6b529bf32814e1296366c92dc7fd97a5d WatchSource:0}: Error finding container 9cc2f9d91c7dce795a5cb9ec2ecc04a6b529bf32814e1296366c92dc7fd97a5d: Status 404 returned error can't find the container with id 9cc2f9d91c7dce795a5cb9ec2ecc04a6b529bf32814e1296366c92dc7fd97a5d
Oct 30 00:38:32.243152 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:38:32.243117    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-jrv82" event={"ID":"d21d8c35-34a7-42ba-89a9-b9754b6b339b","Type":"ContainerStarted","Data":"9cc2f9d91c7dce795a5cb9ec2ecc04a6b529bf32814e1296366c92dc7fd97a5d"}
Oct 30 00:38:36.264287 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:38:36.264247    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-jrv82" event={"ID":"d21d8c35-34a7-42ba-89a9-b9754b6b339b","Type":"ContainerStarted","Data":"52a106375647bac451e5a38101924a583485deaf5f30a03438420da762196a08"}
Oct 30 00:38:36.307591 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:38:36.307496    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-jrv82" podStartSLOduration=1.695998493 podStartE2EDuration="6.307482759s" totalImagesPullingTime="4.611484266s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:38:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:38:36.306468258 +0000 UTC m=+411.922901137" watchObservedRunningTime="2026-10-30 00:38:36.307482759 +0000 UTC m=+411.923915629"
Oct 30 00:38:42.860203 ip-10-0-128-7 kubenswrapper[5002]: E1030 00:38:42.860164    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbjdq" containerName="extractor"
Oct 30 00:38:55.965103 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:38:55.965062    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-gm5b5"]
Oct 30 00:38:55.969627 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:38:55.969603    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-gm5b5"
Oct 30 00:38:55.979308 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:38:55.979278    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-gm5b5"]
Oct 30 00:38:55.991880 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:38:55.991857    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-pj9zs\""
Oct 30 00:38:56.096298 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:38:56.096258    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4d35acb0-477c-436e-bb24-7e15e827a5af-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-57bb9bfff-gm5b5\" (UID: \"4d35acb0-477c-436e-bb24-7e15e827a5af\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-gm5b5"
Oct 30 00:38:56.096522 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:38:56.096307    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xc7g8\" (UniqueName: \"kubernetes.io/projected/4d35acb0-477c-436e-bb24-7e15e827a5af-kube-api-access-xc7g8\") pod \"kuadrant-operator-controller-manager-57bb9bfff-gm5b5\" (UID: \"4d35acb0-477c-436e-bb24-7e15e827a5af\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-gm5b5"
Oct 30 00:38:56.197943 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:38:56.197900    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4d35acb0-477c-436e-bb24-7e15e827a5af-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-57bb9bfff-gm5b5\" (UID: \"4d35acb0-477c-436e-bb24-7e15e827a5af\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-gm5b5"
Oct 30 00:38:56.197943 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:38:56.197944    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xc7g8\" (UniqueName: \"kubernetes.io/projected/4d35acb0-477c-436e-bb24-7e15e827a5af-kube-api-access-xc7g8\") pod \"kuadrant-operator-controller-manager-57bb9bfff-gm5b5\" (UID: \"4d35acb0-477c-436e-bb24-7e15e827a5af\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-gm5b5"
Oct 30 00:38:56.198414 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:38:56.198360    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4d35acb0-477c-436e-bb24-7e15e827a5af-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-57bb9bfff-gm5b5\" (UID: \"4d35acb0-477c-436e-bb24-7e15e827a5af\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-gm5b5"
Oct 30 00:38:56.226423 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:38:56.226333    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xc7g8\" (UniqueName: \"kubernetes.io/projected/4d35acb0-477c-436e-bb24-7e15e827a5af-kube-api-access-xc7g8\") pod \"kuadrant-operator-controller-manager-57bb9bfff-gm5b5\" (UID: \"4d35acb0-477c-436e-bb24-7e15e827a5af\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-gm5b5"
Oct 30 00:38:56.283395 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:38:56.283342    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-gm5b5"
Oct 30 00:38:56.612652 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:38:56.612630    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-gm5b5"]
Oct 30 00:38:56.614426 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:38:56.614392    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d35acb0_477c_436e_bb24_7e15e827a5af.slice/crio-17f17ecfa55702e03cf07df1521e8dc503185a8752fed0442aa36b15a41ffe52 WatchSource:0}: Error finding container 17f17ecfa55702e03cf07df1521e8dc503185a8752fed0442aa36b15a41ffe52: Status 404 returned error can't find the container with id 17f17ecfa55702e03cf07df1521e8dc503185a8752fed0442aa36b15a41ffe52
Oct 30 00:38:57.356093 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:38:57.356046    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-gm5b5" event={"ID":"4d35acb0-477c-436e-bb24-7e15e827a5af","Type":"ContainerStarted","Data":"17f17ecfa55702e03cf07df1521e8dc503185a8752fed0442aa36b15a41ffe52"}
Oct 30 00:39:01.376292 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:39:01.376251    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-gm5b5" event={"ID":"4d35acb0-477c-436e-bb24-7e15e827a5af","Type":"ContainerStarted","Data":"42929532b7cbea74a7a29b0eaa8225607f945ce3c87371fdb38af2c7c1bb89e0"}
Oct 30 00:39:01.376751 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:39:01.376368    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-gm5b5"
Oct 30 00:39:01.451962 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:39:01.451900    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-gm5b5" podStartSLOduration=2.66040949 podStartE2EDuration="6.451884896s" totalImagesPullingTime="3.791475406s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:38:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:39:01.436995363 +0000 UTC m=+437.053428242" watchObservedRunningTime="2026-10-30 00:39:01.451884896 +0000 UTC m=+437.068317774"
Oct 30 00:39:12.383440 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:39:12.383406    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-gm5b5"
Oct 30 00:39:12.860458 ip-10-0-128-7 kubenswrapper[5002]: E1030 00:39:12.860421    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbjdq" containerName="extractor"
Oct 30 00:39:32.026743 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:39:32.026704    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7d8z4"]
Oct 30 00:39:32.031059 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:39:32.031036    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-7d8z4"
Oct 30 00:39:32.048779 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:39:32.048748    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Oct 30 00:39:32.064251 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:39:32.064224    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7d8z4"]
Oct 30 00:39:32.103423 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:39:32.103390    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7d8z4"]
Oct 30 00:39:32.105323 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:39:32.105300    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4z6l\" (UniqueName: \"kubernetes.io/projected/661ea448-7d8a-40ac-951b-383599d5d61f-kube-api-access-q4z6l\") pod \"limitador-limitador-7c9b87d574-7d8z4\" (UID: \"661ea448-7d8a-40ac-951b-383599d5d61f\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-7d8z4"
Oct 30 00:39:32.105398 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:39:32.105355    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/661ea448-7d8a-40ac-951b-383599d5d61f-config-file\") pod \"limitador-limitador-7c9b87d574-7d8z4\" (UID: \"661ea448-7d8a-40ac-951b-383599d5d61f\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-7d8z4"
Oct 30 00:39:32.205918 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:39:32.205876    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/661ea448-7d8a-40ac-951b-383599d5d61f-config-file\") pod \"limitador-limitador-7c9b87d574-7d8z4\" (UID: \"661ea448-7d8a-40ac-951b-383599d5d61f\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-7d8z4"
Oct 30 00:39:32.206100 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:39:32.205961    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4z6l\" (UniqueName: \"kubernetes.io/projected/661ea448-7d8a-40ac-951b-383599d5d61f-kube-api-access-q4z6l\") pod \"limitador-limitador-7c9b87d574-7d8z4\" (UID: \"661ea448-7d8a-40ac-951b-383599d5d61f\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-7d8z4"
Oct 30 00:39:32.206545 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:39:32.206521    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/661ea448-7d8a-40ac-951b-383599d5d61f-config-file\") pod \"limitador-limitador-7c9b87d574-7d8z4\" (UID: \"661ea448-7d8a-40ac-951b-383599d5d61f\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-7d8z4"
Oct 30 00:39:32.231100 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:39:32.231074    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4z6l\" (UniqueName: \"kubernetes.io/projected/661ea448-7d8a-40ac-951b-383599d5d61f-kube-api-access-q4z6l\") pod \"limitador-limitador-7c9b87d574-7d8z4\" (UID: \"661ea448-7d8a-40ac-951b-383599d5d61f\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-7d8z4"
Oct 30 00:39:32.343615 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:39:32.343535    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-7d8z4"
Oct 30 00:39:32.639758 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:39:32.639730    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7d8z4"]
Oct 30 00:39:32.640373 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:39:32.640349    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod661ea448_7d8a_40ac_951b_383599d5d61f.slice/crio-34530a0e1e674a5330aadd93efeec8b203c58b6c6b09ba55f458577e1fa7ca96 WatchSource:0}: Error finding container 34530a0e1e674a5330aadd93efeec8b203c58b6c6b09ba55f458577e1fa7ca96: Status 404 returned error can't find the container with id 34530a0e1e674a5330aadd93efeec8b203c58b6c6b09ba55f458577e1fa7ca96
Oct 30 00:39:33.515273 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:39:33.515239    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-7d8z4" event={"ID":"661ea448-7d8a-40ac-951b-383599d5d61f","Type":"ContainerStarted","Data":"34530a0e1e674a5330aadd93efeec8b203c58b6c6b09ba55f458577e1fa7ca96"}
Oct 30 00:39:34.521411 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:39:34.521359    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-7d8z4" event={"ID":"661ea448-7d8a-40ac-951b-383599d5d61f","Type":"ContainerStarted","Data":"8ca5be04c22195736ca4a807f3b13c18c1b520c5a4988553ddbfc4f84affcf14"}
Oct 30 00:39:34.521913 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:39:34.521442    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-7d8z4"
Oct 30 00:39:34.566332 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:39:34.566267    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7c9b87d574-7d8z4" podStartSLOduration=1.668726714 podStartE2EDuration="2.566253359s" totalImagesPullingTime="897.526645ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:39:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:39:34.565128422 +0000 UTC m=+470.181561301" watchObservedRunningTime="2026-10-30 00:39:34.566253359 +0000 UTC m=+470.182686238"
Oct 30 00:39:42.860403 ip-10-0-128-7 kubenswrapper[5002]: E1030 00:39:42.860340    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbjdq" containerName="extractor"
Oct 30 00:39:45.526190 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:39:45.526160    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-7d8z4"
Oct 30 00:40:12.860187 ip-10-0-128-7 kubenswrapper[5002]: E1030 00:40:12.860109    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbjdq" containerName="extractor"
Oct 30 00:40:42.860115 ip-10-0-128-7 kubenswrapper[5002]: E1030 00:40:42.860067    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbjdq" containerName="extractor"
Oct 30 00:40:49.640963 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:40:49.640928    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7d8z4"]
Oct 30 00:40:57.431884 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:40:57.431841    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-cgxg8"]
Oct 30 00:40:57.435655 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:40:57.435628    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-cgxg8"
Oct 30 00:40:57.454911 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:40:57.454882    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-cgxg8"]
Oct 30 00:40:57.541393 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:40:57.541344    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzpds\" (UniqueName: \"kubernetes.io/projected/9c8859ce-8c25-4422-a91e-d9b56618cca8-kube-api-access-dzpds\") pod \"llmisvc-controller-manager-6466c554d8-cgxg8\" (UID: \"9c8859ce-8c25-4422-a91e-d9b56618cca8\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-cgxg8"
Oct 30 00:40:57.541581 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:40:57.541400    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9c8859ce-8c25-4422-a91e-d9b56618cca8-cert\") pod \"llmisvc-controller-manager-6466c554d8-cgxg8\" (UID: \"9c8859ce-8c25-4422-a91e-d9b56618cca8\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-cgxg8"
Oct 30 00:40:57.642475 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:40:57.642428    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzpds\" (UniqueName: \"kubernetes.io/projected/9c8859ce-8c25-4422-a91e-d9b56618cca8-kube-api-access-dzpds\") pod \"llmisvc-controller-manager-6466c554d8-cgxg8\" (UID: \"9c8859ce-8c25-4422-a91e-d9b56618cca8\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-cgxg8"
Oct 30 00:40:57.642654 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:40:57.642487    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9c8859ce-8c25-4422-a91e-d9b56618cca8-cert\") pod \"llmisvc-controller-manager-6466c554d8-cgxg8\" (UID: \"9c8859ce-8c25-4422-a91e-d9b56618cca8\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-cgxg8"
Oct 30 00:40:57.644872 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:40:57.644847    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9c8859ce-8c25-4422-a91e-d9b56618cca8-cert\") pod \"llmisvc-controller-manager-6466c554d8-cgxg8\" (UID: \"9c8859ce-8c25-4422-a91e-d9b56618cca8\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-cgxg8"
Oct 30 00:40:57.667748 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:40:57.667721    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzpds\" (UniqueName: \"kubernetes.io/projected/9c8859ce-8c25-4422-a91e-d9b56618cca8-kube-api-access-dzpds\") pod \"llmisvc-controller-manager-6466c554d8-cgxg8\" (UID: \"9c8859ce-8c25-4422-a91e-d9b56618cca8\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-cgxg8"
Oct 30 00:40:57.748413 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:40:57.748363    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-cgxg8"
Oct 30 00:40:58.046794 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:40:58.046768    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-cgxg8"]
Oct 30 00:40:58.047898 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:40:58.047869    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c8859ce_8c25_4422_a91e_d9b56618cca8.slice/crio-8778dbd61ee3d6514870a7cbd8625e09c9ef41a6957d1152b033f2767d3ceaa9 WatchSource:0}: Error finding container 8778dbd61ee3d6514870a7cbd8625e09c9ef41a6957d1152b033f2767d3ceaa9: Status 404 returned error can't find the container with id 8778dbd61ee3d6514870a7cbd8625e09c9ef41a6957d1152b033f2767d3ceaa9
Oct 30 00:40:58.887940 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:40:58.887898    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-cgxg8" event={"ID":"9c8859ce-8c25-4422-a91e-d9b56618cca8","Type":"ContainerStarted","Data":"475cddf170a0bb8e098a1ee3af40b42a3a789161a2d4dafdbcc33706792112e2"}
Oct 30 00:40:58.888468 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:40:58.887956    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-cgxg8" event={"ID":"9c8859ce-8c25-4422-a91e-d9b56618cca8","Type":"ContainerStarted","Data":"8778dbd61ee3d6514870a7cbd8625e09c9ef41a6957d1152b033f2767d3ceaa9"}
Oct 30 00:40:58.888468 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:40:58.888036    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-cgxg8"
Oct 30 00:40:58.932726 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:40:58.932675    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-6466c554d8-cgxg8" podStartSLOduration=1.602853208 podStartE2EDuration="1.932661426s" totalImagesPullingTime="329.808218ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:40:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:40:58.932044562 +0000 UTC m=+554.548477457" watchObservedRunningTime="2026-10-30 00:40:58.932661426 +0000 UTC m=+554.549094343"
Oct 30 00:41:12.860084 ip-10-0-128-7 kubenswrapper[5002]: E1030 00:41:12.860044    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbjdq" containerName="extractor"
Oct 30 00:41:29.895339 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:41:29.895264    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-cgxg8"
Oct 30 00:41:29.968194 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:41:29.968161    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-n9mwb"]
Oct 30 00:41:29.968435 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:41:29.968370    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-n9mwb" podUID="6920db4e-c843-4524-a14e-e9e1064b0b9d" containerName="manager" containerID="cri-o://5fe971621f83ace6d08d7971437dca58fa2e3987f5643e1643f4c6f42691f02a" gracePeriod=30
Oct 30 00:41:30.222271 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:41:30.222245    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-n9mwb"
Oct 30 00:41:30.317811 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:41:30.317781    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6920db4e-c843-4524-a14e-e9e1064b0b9d-kube-api-access-6tv82\" (UniqueName: \"kubernetes.io/projected/6920db4e-c843-4524-a14e-e9e1064b0b9d-kube-api-access-6tv82\") pod \"6920db4e-c843-4524-a14e-e9e1064b0b9d\" (UID: \"6920db4e-c843-4524-a14e-e9e1064b0b9d\") "
Oct 30 00:41:30.317811 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:41:30.317815    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6920db4e-c843-4524-a14e-e9e1064b0b9d-cert\" (UniqueName: \"kubernetes.io/secret/6920db4e-c843-4524-a14e-e9e1064b0b9d-cert\") pod \"6920db4e-c843-4524-a14e-e9e1064b0b9d\" (UID: \"6920db4e-c843-4524-a14e-e9e1064b0b9d\") "
Oct 30 00:41:30.320020 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:41:30.319988    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6920db4e-c843-4524-a14e-e9e1064b0b9d-kube-api-access-6tv82" pod "6920db4e-c843-4524-a14e-e9e1064b0b9d" (UID: "6920db4e-c843-4524-a14e-e9e1064b0b9d"). InnerVolumeSpecName "kube-api-access-6tv82". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:41:30.320135 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:41:30.319992    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6920db4e-c843-4524-a14e-e9e1064b0b9d-cert" pod "6920db4e-c843-4524-a14e-e9e1064b0b9d" (UID: "6920db4e-c843-4524-a14e-e9e1064b0b9d"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:41:30.418628 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:41:30.418593    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6tv82\" (UniqueName: \"kubernetes.io/projected/6920db4e-c843-4524-a14e-e9e1064b0b9d-kube-api-access-6tv82\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:41:30.418628 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:41:30.418623    5002 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6920db4e-c843-4524-a14e-e9e1064b0b9d-cert\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:41:31.032553 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:41:31.032508    5002 generic.go:356] "Generic (PLEG): container finished" podID="6920db4e-c843-4524-a14e-e9e1064b0b9d" containerID="5fe971621f83ace6d08d7971437dca58fa2e3987f5643e1643f4c6f42691f02a" exitCode=0
Oct 30 00:41:31.032960 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:41:31.032561    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-n9mwb"
Oct 30 00:41:31.032960 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:41:31.032590    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-n9mwb" event={"ID":"6920db4e-c843-4524-a14e-e9e1064b0b9d","Type":"ContainerDied","Data":"5fe971621f83ace6d08d7971437dca58fa2e3987f5643e1643f4c6f42691f02a"}
Oct 30 00:41:31.032960 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:41:31.032646    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-n9mwb" event={"ID":"6920db4e-c843-4524-a14e-e9e1064b0b9d","Type":"ContainerDied","Data":"05a2b3567f6507c4b1cde01b40781d0d5e27afd6239219f27b13b22d783b3307"}
Oct 30 00:41:31.032960 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:41:31.032672    5002 scope.go:122] "RemoveContainer" containerID="5fe971621f83ace6d08d7971437dca58fa2e3987f5643e1643f4c6f42691f02a"
Oct 30 00:41:31.040977 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:41:31.040943    5002 scope.go:122] "RemoveContainer" containerID="5fe971621f83ace6d08d7971437dca58fa2e3987f5643e1643f4c6f42691f02a"
Oct 30 00:41:31.041225 ip-10-0-128-7 kubenswrapper[5002]: E1030 00:41:31.041207    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5fe971621f83ace6d08d7971437dca58fa2e3987f5643e1643f4c6f42691f02a\": container with ID starting with 5fe971621f83ace6d08d7971437dca58fa2e3987f5643e1643f4c6f42691f02a not found: ID does not exist" containerID="5fe971621f83ace6d08d7971437dca58fa2e3987f5643e1643f4c6f42691f02a"
Oct 30 00:41:31.041284 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:41:31.041238    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5fe971621f83ace6d08d7971437dca58fa2e3987f5643e1643f4c6f42691f02a"} err="failed to get container status \"5fe971621f83ace6d08d7971437dca58fa2e3987f5643e1643f4c6f42691f02a\": rpc error: code = NotFound desc = could not find container \"5fe971621f83ace6d08d7971437dca58fa2e3987f5643e1643f4c6f42691f02a\": container with ID starting with 5fe971621f83ace6d08d7971437dca58fa2e3987f5643e1643f4c6f42691f02a not found: ID does not exist"
Oct 30 00:41:31.093536 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:41:31.093502    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-n9mwb"]
Oct 30 00:41:31.107129 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:41:31.107100    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-n9mwb"]
Oct 30 00:41:32.493008 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:41:32.492978    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6920db4e-c843-4524-a14e-e9e1064b0b9d" path="/var/lib/kubelet/pods/6920db4e-c843-4524-a14e-e9e1064b0b9d/volumes"
Oct 30 00:41:42.859447 ip-10-0-128-7 kubenswrapper[5002]: E1030 00:41:42.859404    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbjdq" containerName="extractor"
Oct 30 00:41:44.545941 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:41:44.545916    5002 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-g7c2w_5dc56e7e-f3c1-43eb-a21b-8be6bb4706cf/console-operator/1.log"
Oct 30 00:41:44.548173 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:41:44.548150    5002 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-g7c2w_5dc56e7e-f3c1-43eb-a21b-8be6bb4706cf/console-operator/1.log"
Oct 30 00:41:48.744465 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:41:48.744426    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7d8z4"]
Oct 30 00:41:53.015563 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:41:53.015521    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7d8z4"]
Oct 30 00:41:58.395412 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:41:58.395360    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7d8z4"]
Oct 30 00:42:04.508630 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:42:04.508589    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7d8z4"]
Oct 30 00:42:12.860701 ip-10-0-128-7 kubenswrapper[5002]: E1030 00:42:12.860662    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbjdq" containerName="extractor"
Oct 30 00:42:14.262240 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:42:14.262206    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-4nbkc"]
Oct 30 00:42:14.262653 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:42:14.262592    5002 cpu_manager.go:414] "containerMap: removing container" podUID="6920db4e-c843-4524-a14e-e9e1064b0b9d" containerName="manager"
Oct 30 00:42:14.262653 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:42:14.262606    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6920db4e-c843-4524-a14e-e9e1064b0b9d" containerName="manager"
Oct 30 00:42:14.262722 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:42:14.262691    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="6920db4e-c843-4524-a14e-e9e1064b0b9d" containerName="manager"
Oct 30 00:42:14.265761 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:42:14.265746    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-4nbkc"
Oct 30 00:42:14.299001 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:42:14.298972    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-4nbkc"]
Oct 30 00:42:14.364936 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:42:14.364908    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Oct 30 00:42:14.365097 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:42:14.365071    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-gwncz\""
Oct 30 00:42:14.384559 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:42:14.384527    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6259aaf4-6f22-4781-8265-5ed5724b0f41-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-4nbkc\" (UID: \"6259aaf4-6f22-4781-8265-5ed5724b0f41\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-4nbkc"
Oct 30 00:42:14.384559 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:42:14.384558    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6259aaf4-6f22-4781-8265-5ed5724b0f41-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-4nbkc\" (UID: \"6259aaf4-6f22-4781-8265-5ed5724b0f41\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-4nbkc"
Oct 30 00:42:14.384758 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:42:14.384579    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6259aaf4-6f22-4781-8265-5ed5724b0f41-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-4nbkc\" (UID: \"6259aaf4-6f22-4781-8265-5ed5724b0f41\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-4nbkc"
Oct 30 00:42:14.384758 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:42:14.384658    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6259aaf4-6f22-4781-8265-5ed5724b0f41-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-4nbkc\" (UID: \"6259aaf4-6f22-4781-8265-5ed5724b0f41\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-4nbkc"
Oct 30 00:42:14.384829 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:42:14.384791    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6259aaf4-6f22-4781-8265-5ed5724b0f41-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-4nbkc\" (UID: \"6259aaf4-6f22-4781-8265-5ed5724b0f41\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-4nbkc"
Oct 30 00:42:14.384829 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:42:14.384814    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6259aaf4-6f22-4781-8265-5ed5724b0f41-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-4nbkc\" (UID: \"6259aaf4-6f22-4781-8265-5ed5724b0f41\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-4nbkc"
Oct 30 00:42:14.384906 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:42:14.384832    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvq7c\" (UniqueName: \"kubernetes.io/projected/6259aaf4-6f22-4781-8265-5ed5724b0f41-kube-api-access-qvq7c\") pod \"maas-default-gateway-openshift-default-7d9897b54c-4nbkc\" (UID: \"6259aaf4-6f22-4781-8265-5ed5724b0f41\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-4nbkc"
Oct 30 00:42:14.384906 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:42:14.384860    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6259aaf4-6f22-4781-8265-5ed5724b0f41-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-4nbkc\" (UID: \"6259aaf4-6f22-4781-8265-5ed5724b0f41\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-4nbkc"
Oct 30 00:42:14.384968 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:42:14.384905    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6259aaf4-6f22-4781-8265-5ed5724b0f41-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-4nbkc\" (UID: \"6259aaf4-6f22-4781-8265-5ed5724b0f41\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-4nbkc"
Oct 30 00:42:14.486297 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:42:14.486257    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6259aaf4-6f22-4781-8265-5ed5724b0f41-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-4nbkc\" (UID: \"6259aaf4-6f22-4781-8265-5ed5724b0f41\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-4nbkc"
Oct 30 00:42:14.486297 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:42:14.486297    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6259aaf4-6f22-4781-8265-5ed5724b0f41-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-4nbkc\" (UID: \"6259aaf4-6f22-4781-8265-5ed5724b0f41\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-4nbkc"
Oct 30 00:42:14.486559 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:42:14.486323    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6259aaf4-6f22-4781-8265-5ed5724b0f41-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-4nbkc\" (UID: \"6259aaf4-6f22-4781-8265-5ed5724b0f41\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-4nbkc"
Oct 30 00:42:14.486559 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:42:14.486341    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6259aaf4-6f22-4781-8265-5ed5724b0f41-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-4nbkc\" (UID: \"6259aaf4-6f22-4781-8265-5ed5724b0f41\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-4nbkc"
Oct 30 00:42:14.486559 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:42:14.486359    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6259aaf4-6f22-4781-8265-5ed5724b0f41-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-4nbkc\" (UID: \"6259aaf4-6f22-4781-8265-5ed5724b0f41\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-4nbkc"
Oct 30 00:42:14.486559 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:42:14.486406    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6259aaf4-6f22-4781-8265-5ed5724b0f41-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-4nbkc\" (UID: \"6259aaf4-6f22-4781-8265-5ed5724b0f41\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-4nbkc"
Oct 30 00:42:14.486559 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:42:14.486454    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6259aaf4-6f22-4781-8265-5ed5724b0f41-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-4nbkc\" (UID: \"6259aaf4-6f22-4781-8265-5ed5724b0f41\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-4nbkc"
Oct 30 00:42:14.486818 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:42:14.486589    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6259aaf4-6f22-4781-8265-5ed5724b0f41-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-4nbkc\" (UID: \"6259aaf4-6f22-4781-8265-5ed5724b0f41\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-4nbkc"
Oct 30 00:42:14.486818 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:42:14.486642    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvq7c\" (UniqueName: \"kubernetes.io/projected/6259aaf4-6f22-4781-8265-5ed5724b0f41-kube-api-access-qvq7c\") pod \"maas-default-gateway-openshift-default-7d9897b54c-4nbkc\" (UID: \"6259aaf4-6f22-4781-8265-5ed5724b0f41\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-4nbkc"
Oct 30 00:42:14.486818 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:42:14.486731    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6259aaf4-6f22-4781-8265-5ed5724b0f41-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-4nbkc\" (UID: \"6259aaf4-6f22-4781-8265-5ed5724b0f41\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-4nbkc"
Oct 30 00:42:14.486818 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:42:14.486796    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6259aaf4-6f22-4781-8265-5ed5724b0f41-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-4nbkc\" (UID: \"6259aaf4-6f22-4781-8265-5ed5724b0f41\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-4nbkc"
Oct 30 00:42:14.487330 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:42:14.487189    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6259aaf4-6f22-4781-8265-5ed5724b0f41-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-4nbkc\" (UID: \"6259aaf4-6f22-4781-8265-5ed5724b0f41\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-4nbkc"
Oct 30 00:42:14.487330 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:42:14.487285    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6259aaf4-6f22-4781-8265-5ed5724b0f41-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-4nbkc\" (UID: \"6259aaf4-6f22-4781-8265-5ed5724b0f41\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-4nbkc"
Oct 30 00:42:14.487330 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:42:14.487289    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6259aaf4-6f22-4781-8265-5ed5724b0f41-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-4nbkc\" (UID: \"6259aaf4-6f22-4781-8265-5ed5724b0f41\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-4nbkc"
Oct 30 00:42:14.489121 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:42:14.489093    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6259aaf4-6f22-4781-8265-5ed5724b0f41-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-4nbkc\" (UID: \"6259aaf4-6f22-4781-8265-5ed5724b0f41\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-4nbkc"
Oct 30 00:42:14.497044 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:42:14.496994    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6259aaf4-6f22-4781-8265-5ed5724b0f41-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-4nbkc\" (UID: \"6259aaf4-6f22-4781-8265-5ed5724b0f41\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-4nbkc"
Oct 30 00:42:14.511113 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:42:14.511088    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6259aaf4-6f22-4781-8265-5ed5724b0f41-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-4nbkc\" (UID: \"6259aaf4-6f22-4781-8265-5ed5724b0f41\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-4nbkc"
Oct 30 00:42:14.511253 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:42:14.511233    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvq7c\" (UniqueName: \"kubernetes.io/projected/6259aaf4-6f22-4781-8265-5ed5724b0f41-kube-api-access-qvq7c\") pod \"maas-default-gateway-openshift-default-7d9897b54c-4nbkc\" (UID: \"6259aaf4-6f22-4781-8265-5ed5724b0f41\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-4nbkc"
Oct 30 00:42:14.578216 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:42:14.578125    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-4nbkc"
Oct 30 00:42:14.892295 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:42:14.892266    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-4nbkc"]
Oct 30 00:42:14.892793 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:42:14.892771    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6259aaf4_6f22_4781_8265_5ed5724b0f41.slice/crio-e3abaed88e13bb32688003b7e123f99daa1358206701083028fa6a5bf4ff3d89 WatchSource:0}: Error finding container e3abaed88e13bb32688003b7e123f99daa1358206701083028fa6a5bf4ff3d89: Status 404 returned error can't find the container with id e3abaed88e13bb32688003b7e123f99daa1358206701083028fa6a5bf4ff3d89
Oct 30 00:42:15.231975 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:42:15.231939    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-4nbkc" event={"ID":"6259aaf4-6f22-4781-8265-5ed5724b0f41","Type":"ContainerStarted","Data":"e3abaed88e13bb32688003b7e123f99daa1358206701083028fa6a5bf4ff3d89"}
Oct 30 00:42:17.570990 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:42:17.570959    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 00:42:17.571286 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:42:17.571030    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 00:42:17.571286 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:42:17.571060    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 00:42:18.248987 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:42:18.248947    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-4nbkc" event={"ID":"6259aaf4-6f22-4781-8265-5ed5724b0f41","Type":"ContainerStarted","Data":"19e6cbf9915af52ce172cbc672957520edc7465be4c5830b184fa03720f63d6d"}
Oct 30 00:42:18.317484 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:42:18.317436    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-4nbkc" podStartSLOduration=1.642039339 podStartE2EDuration="4.317420688s" totalImagesPullingTime="2.675381349s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:42:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:42:18.317233057 +0000 UTC m=+633.933665953" watchObservedRunningTime="2026-10-30 00:42:18.317420688 +0000 UTC m=+633.933853567"
Oct 30 00:42:18.578776 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:42:18.578678    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-4nbkc"
Oct 30 00:42:18.580451 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:42:18.580415    5002 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-4nbkc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body=
Oct 30 00:42:18.580582 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:42:18.580469    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-4nbkc" podUID="6259aaf4-6f22-4781-8265-5ed5724b0f41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused"
Oct 30 00:42:19.579047 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:42:19.579005    5002 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-4nbkc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body=
Oct 30 00:42:19.579488 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:42:19.579066    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-4nbkc" podUID="6259aaf4-6f22-4781-8265-5ed5724b0f41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused"
Oct 30 00:42:20.579268 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:42:20.579222    5002 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-4nbkc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body=
Oct 30 00:42:20.579704 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:42:20.579287    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-4nbkc" podUID="6259aaf4-6f22-4781-8265-5ed5724b0f41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused"
Oct 30 00:42:21.579503 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:42:21.579468    5002 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-4nbkc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body=
Oct 30 00:42:21.579892 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:42:21.579535    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-4nbkc" podUID="6259aaf4-6f22-4781-8265-5ed5724b0f41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused"
Oct 30 00:42:22.578901 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:42:22.578863    5002 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-4nbkc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body=
Oct 30 00:42:22.579089 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:42:22.578919    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-4nbkc" podUID="6259aaf4-6f22-4781-8265-5ed5724b0f41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused"
Oct 30 00:42:23.579522 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:42:23.579486    5002 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-4nbkc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body=
Oct 30 00:42:23.579926 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:42:23.579538    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-4nbkc" podUID="6259aaf4-6f22-4781-8265-5ed5724b0f41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused"
Oct 30 00:42:24.579284 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:42:24.579249    5002 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-4nbkc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body=
Oct 30 00:42:24.579468 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:42:24.579301    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-4nbkc" podUID="6259aaf4-6f22-4781-8265-5ed5724b0f41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused"
Oct 30 00:42:25.844583 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:42:25.844546    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-4nbkc"
Oct 30 00:42:25.845095 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:42:25.845035    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-4nbkc"
Oct 30 00:42:25.845916 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:42:25.845897    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-4nbkc"
Oct 30 00:42:29.311184 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:42:29.311147    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-pdgh4"]
Oct 30 00:42:29.314677 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:42:29.314656    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-pdgh4"
Oct 30 00:42:29.347838 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:42:29.347813    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-pdgh4"]
Oct 30 00:42:29.440539 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:42:29.440502    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbddw\" (UniqueName: \"kubernetes.io/projected/c341e49a-5e59-414d-a432-e338bef60081-kube-api-access-hbddw\") pod \"maas-default-gateway-openshift-default-7d9897b54c-pdgh4\" (UID: \"c341e49a-5e59-414d-a432-e338bef60081\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-pdgh4"
Oct 30 00:42:29.440711 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:42:29.440555    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c341e49a-5e59-414d-a432-e338bef60081-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-pdgh4\" (UID: \"c341e49a-5e59-414d-a432-e338bef60081\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-pdgh4"
Oct 30 00:42:29.440711 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:42:29.440659    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c341e49a-5e59-414d-a432-e338bef60081-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-pdgh4\" (UID: \"c341e49a-5e59-414d-a432-e338bef60081\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-pdgh4"
Oct 30 00:42:29.440711 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:42:29.440700    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c341e49a-5e59-414d-a432-e338bef60081-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-pdgh4\" (UID: \"c341e49a-5e59-414d-a432-e338bef60081\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-pdgh4"
Oct 30 00:42:29.440849 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:42:29.440722    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c341e49a-5e59-414d-a432-e338bef60081-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-pdgh4\" (UID: \"c341e49a-5e59-414d-a432-e338bef60081\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-pdgh4"
Oct 30 00:42:29.440849 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:42:29.440801    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c341e49a-5e59-414d-a432-e338bef60081-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-pdgh4\" (UID: \"c341e49a-5e59-414d-a432-e338bef60081\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-pdgh4"
Oct 30 00:42:29.440924 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:42:29.440849    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c341e49a-5e59-414d-a432-e338bef60081-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-pdgh4\" (UID: \"c341e49a-5e59-414d-a432-e338bef60081\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-pdgh4"
Oct 30 00:42:29.440924 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:42:29.440906    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c341e49a-5e59-414d-a432-e338bef60081-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-pdgh4\" (UID: \"c341e49a-5e59-414d-a432-e338bef60081\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-pdgh4"
Oct 30 00:42:29.441011 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:42:29.440923    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c341e49a-5e59-414d-a432-e338bef60081-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-pdgh4\" (UID: \"c341e49a-5e59-414d-a432-e338bef60081\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-pdgh4"
Oct 30 00:42:29.541945 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:42:29.541902    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c341e49a-5e59-414d-a432-e338bef60081-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-pdgh4\" (UID: \"c341e49a-5e59-414d-a432-e338bef60081\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-pdgh4"
Oct 30 00:42:29.541945 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:42:29.541946    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c341e49a-5e59-414d-a432-e338bef60081-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-pdgh4\" (UID: \"c341e49a-5e59-414d-a432-e338bef60081\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-pdgh4"
Oct 30 00:42:29.542192 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:42:29.541994    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbddw\" (UniqueName: \"kubernetes.io/projected/c341e49a-5e59-414d-a432-e338bef60081-kube-api-access-hbddw\") pod \"maas-default-gateway-openshift-default-7d9897b54c-pdgh4\" (UID: \"c341e49a-5e59-414d-a432-e338bef60081\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-pdgh4"
Oct 30 00:42:29.542192 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:42:29.542026    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c341e49a-5e59-414d-a432-e338bef60081-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-pdgh4\" (UID: \"c341e49a-5e59-414d-a432-e338bef60081\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-pdgh4"
Oct 30 00:42:29.542192 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:42:29.542070    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c341e49a-5e59-414d-a432-e338bef60081-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-pdgh4\" (UID: \"c341e49a-5e59-414d-a432-e338bef60081\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-pdgh4"
Oct 30 00:42:29.542192 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:42:29.542106    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c341e49a-5e59-414d-a432-e338bef60081-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-pdgh4\" (UID: \"c341e49a-5e59-414d-a432-e338bef60081\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-pdgh4"
Oct 30 00:42:29.542192 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:42:29.542133    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c341e49a-5e59-414d-a432-e338bef60081-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-pdgh4\" (UID: \"c341e49a-5e59-414d-a432-e338bef60081\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-pdgh4"
Oct 30 00:42:29.542192 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:42:29.542165    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c341e49a-5e59-414d-a432-e338bef60081-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-pdgh4\" (UID: \"c341e49a-5e59-414d-a432-e338bef60081\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-pdgh4"
Oct 30 00:42:29.542192 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:42:29.542189    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c341e49a-5e59-414d-a432-e338bef60081-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-pdgh4\" (UID: \"c341e49a-5e59-414d-a432-e338bef60081\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-pdgh4"
Oct 30 00:42:29.542581 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:42:29.542442    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c341e49a-5e59-414d-a432-e338bef60081-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-pdgh4\" (UID: \"c341e49a-5e59-414d-a432-e338bef60081\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-pdgh4"
Oct 30 00:42:29.542581 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:42:29.542500    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c341e49a-5e59-414d-a432-e338bef60081-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-pdgh4\" (UID: \"c341e49a-5e59-414d-a432-e338bef60081\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-pdgh4"
Oct 30 00:42:29.542581 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:42:29.542558    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c341e49a-5e59-414d-a432-e338bef60081-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-pdgh4\" (UID: \"c341e49a-5e59-414d-a432-e338bef60081\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-pdgh4"
Oct 30 00:42:29.542732 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:42:29.542652    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c341e49a-5e59-414d-a432-e338bef60081-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-pdgh4\" (UID: \"c341e49a-5e59-414d-a432-e338bef60081\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-pdgh4"
Oct 30 00:42:29.543002 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:42:29.542982    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c341e49a-5e59-414d-a432-e338bef60081-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-pdgh4\" (UID: \"c341e49a-5e59-414d-a432-e338bef60081\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-pdgh4"
Oct 30 00:42:29.544220 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:42:29.544201    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c341e49a-5e59-414d-a432-e338bef60081-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-pdgh4\" (UID: \"c341e49a-5e59-414d-a432-e338bef60081\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-pdgh4"
Oct 30 00:42:29.544549 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:42:29.544531    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c341e49a-5e59-414d-a432-e338bef60081-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-pdgh4\" (UID: \"c341e49a-5e59-414d-a432-e338bef60081\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-pdgh4"
Oct 30 00:42:29.566994 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:42:29.566922    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c341e49a-5e59-414d-a432-e338bef60081-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-pdgh4\" (UID: \"c341e49a-5e59-414d-a432-e338bef60081\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-pdgh4"
Oct 30 00:42:29.567151 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:42:29.567128    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbddw\" (UniqueName: \"kubernetes.io/projected/c341e49a-5e59-414d-a432-e338bef60081-kube-api-access-hbddw\") pod \"maas-default-gateway-openshift-default-7d9897b54c-pdgh4\" (UID: \"c341e49a-5e59-414d-a432-e338bef60081\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-pdgh4"
Oct 30 00:42:29.628246 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:42:29.628211    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-pdgh4"
Oct 30 00:42:29.941010 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:42:29.940981    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-pdgh4"]
Oct 30 00:42:29.942189 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:42:29.942162    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc341e49a_5e59_414d_a432_e338bef60081.slice/crio-4116d2af40c81f3e932119a0115677e91c58ba484d72dafdcbbcd9e0fb61289b WatchSource:0}: Error finding container 4116d2af40c81f3e932119a0115677e91c58ba484d72dafdcbbcd9e0fb61289b: Status 404 returned error can't find the container with id 4116d2af40c81f3e932119a0115677e91c58ba484d72dafdcbbcd9e0fb61289b
Oct 30 00:42:29.945191 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:42:29.945160    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 00:42:29.945297 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:42:29.945234    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 00:42:29.945297 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:42:29.945283    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 00:42:30.327756 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:42:30.327671    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-pdgh4" event={"ID":"c341e49a-5e59-414d-a432-e338bef60081","Type":"ContainerStarted","Data":"303c4ebfb517312080c190dc792f72f93f3f64a919277292f246f0a72dc3beed"}
Oct 30 00:42:30.327756 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:42:30.327723    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-pdgh4" event={"ID":"c341e49a-5e59-414d-a432-e338bef60081","Type":"ContainerStarted","Data":"4116d2af40c81f3e932119a0115677e91c58ba484d72dafdcbbcd9e0fb61289b"}
Oct 30 00:42:30.395076 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:42:30.395022    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-pdgh4" podStartSLOduration=1.395005192 podStartE2EDuration="1.395005192s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:42:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:42:30.393955943 +0000 UTC m=+646.010388821" watchObservedRunningTime="2026-10-30 00:42:30.395005192 +0000 UTC m=+646.011438070"
Oct 30 00:42:30.629017 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:42:30.628930    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-pdgh4"
Oct 30 00:42:30.630634 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:42:30.630607    5002 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-pdgh4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body=
Oct 30 00:42:30.630768 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:42:30.630662    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-pdgh4" podUID="c341e49a-5e59-414d-a432-e338bef60081" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused"
Oct 30 00:42:31.628774 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:42:31.628733    5002 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-pdgh4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body=
Oct 30 00:42:31.629242 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:42:31.628793    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-pdgh4" podUID="c341e49a-5e59-414d-a432-e338bef60081" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused"
Oct 30 00:42:32.628940 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:42:32.628903    5002 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-pdgh4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body=
Oct 30 00:42:32.629336 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:42:32.628959    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-pdgh4" podUID="c341e49a-5e59-414d-a432-e338bef60081" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused"
Oct 30 00:42:33.629044 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:42:33.629002    5002 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-pdgh4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body=
Oct 30 00:42:33.629459 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:42:33.629060    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-pdgh4" podUID="c341e49a-5e59-414d-a432-e338bef60081" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused"
Oct 30 00:42:34.629501 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:42:34.629460    5002 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-pdgh4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body=
Oct 30 00:42:34.629939 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:42:34.629534    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-pdgh4" podUID="c341e49a-5e59-414d-a432-e338bef60081" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused"
Oct 30 00:42:35.628613 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:42:35.628581    5002 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-pdgh4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body=
Oct 30 00:42:35.628780 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:42:35.628640    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-pdgh4" podUID="c341e49a-5e59-414d-a432-e338bef60081" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused"
Oct 30 00:42:36.629031 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:42:36.628994    5002 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-pdgh4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body=
Oct 30 00:42:36.629434 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:42:36.629047    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-pdgh4" podUID="c341e49a-5e59-414d-a432-e338bef60081" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused"
Oct 30 00:42:38.179729 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:42:38.179688    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-pdgh4"
Oct 30 00:42:38.180207 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:42:38.180089    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-pdgh4"
Oct 30 00:42:38.180859 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:42:38.180838    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-pdgh4"
Oct 30 00:42:42.860433 ip-10-0-128-7 kubenswrapper[5002]: E1030 00:42:42.860396    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbjdq" containerName="extractor"
Oct 30 00:43:12.861618 ip-10-0-128-7 kubenswrapper[5002]: E1030 00:43:12.861531    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbjdq" containerName="extractor"
Oct 30 00:43:13.693518 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:43:13.693484    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7d8z4"]
Oct 30 00:43:18.129614 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:43:18.129580    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7d8z4"]
Oct 30 00:43:26.197937 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:43:26.197897    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-cefb27/simulator-w3-cefb27-kserve-6b669c9d9f-zx2vm"]
Oct 30 00:43:26.202735 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:43:26.202710    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-cefb27/simulator-w3-cefb27-kserve-6b669c9d9f-zx2vm"
Oct 30 00:43:26.220820 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:43:26.220786    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-cefb27/simulator-w3-cefb27-kserve-6b669c9d9f-zx2vm"]
Oct 30 00:43:26.233322 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:43:26.233289    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-cefb27\"/\"openshift-service-ca.crt\""
Oct 30 00:43:26.233322 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:43:26.233302    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-cefb27\"/\"kube-root-ca.crt\""
Oct 30 00:43:26.235995 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:43:26.235968    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-cefb27\"/\"default-dockercfg-jzvk5\""
Oct 30 00:43:26.236227 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:43:26.236204    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-cefb27\"/\"simulator-w3-cefb27-kserve-self-signed-certs\""
Oct 30 00:43:26.261161 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:43:26.261129    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4pvsr\" (UniqueName: \"kubernetes.io/projected/7121e81e-f903-42c0-a7a4-bc18d49194e9-kube-api-access-4pvsr\") pod \"simulator-w3-cefb27-kserve-6b669c9d9f-zx2vm\" (UID: \"7121e81e-f903-42c0-a7a4-bc18d49194e9\") " pod="e2e-models-e2e-worker-w3-cefb27/simulator-w3-cefb27-kserve-6b669c9d9f-zx2vm"
Oct 30 00:43:26.261326 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:43:26.261258    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7121e81e-f903-42c0-a7a4-bc18d49194e9-tls-certs\") pod \"simulator-w3-cefb27-kserve-6b669c9d9f-zx2vm\" (UID: \"7121e81e-f903-42c0-a7a4-bc18d49194e9\") " pod="e2e-models-e2e-worker-w3-cefb27/simulator-w3-cefb27-kserve-6b669c9d9f-zx2vm"
Oct 30 00:43:26.261326 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:43:26.261313    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7121e81e-f903-42c0-a7a4-bc18d49194e9-dshm\") pod \"simulator-w3-cefb27-kserve-6b669c9d9f-zx2vm\" (UID: \"7121e81e-f903-42c0-a7a4-bc18d49194e9\") " pod="e2e-models-e2e-worker-w3-cefb27/simulator-w3-cefb27-kserve-6b669c9d9f-zx2vm"
Oct 30 00:43:26.261482 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:43:26.261436    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7121e81e-f903-42c0-a7a4-bc18d49194e9-home\") pod \"simulator-w3-cefb27-kserve-6b669c9d9f-zx2vm\" (UID: \"7121e81e-f903-42c0-a7a4-bc18d49194e9\") " pod="e2e-models-e2e-worker-w3-cefb27/simulator-w3-cefb27-kserve-6b669c9d9f-zx2vm"
Oct 30 00:43:26.261540 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:43:26.261508    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7121e81e-f903-42c0-a7a4-bc18d49194e9-tmp-dir\") pod \"simulator-w3-cefb27-kserve-6b669c9d9f-zx2vm\" (UID: \"7121e81e-f903-42c0-a7a4-bc18d49194e9\") " pod="e2e-models-e2e-worker-w3-cefb27/simulator-w3-cefb27-kserve-6b669c9d9f-zx2vm"
Oct 30 00:43:26.261601 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:43:26.261570    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7121e81e-f903-42c0-a7a4-bc18d49194e9-model-cache\") pod \"simulator-w3-cefb27-kserve-6b669c9d9f-zx2vm\" (UID: \"7121e81e-f903-42c0-a7a4-bc18d49194e9\") " pod="e2e-models-e2e-worker-w3-cefb27/simulator-w3-cefb27-kserve-6b669c9d9f-zx2vm"
Oct 30 00:43:26.362527 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:43:26.362488    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7121e81e-f903-42c0-a7a4-bc18d49194e9-tmp-dir\") pod \"simulator-w3-cefb27-kserve-6b669c9d9f-zx2vm\" (UID: \"7121e81e-f903-42c0-a7a4-bc18d49194e9\") " pod="e2e-models-e2e-worker-w3-cefb27/simulator-w3-cefb27-kserve-6b669c9d9f-zx2vm"
Oct 30 00:43:26.362693 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:43:26.362539    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7121e81e-f903-42c0-a7a4-bc18d49194e9-model-cache\") pod \"simulator-w3-cefb27-kserve-6b669c9d9f-zx2vm\" (UID: \"7121e81e-f903-42c0-a7a4-bc18d49194e9\") " pod="e2e-models-e2e-worker-w3-cefb27/simulator-w3-cefb27-kserve-6b669c9d9f-zx2vm"
Oct 30 00:43:26.362693 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:43:26.362583    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pvsr\" (UniqueName: \"kubernetes.io/projected/7121e81e-f903-42c0-a7a4-bc18d49194e9-kube-api-access-4pvsr\") pod \"simulator-w3-cefb27-kserve-6b669c9d9f-zx2vm\" (UID: \"7121e81e-f903-42c0-a7a4-bc18d49194e9\") " pod="e2e-models-e2e-worker-w3-cefb27/simulator-w3-cefb27-kserve-6b669c9d9f-zx2vm"
Oct 30 00:43:26.362819 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:43:26.362733    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7121e81e-f903-42c0-a7a4-bc18d49194e9-tls-certs\") pod \"simulator-w3-cefb27-kserve-6b669c9d9f-zx2vm\" (UID: \"7121e81e-f903-42c0-a7a4-bc18d49194e9\") " pod="e2e-models-e2e-worker-w3-cefb27/simulator-w3-cefb27-kserve-6b669c9d9f-zx2vm"
Oct 30 00:43:26.362819 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:43:26.362782    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7121e81e-f903-42c0-a7a4-bc18d49194e9-dshm\") pod \"simulator-w3-cefb27-kserve-6b669c9d9f-zx2vm\" (UID: \"7121e81e-f903-42c0-a7a4-bc18d49194e9\") " pod="e2e-models-e2e-worker-w3-cefb27/simulator-w3-cefb27-kserve-6b669c9d9f-zx2vm"
Oct 30 00:43:26.362819 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:43:26.362807    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7121e81e-f903-42c0-a7a4-bc18d49194e9-home\") pod \"simulator-w3-cefb27-kserve-6b669c9d9f-zx2vm\" (UID: \"7121e81e-f903-42c0-a7a4-bc18d49194e9\") " pod="e2e-models-e2e-worker-w3-cefb27/simulator-w3-cefb27-kserve-6b669c9d9f-zx2vm"
Oct 30 00:43:26.362959 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:43:26.362930    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7121e81e-f903-42c0-a7a4-bc18d49194e9-tmp-dir\") pod \"simulator-w3-cefb27-kserve-6b669c9d9f-zx2vm\" (UID: \"7121e81e-f903-42c0-a7a4-bc18d49194e9\") " pod="e2e-models-e2e-worker-w3-cefb27/simulator-w3-cefb27-kserve-6b669c9d9f-zx2vm"
Oct 30 00:43:26.363010 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:43:26.362982    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7121e81e-f903-42c0-a7a4-bc18d49194e9-model-cache\") pod \"simulator-w3-cefb27-kserve-6b669c9d9f-zx2vm\" (UID: \"7121e81e-f903-42c0-a7a4-bc18d49194e9\") " pod="e2e-models-e2e-worker-w3-cefb27/simulator-w3-cefb27-kserve-6b669c9d9f-zx2vm"
Oct 30 00:43:26.363176 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:43:26.363152    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7121e81e-f903-42c0-a7a4-bc18d49194e9-home\") pod \"simulator-w3-cefb27-kserve-6b669c9d9f-zx2vm\" (UID: \"7121e81e-f903-42c0-a7a4-bc18d49194e9\") " pod="e2e-models-e2e-worker-w3-cefb27/simulator-w3-cefb27-kserve-6b669c9d9f-zx2vm"
Oct 30 00:43:26.365029 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:43:26.365004    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7121e81e-f903-42c0-a7a4-bc18d49194e9-dshm\") pod \"simulator-w3-cefb27-kserve-6b669c9d9f-zx2vm\" (UID: \"7121e81e-f903-42c0-a7a4-bc18d49194e9\") " pod="e2e-models-e2e-worker-w3-cefb27/simulator-w3-cefb27-kserve-6b669c9d9f-zx2vm"
Oct 30 00:43:26.365186 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:43:26.365168    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7121e81e-f903-42c0-a7a4-bc18d49194e9-tls-certs\") pod \"simulator-w3-cefb27-kserve-6b669c9d9f-zx2vm\" (UID: \"7121e81e-f903-42c0-a7a4-bc18d49194e9\") " pod="e2e-models-e2e-worker-w3-cefb27/simulator-w3-cefb27-kserve-6b669c9d9f-zx2vm"
Oct 30 00:43:26.390778 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:43:26.390744    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pvsr\" (UniqueName: \"kubernetes.io/projected/7121e81e-f903-42c0-a7a4-bc18d49194e9-kube-api-access-4pvsr\") pod \"simulator-w3-cefb27-kserve-6b669c9d9f-zx2vm\" (UID: \"7121e81e-f903-42c0-a7a4-bc18d49194e9\") " pod="e2e-models-e2e-worker-w3-cefb27/simulator-w3-cefb27-kserve-6b669c9d9f-zx2vm"
Oct 30 00:43:26.515119 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:43:26.515026    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-cefb27/simulator-w3-cefb27-kserve-6b669c9d9f-zx2vm"
Oct 30 00:43:26.822663 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:43:26.822634    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-cefb27/simulator-w3-cefb27-kserve-6b669c9d9f-zx2vm"]
Oct 30 00:43:26.823589 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:43:26.823559    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7121e81e_f903_42c0_a7a4_bc18d49194e9.slice/crio-f8c2389e04582bc971a71271a75ed7ebb2c225cfa128ac171a1a25c5df26f926 WatchSource:0}: Error finding container f8c2389e04582bc971a71271a75ed7ebb2c225cfa128ac171a1a25c5df26f926: Status 404 returned error can't find the container with id f8c2389e04582bc971a71271a75ed7ebb2c225cfa128ac171a1a25c5df26f926
Oct 30 00:43:26.825781 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:43:26.825757    5002 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 00:43:27.604998 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:43:27.604957    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-cefb27/simulator-w3-cefb27-kserve-6b669c9d9f-zx2vm" event={"ID":"7121e81e-f903-42c0-a7a4-bc18d49194e9","Type":"ContainerStarted","Data":"f8c2389e04582bc971a71271a75ed7ebb2c225cfa128ac171a1a25c5df26f926"}
Oct 30 00:43:28.612295 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:43:28.612256    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-cefb27/simulator-w3-cefb27-kserve-6b669c9d9f-zx2vm" event={"ID":"7121e81e-f903-42c0-a7a4-bc18d49194e9","Type":"ContainerStarted","Data":"a882c5a3943ec03c8f2fb3309811988b702de92703a7ca1d17eafa3b4a115adb"}
Oct 30 00:43:28.657627 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:43:28.657564    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-cefb27/simulator-w3-cefb27-kserve-6b669c9d9f-zx2vm" podStartSLOduration=1.466092094 podStartE2EDuration="2.657543724s" totalImagesPullingTime="1.19145163s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:43:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:43:28.656998104 +0000 UTC m=+704.273430983" watchObservedRunningTime="2026-10-30 00:43:28.657543724 +0000 UTC m=+704.273976604"
Oct 30 00:43:29.439364 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:43:29.439323    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7d8z4"]
Oct 30 00:43:29.515691 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:43:29.515620    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-cefb27/simulator-w3-cefb27-kserve-6b669c9d9f-zx2vm"
Oct 30 00:43:36.516285 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:43:36.516246    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-cefb27/simulator-w3-cefb27-kserve-6b669c9d9f-zx2vm"
Oct 30 00:43:36.529759 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:43:36.529731    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-cefb27/simulator-w3-cefb27-kserve-6b669c9d9f-zx2vm"
Oct 30 00:43:36.661769 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:43:36.661741    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-cefb27/simulator-w3-cefb27-kserve-6b669c9d9f-zx2vm"
Oct 30 00:43:41.113677 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:43:41.113643    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-6f4fba/simulator-w5-6f4fba-kserve-5bf8dfddd5-wcwzh"]
Oct 30 00:43:41.117778 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:43:41.117753    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-6f4fba/simulator-w5-6f4fba-kserve-5bf8dfddd5-wcwzh"
Oct 30 00:43:41.151961 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:43:41.151929    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-6f4fba\"/\"openshift-service-ca.crt\""
Oct 30 00:43:41.151961 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:43:41.151937    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-6f4fba\"/\"kube-root-ca.crt\""
Oct 30 00:43:41.153993 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:43:41.153968    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-6f4fba/simulator-w5-6f4fba-kserve-5bf8dfddd5-wcwzh"]
Oct 30 00:43:41.205323 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:43:41.205283    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a2c2f8f-649c-4ab6-af98-5d87df0e511f-tmp-dir\") pod \"simulator-w5-6f4fba-kserve-5bf8dfddd5-wcwzh\" (UID: \"3a2c2f8f-649c-4ab6-af98-5d87df0e511f\") " pod="e2e-models-e2e-worker-w5-6f4fba/simulator-w5-6f4fba-kserve-5bf8dfddd5-wcwzh"
Oct 30 00:43:41.205323 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:43:41.205320    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a2c2f8f-649c-4ab6-af98-5d87df0e511f-tls-certs\") pod \"simulator-w5-6f4fba-kserve-5bf8dfddd5-wcwzh\" (UID: \"3a2c2f8f-649c-4ab6-af98-5d87df0e511f\") " pod="e2e-models-e2e-worker-w5-6f4fba/simulator-w5-6f4fba-kserve-5bf8dfddd5-wcwzh"
Oct 30 00:43:41.205548 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:43:41.205336    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cj9tn\" (UniqueName: \"kubernetes.io/projected/3a2c2f8f-649c-4ab6-af98-5d87df0e511f-kube-api-access-cj9tn\") pod \"simulator-w5-6f4fba-kserve-5bf8dfddd5-wcwzh\" (UID: \"3a2c2f8f-649c-4ab6-af98-5d87df0e511f\") " pod="e2e-models-e2e-worker-w5-6f4fba/simulator-w5-6f4fba-kserve-5bf8dfddd5-wcwzh"
Oct 30 00:43:41.205548 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:43:41.205362    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a2c2f8f-649c-4ab6-af98-5d87df0e511f-model-cache\") pod \"simulator-w5-6f4fba-kserve-5bf8dfddd5-wcwzh\" (UID: \"3a2c2f8f-649c-4ab6-af98-5d87df0e511f\") " pod="e2e-models-e2e-worker-w5-6f4fba/simulator-w5-6f4fba-kserve-5bf8dfddd5-wcwzh"
Oct 30 00:43:41.205613 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:43:41.205560    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a2c2f8f-649c-4ab6-af98-5d87df0e511f-dshm\") pod \"simulator-w5-6f4fba-kserve-5bf8dfddd5-wcwzh\" (UID: \"3a2c2f8f-649c-4ab6-af98-5d87df0e511f\") " pod="e2e-models-e2e-worker-w5-6f4fba/simulator-w5-6f4fba-kserve-5bf8dfddd5-wcwzh"
Oct 30 00:43:41.205613 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:43:41.205597    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a2c2f8f-649c-4ab6-af98-5d87df0e511f-home\") pod \"simulator-w5-6f4fba-kserve-5bf8dfddd5-wcwzh\" (UID: \"3a2c2f8f-649c-4ab6-af98-5d87df0e511f\") " pod="e2e-models-e2e-worker-w5-6f4fba/simulator-w5-6f4fba-kserve-5bf8dfddd5-wcwzh"
Oct 30 00:43:41.209147 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:43:41.209125    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-6f4fba\"/\"simulator-w5-6f4fba-kserve-self-signed-certs\""
Oct 30 00:43:41.209147 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:43:41.209130    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-6f4fba\"/\"default-dockercfg-ffpkz\""
Oct 30 00:43:41.307143 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:43:41.307094    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a2c2f8f-649c-4ab6-af98-5d87df0e511f-tmp-dir\") pod \"simulator-w5-6f4fba-kserve-5bf8dfddd5-wcwzh\" (UID: \"3a2c2f8f-649c-4ab6-af98-5d87df0e511f\") " pod="e2e-models-e2e-worker-w5-6f4fba/simulator-w5-6f4fba-kserve-5bf8dfddd5-wcwzh"
Oct 30 00:43:41.307143 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:43:41.307142    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a2c2f8f-649c-4ab6-af98-5d87df0e511f-tls-certs\") pod \"simulator-w5-6f4fba-kserve-5bf8dfddd5-wcwzh\" (UID: \"3a2c2f8f-649c-4ab6-af98-5d87df0e511f\") " pod="e2e-models-e2e-worker-w5-6f4fba/simulator-w5-6f4fba-kserve-5bf8dfddd5-wcwzh"
Oct 30 00:43:41.307417 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:43:41.307172    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cj9tn\" (UniqueName: \"kubernetes.io/projected/3a2c2f8f-649c-4ab6-af98-5d87df0e511f-kube-api-access-cj9tn\") pod \"simulator-w5-6f4fba-kserve-5bf8dfddd5-wcwzh\" (UID: \"3a2c2f8f-649c-4ab6-af98-5d87df0e511f\") " pod="e2e-models-e2e-worker-w5-6f4fba/simulator-w5-6f4fba-kserve-5bf8dfddd5-wcwzh"
Oct 30 00:43:41.307417 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:43:41.307212    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a2c2f8f-649c-4ab6-af98-5d87df0e511f-model-cache\") pod \"simulator-w5-6f4fba-kserve-5bf8dfddd5-wcwzh\" (UID: \"3a2c2f8f-649c-4ab6-af98-5d87df0e511f\") " pod="e2e-models-e2e-worker-w5-6f4fba/simulator-w5-6f4fba-kserve-5bf8dfddd5-wcwzh"
Oct 30 00:43:41.307417 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:43:41.307268    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a2c2f8f-649c-4ab6-af98-5d87df0e511f-dshm\") pod \"simulator-w5-6f4fba-kserve-5bf8dfddd5-wcwzh\" (UID: \"3a2c2f8f-649c-4ab6-af98-5d87df0e511f\") " pod="e2e-models-e2e-worker-w5-6f4fba/simulator-w5-6f4fba-kserve-5bf8dfddd5-wcwzh"
Oct 30 00:43:41.307417 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:43:41.307305    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a2c2f8f-649c-4ab6-af98-5d87df0e511f-home\") pod \"simulator-w5-6f4fba-kserve-5bf8dfddd5-wcwzh\" (UID: \"3a2c2f8f-649c-4ab6-af98-5d87df0e511f\") " pod="e2e-models-e2e-worker-w5-6f4fba/simulator-w5-6f4fba-kserve-5bf8dfddd5-wcwzh"
Oct 30 00:43:41.308152 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:43:41.308101    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a2c2f8f-649c-4ab6-af98-5d87df0e511f-tmp-dir\") pod \"simulator-w5-6f4fba-kserve-5bf8dfddd5-wcwzh\" (UID: \"3a2c2f8f-649c-4ab6-af98-5d87df0e511f\") " pod="e2e-models-e2e-worker-w5-6f4fba/simulator-w5-6f4fba-kserve-5bf8dfddd5-wcwzh"
Oct 30 00:43:41.308424 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:43:41.308399    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a2c2f8f-649c-4ab6-af98-5d87df0e511f-home\") pod \"simulator-w5-6f4fba-kserve-5bf8dfddd5-wcwzh\" (UID: \"3a2c2f8f-649c-4ab6-af98-5d87df0e511f\") " pod="e2e-models-e2e-worker-w5-6f4fba/simulator-w5-6f4fba-kserve-5bf8dfddd5-wcwzh"
Oct 30 00:43:41.308711 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:43:41.308644    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a2c2f8f-649c-4ab6-af98-5d87df0e511f-model-cache\") pod \"simulator-w5-6f4fba-kserve-5bf8dfddd5-wcwzh\" (UID: \"3a2c2f8f-649c-4ab6-af98-5d87df0e511f\") " pod="e2e-models-e2e-worker-w5-6f4fba/simulator-w5-6f4fba-kserve-5bf8dfddd5-wcwzh"
Oct 30 00:43:41.310410 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:43:41.310357    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a2c2f8f-649c-4ab6-af98-5d87df0e511f-tls-certs\") pod \"simulator-w5-6f4fba-kserve-5bf8dfddd5-wcwzh\" (UID: \"3a2c2f8f-649c-4ab6-af98-5d87df0e511f\") " pod="e2e-models-e2e-worker-w5-6f4fba/simulator-w5-6f4fba-kserve-5bf8dfddd5-wcwzh"
Oct 30 00:43:41.310515 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:43:41.310442    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a2c2f8f-649c-4ab6-af98-5d87df0e511f-dshm\") pod \"simulator-w5-6f4fba-kserve-5bf8dfddd5-wcwzh\" (UID: \"3a2c2f8f-649c-4ab6-af98-5d87df0e511f\") " pod="e2e-models-e2e-worker-w5-6f4fba/simulator-w5-6f4fba-kserve-5bf8dfddd5-wcwzh"
Oct 30 00:43:41.339403 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:43:41.336350    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cj9tn\" (UniqueName: \"kubernetes.io/projected/3a2c2f8f-649c-4ab6-af98-5d87df0e511f-kube-api-access-cj9tn\") pod \"simulator-w5-6f4fba-kserve-5bf8dfddd5-wcwzh\" (UID: \"3a2c2f8f-649c-4ab6-af98-5d87df0e511f\") " pod="e2e-models-e2e-worker-w5-6f4fba/simulator-w5-6f4fba-kserve-5bf8dfddd5-wcwzh"
Oct 30 00:43:41.431535 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:43:41.431441    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-6f4fba/simulator-w5-6f4fba-kserve-5bf8dfddd5-wcwzh"
Oct 30 00:43:41.739468 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:43:41.739434    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-6f4fba/simulator-w5-6f4fba-kserve-5bf8dfddd5-wcwzh"]
Oct 30 00:43:41.741677 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:43:41.741649    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3a2c2f8f_649c_4ab6_af98_5d87df0e511f.slice/crio-d74f16da72402e61bf2badd1275a05d2b7ac28f968d73c9d2de86ca15aff92ff WatchSource:0}: Error finding container d74f16da72402e61bf2badd1275a05d2b7ac28f968d73c9d2de86ca15aff92ff: Status 404 returned error can't find the container with id d74f16da72402e61bf2badd1275a05d2b7ac28f968d73c9d2de86ca15aff92ff
Oct 30 00:43:42.679231 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:43:42.679191    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-6f4fba/simulator-w5-6f4fba-kserve-5bf8dfddd5-wcwzh" event={"ID":"3a2c2f8f-649c-4ab6-af98-5d87df0e511f","Type":"ContainerStarted","Data":"2f55590b96efc9d7861b27804d263d18b107d5ab17f5fb4bfd04c2d7db0f2477"}
Oct 30 00:43:42.679231 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:43:42.679238    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-6f4fba/simulator-w5-6f4fba-kserve-5bf8dfddd5-wcwzh" event={"ID":"3a2c2f8f-649c-4ab6-af98-5d87df0e511f","Type":"ContainerStarted","Data":"d74f16da72402e61bf2badd1275a05d2b7ac28f968d73c9d2de86ca15aff92ff"}
Oct 30 00:43:42.724739 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:43:42.724681    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-6f4fba/simulator-w5-6f4fba-kserve-5bf8dfddd5-wcwzh" podStartSLOduration=1.724665419 podStartE2EDuration="1.724665419s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:43:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:43:42.723502782 +0000 UTC m=+718.339935658" watchObservedRunningTime="2026-10-30 00:43:42.724665419 +0000 UTC m=+718.341098297"
Oct 30 00:43:42.868981 ip-10-0-128-7 kubenswrapper[5002]: E1030 00:43:42.868926    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbjdq" containerName="extractor"
Oct 30 00:43:43.431552 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:43:43.431512    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-6f4fba/simulator-w5-6f4fba-kserve-5bf8dfddd5-wcwzh"
Oct 30 00:43:44.364122 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:43:44.364081    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7d8z4"]
Oct 30 00:43:51.431906 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:43:51.431861    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-6f4fba/simulator-w5-6f4fba-kserve-5bf8dfddd5-wcwzh"
Oct 30 00:43:51.444564 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:43:51.444538    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-6f4fba/simulator-w5-6f4fba-kserve-5bf8dfddd5-wcwzh"
Oct 30 00:43:51.735883 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:43:51.735856    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-6f4fba/simulator-w5-6f4fba-kserve-5bf8dfddd5-wcwzh"
Oct 30 00:43:52.736512 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:43:52.736478    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-29be73/simulator-w1-29be73-kserve-9958d76db-fks2t"]
Oct 30 00:43:52.741269 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:43:52.741242    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-29be73/simulator-w1-29be73-kserve-9958d76db-fks2t"
Oct 30 00:43:52.761216 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:43:52.761185    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-29be73/simulator-w1-29be73-kserve-9958d76db-fks2t"]
Oct 30 00:43:52.772915 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:43:52.772877    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-29be73\"/\"openshift-service-ca.crt\""
Oct 30 00:43:52.772915 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:43:52.772906    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-29be73\"/\"simulator-w1-29be73-kserve-self-signed-certs\""
Oct 30 00:43:52.773125 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:43:52.772960    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-29be73\"/\"default-dockercfg-vwhx2\""
Oct 30 00:43:52.773125 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:43:52.773019    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-29be73\"/\"kube-root-ca.crt\""
Oct 30 00:43:52.813957 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:43:52.813926    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fbb3c806-8967-4d39-84c6-9dc022fc5cae-tls-certs\") pod \"simulator-w1-29be73-kserve-9958d76db-fks2t\" (UID: \"fbb3c806-8967-4d39-84c6-9dc022fc5cae\") " pod="e2e-models-e2e-worker-w1-29be73/simulator-w1-29be73-kserve-9958d76db-fks2t"
Oct 30 00:43:52.814141 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:43:52.813972    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fbb3c806-8967-4d39-84c6-9dc022fc5cae-tmp-dir\") pod \"simulator-w1-29be73-kserve-9958d76db-fks2t\" (UID: \"fbb3c806-8967-4d39-84c6-9dc022fc5cae\") " pod="e2e-models-e2e-worker-w1-29be73/simulator-w1-29be73-kserve-9958d76db-fks2t"
Oct 30 00:43:52.814141 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:43:52.813996    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fbb3c806-8967-4d39-84c6-9dc022fc5cae-dshm\") pod \"simulator-w1-29be73-kserve-9958d76db-fks2t\" (UID: \"fbb3c806-8967-4d39-84c6-9dc022fc5cae\") " pod="e2e-models-e2e-worker-w1-29be73/simulator-w1-29be73-kserve-9958d76db-fks2t"
Oct 30 00:43:52.814282 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:43:52.814161    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fbb3c806-8967-4d39-84c6-9dc022fc5cae-model-cache\") pod \"simulator-w1-29be73-kserve-9958d76db-fks2t\" (UID: \"fbb3c806-8967-4d39-84c6-9dc022fc5cae\") " pod="e2e-models-e2e-worker-w1-29be73/simulator-w1-29be73-kserve-9958d76db-fks2t"
Oct 30 00:43:52.814336 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:43:52.814281    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fbb3c806-8967-4d39-84c6-9dc022fc5cae-home\") pod \"simulator-w1-29be73-kserve-9958d76db-fks2t\" (UID: \"fbb3c806-8967-4d39-84c6-9dc022fc5cae\") " pod="e2e-models-e2e-worker-w1-29be73/simulator-w1-29be73-kserve-9958d76db-fks2t"
Oct 30 00:43:52.814419 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:43:52.814346    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2xb8\" (UniqueName: \"kubernetes.io/projected/fbb3c806-8967-4d39-84c6-9dc022fc5cae-kube-api-access-m2xb8\") pod \"simulator-w1-29be73-kserve-9958d76db-fks2t\" (UID: \"fbb3c806-8967-4d39-84c6-9dc022fc5cae\") " pod="e2e-models-e2e-worker-w1-29be73/simulator-w1-29be73-kserve-9958d76db-fks2t"
Oct 30 00:43:52.915494 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:43:52.915443    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fbb3c806-8967-4d39-84c6-9dc022fc5cae-home\") pod \"simulator-w1-29be73-kserve-9958d76db-fks2t\" (UID: \"fbb3c806-8967-4d39-84c6-9dc022fc5cae\") " pod="e2e-models-e2e-worker-w1-29be73/simulator-w1-29be73-kserve-9958d76db-fks2t"
Oct 30 00:43:52.915701 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:43:52.915554    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2xb8\" (UniqueName: \"kubernetes.io/projected/fbb3c806-8967-4d39-84c6-9dc022fc5cae-kube-api-access-m2xb8\") pod \"simulator-w1-29be73-kserve-9958d76db-fks2t\" (UID: \"fbb3c806-8967-4d39-84c6-9dc022fc5cae\") " pod="e2e-models-e2e-worker-w1-29be73/simulator-w1-29be73-kserve-9958d76db-fks2t"
Oct 30 00:43:52.915701 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:43:52.915594    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fbb3c806-8967-4d39-84c6-9dc022fc5cae-tls-certs\") pod \"simulator-w1-29be73-kserve-9958d76db-fks2t\" (UID: \"fbb3c806-8967-4d39-84c6-9dc022fc5cae\") " pod="e2e-models-e2e-worker-w1-29be73/simulator-w1-29be73-kserve-9958d76db-fks2t"
Oct 30 00:43:52.915701 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:43:52.915645    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fbb3c806-8967-4d39-84c6-9dc022fc5cae-tmp-dir\") pod \"simulator-w1-29be73-kserve-9958d76db-fks2t\" (UID: \"fbb3c806-8967-4d39-84c6-9dc022fc5cae\") " pod="e2e-models-e2e-worker-w1-29be73/simulator-w1-29be73-kserve-9958d76db-fks2t"
Oct 30 00:43:52.915701 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:43:52.915678    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fbb3c806-8967-4d39-84c6-9dc022fc5cae-dshm\") pod \"simulator-w1-29be73-kserve-9958d76db-fks2t\" (UID: \"fbb3c806-8967-4d39-84c6-9dc022fc5cae\") " pod="e2e-models-e2e-worker-w1-29be73/simulator-w1-29be73-kserve-9958d76db-fks2t"
Oct 30 00:43:52.915926 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:43:52.915714    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fbb3c806-8967-4d39-84c6-9dc022fc5cae-model-cache\") pod \"simulator-w1-29be73-kserve-9958d76db-fks2t\" (UID: \"fbb3c806-8967-4d39-84c6-9dc022fc5cae\") " pod="e2e-models-e2e-worker-w1-29be73/simulator-w1-29be73-kserve-9958d76db-fks2t"
Oct 30 00:43:52.916003 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:43:52.915969    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fbb3c806-8967-4d39-84c6-9dc022fc5cae-home\") pod \"simulator-w1-29be73-kserve-9958d76db-fks2t\" (UID: \"fbb3c806-8967-4d39-84c6-9dc022fc5cae\") " pod="e2e-models-e2e-worker-w1-29be73/simulator-w1-29be73-kserve-9958d76db-fks2t"
Oct 30 00:43:52.916119 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:43:52.916094    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fbb3c806-8967-4d39-84c6-9dc022fc5cae-model-cache\") pod \"simulator-w1-29be73-kserve-9958d76db-fks2t\" (UID: \"fbb3c806-8967-4d39-84c6-9dc022fc5cae\") " pod="e2e-models-e2e-worker-w1-29be73/simulator-w1-29be73-kserve-9958d76db-fks2t"
Oct 30 00:43:52.916300 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:43:52.916159    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fbb3c806-8967-4d39-84c6-9dc022fc5cae-tmp-dir\") pod \"simulator-w1-29be73-kserve-9958d76db-fks2t\" (UID: \"fbb3c806-8967-4d39-84c6-9dc022fc5cae\") " pod="e2e-models-e2e-worker-w1-29be73/simulator-w1-29be73-kserve-9958d76db-fks2t"
Oct 30 00:43:52.918315 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:43:52.918291    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fbb3c806-8967-4d39-84c6-9dc022fc5cae-tls-certs\") pod \"simulator-w1-29be73-kserve-9958d76db-fks2t\" (UID: \"fbb3c806-8967-4d39-84c6-9dc022fc5cae\") " pod="e2e-models-e2e-worker-w1-29be73/simulator-w1-29be73-kserve-9958d76db-fks2t"
Oct 30 00:43:52.918445 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:43:52.918356    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fbb3c806-8967-4d39-84c6-9dc022fc5cae-dshm\") pod \"simulator-w1-29be73-kserve-9958d76db-fks2t\" (UID: \"fbb3c806-8967-4d39-84c6-9dc022fc5cae\") " pod="e2e-models-e2e-worker-w1-29be73/simulator-w1-29be73-kserve-9958d76db-fks2t"
Oct 30 00:43:52.940603 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:43:52.940576    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2xb8\" (UniqueName: \"kubernetes.io/projected/fbb3c806-8967-4d39-84c6-9dc022fc5cae-kube-api-access-m2xb8\") pod \"simulator-w1-29be73-kserve-9958d76db-fks2t\" (UID: \"fbb3c806-8967-4d39-84c6-9dc022fc5cae\") " pod="e2e-models-e2e-worker-w1-29be73/simulator-w1-29be73-kserve-9958d76db-fks2t"
Oct 30 00:43:53.055784 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:43:53.055684    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-29be73/simulator-w1-29be73-kserve-9958d76db-fks2t"
Oct 30 00:43:53.360412 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:43:53.360367    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-29be73/simulator-w1-29be73-kserve-9958d76db-fks2t"]
Oct 30 00:43:53.361772 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:43:53.361750    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfbb3c806_8967_4d39_84c6_9dc022fc5cae.slice/crio-a1976db29b972df66fbb9ebe249adb3d56df9c803db3df1d95169879749c360c WatchSource:0}: Error finding container a1976db29b972df66fbb9ebe249adb3d56df9c803db3df1d95169879749c360c: Status 404 returned error can't find the container with id a1976db29b972df66fbb9ebe249adb3d56df9c803db3df1d95169879749c360c
Oct 30 00:43:53.740781 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:43:53.740739    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-29be73/simulator-w1-29be73-kserve-9958d76db-fks2t" event={"ID":"fbb3c806-8967-4d39-84c6-9dc022fc5cae","Type":"ContainerStarted","Data":"7755ab725791bbc2c78a4725e93d034cf9dc02a71ae74c42d2456ac2b24758b7"}
Oct 30 00:43:53.741181 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:43:53.740792    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-29be73/simulator-w1-29be73-kserve-9958d76db-fks2t" event={"ID":"fbb3c806-8967-4d39-84c6-9dc022fc5cae","Type":"ContainerStarted","Data":"a1976db29b972df66fbb9ebe249adb3d56df9c803db3df1d95169879749c360c"}
Oct 30 00:43:53.793333 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:43:53.793272    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-29be73/simulator-w1-29be73-kserve-9958d76db-fks2t" podStartSLOduration=1.79325601 podStartE2EDuration="1.79325601s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:43:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:43:53.791881977 +0000 UTC m=+729.408314858" watchObservedRunningTime="2026-10-30 00:43:53.79325601 +0000 UTC m=+729.409688890"
Oct 30 00:43:53.938947 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:43:53.938914    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-52650f/simulator-w0-52650f-kserve-58b64766c-94qxx"]
Oct 30 00:43:53.942696 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:43:53.942666    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-52650f/simulator-w0-52650f-kserve-58b64766c-94qxx"
Oct 30 00:43:53.962217 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:43:53.962188    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-52650f/simulator-w0-52650f-kserve-58b64766c-94qxx"]
Oct 30 00:43:53.968812 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:43:53.968785    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-52650f\"/\"default-dockercfg-7qbxd\""
Oct 30 00:43:53.968942 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:43:53.968871    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-52650f\"/\"simulator-w0-52650f-kserve-self-signed-certs\""
Oct 30 00:43:54.031055 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:43:54.030955    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-52650f\"/\"openshift-service-ca.crt\""
Oct 30 00:43:54.031218 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:43:54.031089    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-52650f\"/\"kube-root-ca.crt\""
Oct 30 00:43:54.032487 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:43:54.032463    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/659c1425-5b67-4d01-bfc2-f1ba56b9a59b-dshm\") pod \"simulator-w0-52650f-kserve-58b64766c-94qxx\" (UID: \"659c1425-5b67-4d01-bfc2-f1ba56b9a59b\") " pod="e2e-models-e2e-worker-w0-52650f/simulator-w0-52650f-kserve-58b64766c-94qxx"
Oct 30 00:43:54.032569 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:43:54.032529    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/659c1425-5b67-4d01-bfc2-f1ba56b9a59b-model-cache\") pod \"simulator-w0-52650f-kserve-58b64766c-94qxx\" (UID: \"659c1425-5b67-4d01-bfc2-f1ba56b9a59b\") " pod="e2e-models-e2e-worker-w0-52650f/simulator-w0-52650f-kserve-58b64766c-94qxx"
Oct 30 00:43:54.032680 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:43:54.032663    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/659c1425-5b67-4d01-bfc2-f1ba56b9a59b-tmp-dir\") pod \"simulator-w0-52650f-kserve-58b64766c-94qxx\" (UID: \"659c1425-5b67-4d01-bfc2-f1ba56b9a59b\") " pod="e2e-models-e2e-worker-w0-52650f/simulator-w0-52650f-kserve-58b64766c-94qxx"
Oct 30 00:43:54.032731 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:43:54.032691    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qtkz\" (UniqueName: \"kubernetes.io/projected/659c1425-5b67-4d01-bfc2-f1ba56b9a59b-kube-api-access-5qtkz\") pod \"simulator-w0-52650f-kserve-58b64766c-94qxx\" (UID: \"659c1425-5b67-4d01-bfc2-f1ba56b9a59b\") " pod="e2e-models-e2e-worker-w0-52650f/simulator-w0-52650f-kserve-58b64766c-94qxx"
Oct 30 00:43:54.032731 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:43:54.032722    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/659c1425-5b67-4d01-bfc2-f1ba56b9a59b-home\") pod \"simulator-w0-52650f-kserve-58b64766c-94qxx\" (UID: \"659c1425-5b67-4d01-bfc2-f1ba56b9a59b\") " pod="e2e-models-e2e-worker-w0-52650f/simulator-w0-52650f-kserve-58b64766c-94qxx"
Oct 30 00:43:54.032814 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:43:54.032798    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/659c1425-5b67-4d01-bfc2-f1ba56b9a59b-tls-certs\") pod \"simulator-w0-52650f-kserve-58b64766c-94qxx\" (UID: \"659c1425-5b67-4d01-bfc2-f1ba56b9a59b\") " pod="e2e-models-e2e-worker-w0-52650f/simulator-w0-52650f-kserve-58b64766c-94qxx"
Oct 30 00:43:54.056084 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:43:54.056053    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-29be73/simulator-w1-29be73-kserve-9958d76db-fks2t"
Oct 30 00:43:54.133366 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:43:54.133324    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/659c1425-5b67-4d01-bfc2-f1ba56b9a59b-dshm\") pod \"simulator-w0-52650f-kserve-58b64766c-94qxx\" (UID: \"659c1425-5b67-4d01-bfc2-f1ba56b9a59b\") " pod="e2e-models-e2e-worker-w0-52650f/simulator-w0-52650f-kserve-58b64766c-94qxx"
Oct 30 00:43:54.133681 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:43:54.133660    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/659c1425-5b67-4d01-bfc2-f1ba56b9a59b-model-cache\") pod \"simulator-w0-52650f-kserve-58b64766c-94qxx\" (UID: \"659c1425-5b67-4d01-bfc2-f1ba56b9a59b\") " pod="e2e-models-e2e-worker-w0-52650f/simulator-w0-52650f-kserve-58b64766c-94qxx"
Oct 30 00:43:54.133850 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:43:54.133834    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/659c1425-5b67-4d01-bfc2-f1ba56b9a59b-tmp-dir\") pod \"simulator-w0-52650f-kserve-58b64766c-94qxx\" (UID: \"659c1425-5b67-4d01-bfc2-f1ba56b9a59b\") " pod="e2e-models-e2e-worker-w0-52650f/simulator-w0-52650f-kserve-58b64766c-94qxx"
Oct 30 00:43:54.133963 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:43:54.133947    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qtkz\" (UniqueName: \"kubernetes.io/projected/659c1425-5b67-4d01-bfc2-f1ba56b9a59b-kube-api-access-5qtkz\") pod \"simulator-w0-52650f-kserve-58b64766c-94qxx\" (UID: \"659c1425-5b67-4d01-bfc2-f1ba56b9a59b\") " pod="e2e-models-e2e-worker-w0-52650f/simulator-w0-52650f-kserve-58b64766c-94qxx"
Oct 30 00:43:54.134080 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:43:54.134064    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/659c1425-5b67-4d01-bfc2-f1ba56b9a59b-home\") pod \"simulator-w0-52650f-kserve-58b64766c-94qxx\" (UID: \"659c1425-5b67-4d01-bfc2-f1ba56b9a59b\") " pod="e2e-models-e2e-worker-w0-52650f/simulator-w0-52650f-kserve-58b64766c-94qxx"
Oct 30 00:43:54.134185 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:43:54.134172    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/659c1425-5b67-4d01-bfc2-f1ba56b9a59b-tls-certs\") pod \"simulator-w0-52650f-kserve-58b64766c-94qxx\" (UID: \"659c1425-5b67-4d01-bfc2-f1ba56b9a59b\") " pod="e2e-models-e2e-worker-w0-52650f/simulator-w0-52650f-kserve-58b64766c-94qxx"
Oct 30 00:43:54.134273 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:43:54.134247    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/659c1425-5b67-4d01-bfc2-f1ba56b9a59b-model-cache\") pod \"simulator-w0-52650f-kserve-58b64766c-94qxx\" (UID: \"659c1425-5b67-4d01-bfc2-f1ba56b9a59b\") " pod="e2e-models-e2e-worker-w0-52650f/simulator-w0-52650f-kserve-58b64766c-94qxx"
Oct 30 00:43:54.134699 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:43:54.134669    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/659c1425-5b67-4d01-bfc2-f1ba56b9a59b-home\") pod \"simulator-w0-52650f-kserve-58b64766c-94qxx\" (UID: \"659c1425-5b67-4d01-bfc2-f1ba56b9a59b\") " pod="e2e-models-e2e-worker-w0-52650f/simulator-w0-52650f-kserve-58b64766c-94qxx"
Oct 30 00:43:54.134893 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:43:54.134868    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/659c1425-5b67-4d01-bfc2-f1ba56b9a59b-tmp-dir\") pod \"simulator-w0-52650f-kserve-58b64766c-94qxx\" (UID: \"659c1425-5b67-4d01-bfc2-f1ba56b9a59b\") " pod="e2e-models-e2e-worker-w0-52650f/simulator-w0-52650f-kserve-58b64766c-94qxx"
Oct 30 00:43:54.137310 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:43:54.137288    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/659c1425-5b67-4d01-bfc2-f1ba56b9a59b-dshm\") pod \"simulator-w0-52650f-kserve-58b64766c-94qxx\" (UID: \"659c1425-5b67-4d01-bfc2-f1ba56b9a59b\") " pod="e2e-models-e2e-worker-w0-52650f/simulator-w0-52650f-kserve-58b64766c-94qxx"
Oct 30 00:43:54.139333 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:43:54.139307    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/659c1425-5b67-4d01-bfc2-f1ba56b9a59b-tls-certs\") pod \"simulator-w0-52650f-kserve-58b64766c-94qxx\" (UID: \"659c1425-5b67-4d01-bfc2-f1ba56b9a59b\") " pod="e2e-models-e2e-worker-w0-52650f/simulator-w0-52650f-kserve-58b64766c-94qxx"
Oct 30 00:43:54.160345 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:43:54.160310    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qtkz\" (UniqueName: \"kubernetes.io/projected/659c1425-5b67-4d01-bfc2-f1ba56b9a59b-kube-api-access-5qtkz\") pod \"simulator-w0-52650f-kserve-58b64766c-94qxx\" (UID: \"659c1425-5b67-4d01-bfc2-f1ba56b9a59b\") " pod="e2e-models-e2e-worker-w0-52650f/simulator-w0-52650f-kserve-58b64766c-94qxx"
Oct 30 00:43:54.258916 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:43:54.258873    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-52650f/simulator-w0-52650f-kserve-58b64766c-94qxx"
Oct 30 00:43:54.564317 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:43:54.564285    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-52650f/simulator-w0-52650f-kserve-58b64766c-94qxx"]
Oct 30 00:43:54.565204 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:43:54.565178    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod659c1425_5b67_4d01_bfc2_f1ba56b9a59b.slice/crio-63fc43ac43e6fbfa4ccb102ca3cb2e40f5c7930c14a74374bf0312b7cc7e5375 WatchSource:0}: Error finding container 63fc43ac43e6fbfa4ccb102ca3cb2e40f5c7930c14a74374bf0312b7cc7e5375: Status 404 returned error can't find the container with id 63fc43ac43e6fbfa4ccb102ca3cb2e40f5c7930c14a74374bf0312b7cc7e5375
Oct 30 00:43:54.747969 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:43:54.747923    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-52650f/simulator-w0-52650f-kserve-58b64766c-94qxx" event={"ID":"659c1425-5b67-4d01-bfc2-f1ba56b9a59b","Type":"ContainerStarted","Data":"d1ec256d75183565f2cd8e1b1a95e9be6778c4671262e46758e7e8e46e506a3e"}
Oct 30 00:43:54.748393 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:43:54.747983    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-52650f/simulator-w0-52650f-kserve-58b64766c-94qxx" event={"ID":"659c1425-5b67-4d01-bfc2-f1ba56b9a59b","Type":"ContainerStarted","Data":"63fc43ac43e6fbfa4ccb102ca3cb2e40f5c7930c14a74374bf0312b7cc7e5375"}
Oct 30 00:43:54.792982 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:43:54.792927    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-52650f/simulator-w0-52650f-kserve-58b64766c-94qxx" podStartSLOduration=1.7929142489999998 podStartE2EDuration="1.792914249s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:43:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:43:54.791611822 +0000 UTC m=+730.408044698" watchObservedRunningTime="2026-10-30 00:43:54.792914249 +0000 UTC m=+730.409347128"
Oct 30 00:43:55.259672 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:43:55.259639    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-52650f/simulator-w0-52650f-kserve-58b64766c-94qxx"
Oct 30 00:43:57.734265 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:43:57.734228    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7d8z4"]
Oct 30 00:44:00.245163 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:00.245129    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-021d6d/simulator-w6-021d6d-kserve-6cc97fdf86-dqtpw"]
Oct 30 00:44:00.248569 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:00.248548    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-021d6d/simulator-w6-021d6d-kserve-6cc97fdf86-dqtpw"
Oct 30 00:44:00.265848 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:00.265813    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-021d6d\"/\"kube-root-ca.crt\""
Oct 30 00:44:00.268355 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:00.268305    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-021d6d\"/\"openshift-service-ca.crt\""
Oct 30 00:44:00.281961 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:00.281934    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-021d6d/simulator-w6-021d6d-kserve-6cc97fdf86-dqtpw"]
Oct 30 00:44:00.281961 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:00.281939    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-021d6d\"/\"simulator-w6-021d6d-kserve-self-signed-certs\""
Oct 30 00:44:00.282192 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:00.282178    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-021d6d\"/\"default-dockercfg-mtkh7\""
Oct 30 00:44:00.400524 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:00.400479    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/50586f76-4f3b-4e7a-83ac-6c573faee415-dshm\") pod \"simulator-w6-021d6d-kserve-6cc97fdf86-dqtpw\" (UID: \"50586f76-4f3b-4e7a-83ac-6c573faee415\") " pod="e2e-models-e2e-worker-w6-021d6d/simulator-w6-021d6d-kserve-6cc97fdf86-dqtpw"
Oct 30 00:44:00.400710 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:00.400550    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/50586f76-4f3b-4e7a-83ac-6c573faee415-home\") pod \"simulator-w6-021d6d-kserve-6cc97fdf86-dqtpw\" (UID: \"50586f76-4f3b-4e7a-83ac-6c573faee415\") " pod="e2e-models-e2e-worker-w6-021d6d/simulator-w6-021d6d-kserve-6cc97fdf86-dqtpw"
Oct 30 00:44:00.400760 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:00.400723    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50586f76-4f3b-4e7a-83ac-6c573faee415-model-cache\") pod \"simulator-w6-021d6d-kserve-6cc97fdf86-dqtpw\" (UID: \"50586f76-4f3b-4e7a-83ac-6c573faee415\") " pod="e2e-models-e2e-worker-w6-021d6d/simulator-w6-021d6d-kserve-6cc97fdf86-dqtpw"
Oct 30 00:44:00.400798 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:00.400784    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fslcn\" (UniqueName: \"kubernetes.io/projected/50586f76-4f3b-4e7a-83ac-6c573faee415-kube-api-access-fslcn\") pod \"simulator-w6-021d6d-kserve-6cc97fdf86-dqtpw\" (UID: \"50586f76-4f3b-4e7a-83ac-6c573faee415\") " pod="e2e-models-e2e-worker-w6-021d6d/simulator-w6-021d6d-kserve-6cc97fdf86-dqtpw"
Oct 30 00:44:00.400963 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:00.400936    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50586f76-4f3b-4e7a-83ac-6c573faee415-tmp-dir\") pod \"simulator-w6-021d6d-kserve-6cc97fdf86-dqtpw\" (UID: \"50586f76-4f3b-4e7a-83ac-6c573faee415\") " pod="e2e-models-e2e-worker-w6-021d6d/simulator-w6-021d6d-kserve-6cc97fdf86-dqtpw"
Oct 30 00:44:00.401015 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:00.401001    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50586f76-4f3b-4e7a-83ac-6c573faee415-tls-certs\") pod \"simulator-w6-021d6d-kserve-6cc97fdf86-dqtpw\" (UID: \"50586f76-4f3b-4e7a-83ac-6c573faee415\") " pod="e2e-models-e2e-worker-w6-021d6d/simulator-w6-021d6d-kserve-6cc97fdf86-dqtpw"
Oct 30 00:44:00.502296 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:00.502205    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50586f76-4f3b-4e7a-83ac-6c573faee415-model-cache\") pod \"simulator-w6-021d6d-kserve-6cc97fdf86-dqtpw\" (UID: \"50586f76-4f3b-4e7a-83ac-6c573faee415\") " pod="e2e-models-e2e-worker-w6-021d6d/simulator-w6-021d6d-kserve-6cc97fdf86-dqtpw"
Oct 30 00:44:00.502296 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:00.502260    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fslcn\" (UniqueName: \"kubernetes.io/projected/50586f76-4f3b-4e7a-83ac-6c573faee415-kube-api-access-fslcn\") pod \"simulator-w6-021d6d-kserve-6cc97fdf86-dqtpw\" (UID: \"50586f76-4f3b-4e7a-83ac-6c573faee415\") " pod="e2e-models-e2e-worker-w6-021d6d/simulator-w6-021d6d-kserve-6cc97fdf86-dqtpw"
Oct 30 00:44:00.502560 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:00.502307    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50586f76-4f3b-4e7a-83ac-6c573faee415-tmp-dir\") pod \"simulator-w6-021d6d-kserve-6cc97fdf86-dqtpw\" (UID: \"50586f76-4f3b-4e7a-83ac-6c573faee415\") " pod="e2e-models-e2e-worker-w6-021d6d/simulator-w6-021d6d-kserve-6cc97fdf86-dqtpw"
Oct 30 00:44:00.502560 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:00.502352    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50586f76-4f3b-4e7a-83ac-6c573faee415-tls-certs\") pod \"simulator-w6-021d6d-kserve-6cc97fdf86-dqtpw\" (UID: \"50586f76-4f3b-4e7a-83ac-6c573faee415\") " pod="e2e-models-e2e-worker-w6-021d6d/simulator-w6-021d6d-kserve-6cc97fdf86-dqtpw"
Oct 30 00:44:00.502560 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:00.502424    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/50586f76-4f3b-4e7a-83ac-6c573faee415-dshm\") pod \"simulator-w6-021d6d-kserve-6cc97fdf86-dqtpw\" (UID: \"50586f76-4f3b-4e7a-83ac-6c573faee415\") " pod="e2e-models-e2e-worker-w6-021d6d/simulator-w6-021d6d-kserve-6cc97fdf86-dqtpw"
Oct 30 00:44:00.502560 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:00.502463    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/50586f76-4f3b-4e7a-83ac-6c573faee415-home\") pod \"simulator-w6-021d6d-kserve-6cc97fdf86-dqtpw\" (UID: \"50586f76-4f3b-4e7a-83ac-6c573faee415\") " pod="e2e-models-e2e-worker-w6-021d6d/simulator-w6-021d6d-kserve-6cc97fdf86-dqtpw"
Oct 30 00:44:00.502729 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:00.502691    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50586f76-4f3b-4e7a-83ac-6c573faee415-model-cache\") pod \"simulator-w6-021d6d-kserve-6cc97fdf86-dqtpw\" (UID: \"50586f76-4f3b-4e7a-83ac-6c573faee415\") " pod="e2e-models-e2e-worker-w6-021d6d/simulator-w6-021d6d-kserve-6cc97fdf86-dqtpw"
Oct 30 00:44:00.502769 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:00.502755    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50586f76-4f3b-4e7a-83ac-6c573faee415-tmp-dir\") pod \"simulator-w6-021d6d-kserve-6cc97fdf86-dqtpw\" (UID: \"50586f76-4f3b-4e7a-83ac-6c573faee415\") " pod="e2e-models-e2e-worker-w6-021d6d/simulator-w6-021d6d-kserve-6cc97fdf86-dqtpw"
Oct 30 00:44:00.502865 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:00.502845    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/50586f76-4f3b-4e7a-83ac-6c573faee415-home\") pod \"simulator-w6-021d6d-kserve-6cc97fdf86-dqtpw\" (UID: \"50586f76-4f3b-4e7a-83ac-6c573faee415\") " pod="e2e-models-e2e-worker-w6-021d6d/simulator-w6-021d6d-kserve-6cc97fdf86-dqtpw"
Oct 30 00:44:00.504601 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:00.504577    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/50586f76-4f3b-4e7a-83ac-6c573faee415-dshm\") pod \"simulator-w6-021d6d-kserve-6cc97fdf86-dqtpw\" (UID: \"50586f76-4f3b-4e7a-83ac-6c573faee415\") " pod="e2e-models-e2e-worker-w6-021d6d/simulator-w6-021d6d-kserve-6cc97fdf86-dqtpw"
Oct 30 00:44:00.504982 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:00.504956    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50586f76-4f3b-4e7a-83ac-6c573faee415-tls-certs\") pod \"simulator-w6-021d6d-kserve-6cc97fdf86-dqtpw\" (UID: \"50586f76-4f3b-4e7a-83ac-6c573faee415\") " pod="e2e-models-e2e-worker-w6-021d6d/simulator-w6-021d6d-kserve-6cc97fdf86-dqtpw"
Oct 30 00:44:00.531787 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:00.531758    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fslcn\" (UniqueName: \"kubernetes.io/projected/50586f76-4f3b-4e7a-83ac-6c573faee415-kube-api-access-fslcn\") pod \"simulator-w6-021d6d-kserve-6cc97fdf86-dqtpw\" (UID: \"50586f76-4f3b-4e7a-83ac-6c573faee415\") " pod="e2e-models-e2e-worker-w6-021d6d/simulator-w6-021d6d-kserve-6cc97fdf86-dqtpw"
Oct 30 00:44:00.562968 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:00.562940    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-021d6d/simulator-w6-021d6d-kserve-6cc97fdf86-dqtpw"
Oct 30 00:44:00.877430 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:00.877395    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-021d6d/simulator-w6-021d6d-kserve-6cc97fdf86-dqtpw"]
Oct 30 00:44:00.877876 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:44:00.877845    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod50586f76_4f3b_4e7a_83ac_6c573faee415.slice/crio-0b328238f902a468aafb5896fea00bc84ca0dbf2641606f30109008020ffe2ab WatchSource:0}: Error finding container 0b328238f902a468aafb5896fea00bc84ca0dbf2641606f30109008020ffe2ab: Status 404 returned error can't find the container with id 0b328238f902a468aafb5896fea00bc84ca0dbf2641606f30109008020ffe2ab
Oct 30 00:44:01.804547 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:01.804508    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-021d6d/simulator-w6-021d6d-kserve-6cc97fdf86-dqtpw" event={"ID":"50586f76-4f3b-4e7a-83ac-6c573faee415","Type":"ContainerStarted","Data":"56454254bb774ad52b1f7fc34bf1f2e5fe8215deb7a28707d8a280186ca001d0"}
Oct 30 00:44:01.804963 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:01.804553    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-021d6d/simulator-w6-021d6d-kserve-6cc97fdf86-dqtpw" event={"ID":"50586f76-4f3b-4e7a-83ac-6c573faee415","Type":"ContainerStarted","Data":"0b328238f902a468aafb5896fea00bc84ca0dbf2641606f30109008020ffe2ab"}
Oct 30 00:44:01.858144 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:01.858090    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-021d6d/simulator-w6-021d6d-kserve-6cc97fdf86-dqtpw" podStartSLOduration=1.858076227 podStartE2EDuration="1.858076227s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:44:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:44:01.857540037 +0000 UTC m=+737.473972919" watchObservedRunningTime="2026-10-30 00:44:01.858076227 +0000 UTC m=+737.474509105"
Oct 30 00:44:02.564159 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:02.564122    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-021d6d/simulator-w6-021d6d-kserve-6cc97fdf86-dqtpw"
Oct 30 00:44:03.055939 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:03.055900    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-29be73/simulator-w1-29be73-kserve-9958d76db-fks2t"
Oct 30 00:44:03.068999 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:03.068970    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-29be73/simulator-w1-29be73-kserve-9958d76db-fks2t"
Oct 30 00:44:03.826414 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:03.826371    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-29be73/simulator-w1-29be73-kserve-9958d76db-fks2t"
Oct 30 00:44:04.259218 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:04.259175    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-52650f/simulator-w0-52650f-kserve-58b64766c-94qxx"
Oct 30 00:44:04.272312 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:04.272283    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-52650f/simulator-w0-52650f-kserve-58b64766c-94qxx"
Oct 30 00:44:04.830836 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:04.830807    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-52650f/simulator-w0-52650f-kserve-58b64766c-94qxx"
Oct 30 00:44:10.563160 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:10.563120    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-021d6d/simulator-w6-021d6d-kserve-6cc97fdf86-dqtpw"
Oct 30 00:44:10.575936 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:10.575910    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-021d6d/simulator-w6-021d6d-kserve-6cc97fdf86-dqtpw"
Oct 30 00:44:10.863876 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:10.863794    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-021d6d/simulator-w6-021d6d-kserve-6cc97fdf86-dqtpw"
Oct 30 00:44:12.862630 ip-10-0-128-7 kubenswrapper[5002]: E1030 00:44:12.862579    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbjdq" containerName="extractor"
Oct 30 00:44:17.242117 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:17.242084    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7d8z4"]
Oct 30 00:44:20.183447 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:20.183405    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-lrlct"]
Oct 30 00:44:20.190153 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:20.190126    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-lrlct"
Oct 30 00:44:20.207462 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:20.207438    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-021d6d-openshift-default-dockercfg-trf72\""
Oct 30 00:44:20.223951 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:20.223917    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-lrlct"]
Oct 30 00:44:20.294004 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:20.293956    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/099508e4-19ee-4e9b-93f3-0ef7a0d5b8b7-istio-envoy\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-lrlct\" (UID: \"099508e4-19ee-4e9b-93f3-0ef7a0d5b8b7\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-lrlct"
Oct 30 00:44:20.294210 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:20.294071    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/099508e4-19ee-4e9b-93f3-0ef7a0d5b8b7-istiod-ca-cert\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-lrlct\" (UID: \"099508e4-19ee-4e9b-93f3-0ef7a0d5b8b7\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-lrlct"
Oct 30 00:44:20.294210 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:20.294193    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/099508e4-19ee-4e9b-93f3-0ef7a0d5b8b7-credential-socket\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-lrlct\" (UID: \"099508e4-19ee-4e9b-93f3-0ef7a0d5b8b7\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-lrlct"
Oct 30 00:44:20.294343 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:20.294227    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvgzp\" (UniqueName: \"kubernetes.io/projected/099508e4-19ee-4e9b-93f3-0ef7a0d5b8b7-kube-api-access-pvgzp\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-lrlct\" (UID: \"099508e4-19ee-4e9b-93f3-0ef7a0d5b8b7\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-lrlct"
Oct 30 00:44:20.294428 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:20.294361    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/099508e4-19ee-4e9b-93f3-0ef7a0d5b8b7-workload-certs\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-lrlct\" (UID: \"099508e4-19ee-4e9b-93f3-0ef7a0d5b8b7\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-lrlct"
Oct 30 00:44:20.294490 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:20.294460    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/099508e4-19ee-4e9b-93f3-0ef7a0d5b8b7-istio-podinfo\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-lrlct\" (UID: \"099508e4-19ee-4e9b-93f3-0ef7a0d5b8b7\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-lrlct"
Oct 30 00:44:20.294545 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:20.294508    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/099508e4-19ee-4e9b-93f3-0ef7a0d5b8b7-istio-token\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-lrlct\" (UID: \"099508e4-19ee-4e9b-93f3-0ef7a0d5b8b7\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-lrlct"
Oct 30 00:44:20.294697 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:20.294668    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/099508e4-19ee-4e9b-93f3-0ef7a0d5b8b7-workload-socket\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-lrlct\" (UID: \"099508e4-19ee-4e9b-93f3-0ef7a0d5b8b7\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-lrlct"
Oct 30 00:44:20.295911 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:20.295884    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/099508e4-19ee-4e9b-93f3-0ef7a0d5b8b7-istio-data\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-lrlct\" (UID: \"099508e4-19ee-4e9b-93f3-0ef7a0d5b8b7\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-lrlct"
Oct 30 00:44:20.297980 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:20.297957    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-947lv"]
Oct 30 00:44:20.302899 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:20.302876    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-947lv"
Oct 30 00:44:20.323542 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:20.323519    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-1a11cf-openshift-default-dockercfg-j69mc\""
Oct 30 00:44:20.334123 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:20.334097    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-947lv"]
Oct 30 00:44:20.396759 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:20.396717    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htz6l\" (UniqueName: \"kubernetes.io/projected/ee192122-5e45-439e-93b4-eb2a0bf03260-kube-api-access-htz6l\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-947lv\" (UID: \"ee192122-5e45-439e-93b4-eb2a0bf03260\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-947lv"
Oct 30 00:44:20.396947 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:20.396777    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/099508e4-19ee-4e9b-93f3-0ef7a0d5b8b7-credential-socket\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-lrlct\" (UID: \"099508e4-19ee-4e9b-93f3-0ef7a0d5b8b7\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-lrlct"
Oct 30 00:44:20.396947 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:20.396851    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvgzp\" (UniqueName: \"kubernetes.io/projected/099508e4-19ee-4e9b-93f3-0ef7a0d5b8b7-kube-api-access-pvgzp\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-lrlct\" (UID: \"099508e4-19ee-4e9b-93f3-0ef7a0d5b8b7\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-lrlct"
Oct 30 00:44:20.396947 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:20.396915    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/099508e4-19ee-4e9b-93f3-0ef7a0d5b8b7-workload-certs\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-lrlct\" (UID: \"099508e4-19ee-4e9b-93f3-0ef7a0d5b8b7\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-lrlct"
Oct 30 00:44:20.397114 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:20.396953    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee192122-5e45-439e-93b4-eb2a0bf03260-istiod-ca-cert\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-947lv\" (UID: \"ee192122-5e45-439e-93b4-eb2a0bf03260\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-947lv"
Oct 30 00:44:20.397114 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:20.396984    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/099508e4-19ee-4e9b-93f3-0ef7a0d5b8b7-istio-podinfo\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-lrlct\" (UID: \"099508e4-19ee-4e9b-93f3-0ef7a0d5b8b7\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-lrlct"
Oct 30 00:44:20.397114 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:20.397082    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee192122-5e45-439e-93b4-eb2a0bf03260-istio-podinfo\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-947lv\" (UID: \"ee192122-5e45-439e-93b4-eb2a0bf03260\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-947lv"
Oct 30 00:44:20.397259 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:20.397133    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/099508e4-19ee-4e9b-93f3-0ef7a0d5b8b7-istio-token\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-lrlct\" (UID: \"099508e4-19ee-4e9b-93f3-0ef7a0d5b8b7\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-lrlct"
Oct 30 00:44:20.397259 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:20.397167    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee192122-5e45-439e-93b4-eb2a0bf03260-istio-data\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-947lv\" (UID: \"ee192122-5e45-439e-93b4-eb2a0bf03260\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-947lv"
Oct 30 00:44:20.397259 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:20.397203    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee192122-5e45-439e-93b4-eb2a0bf03260-workload-socket\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-947lv\" (UID: \"ee192122-5e45-439e-93b4-eb2a0bf03260\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-947lv"
Oct 30 00:44:20.397259 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:20.397218    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/099508e4-19ee-4e9b-93f3-0ef7a0d5b8b7-credential-socket\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-lrlct\" (UID: \"099508e4-19ee-4e9b-93f3-0ef7a0d5b8b7\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-lrlct"
Oct 30 00:44:20.397483 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:20.397266    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/099508e4-19ee-4e9b-93f3-0ef7a0d5b8b7-workload-certs\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-lrlct\" (UID: \"099508e4-19ee-4e9b-93f3-0ef7a0d5b8b7\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-lrlct"
Oct 30 00:44:20.397483 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:20.397309    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee192122-5e45-439e-93b4-eb2a0bf03260-workload-certs\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-947lv\" (UID: \"ee192122-5e45-439e-93b4-eb2a0bf03260\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-947lv"
Oct 30 00:44:20.397483 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:20.397341    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/099508e4-19ee-4e9b-93f3-0ef7a0d5b8b7-workload-socket\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-lrlct\" (UID: \"099508e4-19ee-4e9b-93f3-0ef7a0d5b8b7\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-lrlct"
Oct 30 00:44:20.397483 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:20.397443    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/099508e4-19ee-4e9b-93f3-0ef7a0d5b8b7-istio-data\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-lrlct\" (UID: \"099508e4-19ee-4e9b-93f3-0ef7a0d5b8b7\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-lrlct"
Oct 30 00:44:20.397483 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:20.397478    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee192122-5e45-439e-93b4-eb2a0bf03260-credential-socket\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-947lv\" (UID: \"ee192122-5e45-439e-93b4-eb2a0bf03260\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-947lv"
Oct 30 00:44:20.397742 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:20.397508    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee192122-5e45-439e-93b4-eb2a0bf03260-istio-token\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-947lv\" (UID: \"ee192122-5e45-439e-93b4-eb2a0bf03260\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-947lv"
Oct 30 00:44:20.397742 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:20.397586    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/099508e4-19ee-4e9b-93f3-0ef7a0d5b8b7-workload-socket\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-lrlct\" (UID: \"099508e4-19ee-4e9b-93f3-0ef7a0d5b8b7\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-lrlct"
Oct 30 00:44:20.397742 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:20.397634    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/099508e4-19ee-4e9b-93f3-0ef7a0d5b8b7-istio-envoy\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-lrlct\" (UID: \"099508e4-19ee-4e9b-93f3-0ef7a0d5b8b7\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-lrlct"
Oct 30 00:44:20.397742 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:20.397667    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/099508e4-19ee-4e9b-93f3-0ef7a0d5b8b7-istiod-ca-cert\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-lrlct\" (UID: \"099508e4-19ee-4e9b-93f3-0ef7a0d5b8b7\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-lrlct"
Oct 30 00:44:20.397742 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:20.397702    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee192122-5e45-439e-93b4-eb2a0bf03260-istio-envoy\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-947lv\" (UID: \"ee192122-5e45-439e-93b4-eb2a0bf03260\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-947lv"
Oct 30 00:44:20.397920 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:20.397783    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/099508e4-19ee-4e9b-93f3-0ef7a0d5b8b7-istio-data\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-lrlct\" (UID: \"099508e4-19ee-4e9b-93f3-0ef7a0d5b8b7\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-lrlct"
Oct 30 00:44:20.398186 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:20.398163    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/099508e4-19ee-4e9b-93f3-0ef7a0d5b8b7-istiod-ca-cert\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-lrlct\" (UID: \"099508e4-19ee-4e9b-93f3-0ef7a0d5b8b7\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-lrlct"
Oct 30 00:44:20.399644 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:20.399618    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/099508e4-19ee-4e9b-93f3-0ef7a0d5b8b7-istio-podinfo\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-lrlct\" (UID: \"099508e4-19ee-4e9b-93f3-0ef7a0d5b8b7\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-lrlct"
Oct 30 00:44:20.399736 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:20.399690    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/099508e4-19ee-4e9b-93f3-0ef7a0d5b8b7-istio-envoy\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-lrlct\" (UID: \"099508e4-19ee-4e9b-93f3-0ef7a0d5b8b7\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-lrlct"
Oct 30 00:44:20.421806 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:20.421783    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/099508e4-19ee-4e9b-93f3-0ef7a0d5b8b7-istio-token\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-lrlct\" (UID: \"099508e4-19ee-4e9b-93f3-0ef7a0d5b8b7\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-lrlct"
Oct 30 00:44:20.422045 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:20.422028    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvgzp\" (UniqueName: \"kubernetes.io/projected/099508e4-19ee-4e9b-93f3-0ef7a0d5b8b7-kube-api-access-pvgzp\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-lrlct\" (UID: \"099508e4-19ee-4e9b-93f3-0ef7a0d5b8b7\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-lrlct"
Oct 30 00:44:20.498971 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:20.498882    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee192122-5e45-439e-93b4-eb2a0bf03260-workload-certs\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-947lv\" (UID: \"ee192122-5e45-439e-93b4-eb2a0bf03260\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-947lv"
Oct 30 00:44:20.499127 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:20.499048    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee192122-5e45-439e-93b4-eb2a0bf03260-credential-socket\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-947lv\" (UID: \"ee192122-5e45-439e-93b4-eb2a0bf03260\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-947lv"
Oct 30 00:44:20.499127 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:20.499088    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee192122-5e45-439e-93b4-eb2a0bf03260-istio-token\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-947lv\" (UID: \"ee192122-5e45-439e-93b4-eb2a0bf03260\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-947lv"
Oct 30 00:44:20.499248 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:20.499127    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee192122-5e45-439e-93b4-eb2a0bf03260-istio-envoy\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-947lv\" (UID: \"ee192122-5e45-439e-93b4-eb2a0bf03260\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-947lv"
Oct 30 00:44:20.499248 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:20.499178    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htz6l\" (UniqueName: \"kubernetes.io/projected/ee192122-5e45-439e-93b4-eb2a0bf03260-kube-api-access-htz6l\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-947lv\" (UID: \"ee192122-5e45-439e-93b4-eb2a0bf03260\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-947lv"
Oct 30 00:44:20.499351 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:20.499259    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee192122-5e45-439e-93b4-eb2a0bf03260-istiod-ca-cert\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-947lv\" (UID: \"ee192122-5e45-439e-93b4-eb2a0bf03260\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-947lv"
Oct 30 00:44:20.499351 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:20.499303    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee192122-5e45-439e-93b4-eb2a0bf03260-istio-podinfo\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-947lv\" (UID: \"ee192122-5e45-439e-93b4-eb2a0bf03260\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-947lv"
Oct 30 00:44:20.499351 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:20.499344    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee192122-5e45-439e-93b4-eb2a0bf03260-istio-data\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-947lv\" (UID: \"ee192122-5e45-439e-93b4-eb2a0bf03260\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-947lv"
Oct 30 00:44:20.499536 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:20.499406    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee192122-5e45-439e-93b4-eb2a0bf03260-workload-socket\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-947lv\" (UID: \"ee192122-5e45-439e-93b4-eb2a0bf03260\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-947lv"
Oct 30 00:44:20.500122 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:20.500092    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee192122-5e45-439e-93b4-eb2a0bf03260-workload-certs\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-947lv\" (UID: \"ee192122-5e45-439e-93b4-eb2a0bf03260\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-947lv"
Oct 30 00:44:20.500258 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:20.500155    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee192122-5e45-439e-93b4-eb2a0bf03260-credential-socket\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-947lv\" (UID: \"ee192122-5e45-439e-93b4-eb2a0bf03260\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-947lv"
Oct 30 00:44:20.500358 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:20.500337    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee192122-5e45-439e-93b4-eb2a0bf03260-workload-socket\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-947lv\" (UID: \"ee192122-5e45-439e-93b4-eb2a0bf03260\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-947lv"
Oct 30 00:44:20.500461 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:20.500402    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee192122-5e45-439e-93b4-eb2a0bf03260-istio-data\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-947lv\" (UID: \"ee192122-5e45-439e-93b4-eb2a0bf03260\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-947lv"
Oct 30 00:44:20.500461 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:20.500435    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee192122-5e45-439e-93b4-eb2a0bf03260-istiod-ca-cert\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-947lv\" (UID: \"ee192122-5e45-439e-93b4-eb2a0bf03260\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-947lv"
Oct 30 00:44:20.501800 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:20.501778    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee192122-5e45-439e-93b4-eb2a0bf03260-istio-envoy\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-947lv\" (UID: \"ee192122-5e45-439e-93b4-eb2a0bf03260\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-947lv"
Oct 30 00:44:20.501935 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:20.501912    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee192122-5e45-439e-93b4-eb2a0bf03260-istio-podinfo\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-947lv\" (UID: \"ee192122-5e45-439e-93b4-eb2a0bf03260\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-947lv"
Oct 30 00:44:20.503797 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:20.503776    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-lrlct"
Oct 30 00:44:20.525087 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:20.525051    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-htz6l\" (UniqueName: \"kubernetes.io/projected/ee192122-5e45-439e-93b4-eb2a0bf03260-kube-api-access-htz6l\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-947lv\" (UID: \"ee192122-5e45-439e-93b4-eb2a0bf03260\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-947lv"
Oct 30 00:44:20.526018 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:20.525998    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee192122-5e45-439e-93b4-eb2a0bf03260-istio-token\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-947lv\" (UID: \"ee192122-5e45-439e-93b4-eb2a0bf03260\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-947lv"
Oct 30 00:44:20.620071 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:20.620039    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-947lv"
Oct 30 00:44:20.821499 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:20.821470    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-lrlct"]
Oct 30 00:44:20.822143 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:44:20.822114    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod099508e4_19ee_4e9b_93f3_0ef7a0d5b8b7.slice/crio-d46b75111b0a4ae94b0aa66ed1efc9a703e2543f928bd353e9a3a3987dcdb5a4 WatchSource:0}: Error finding container d46b75111b0a4ae94b0aa66ed1efc9a703e2543f928bd353e9a3a3987dcdb5a4: Status 404 returned error can't find the container with id d46b75111b0a4ae94b0aa66ed1efc9a703e2543f928bd353e9a3a3987dcdb5a4
Oct 30 00:44:20.824688 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:20.824658    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 00:44:20.824790 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:20.824724    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 00:44:20.824790 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:20.824766    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 00:44:20.903219 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:20.902773    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-lrlct" event={"ID":"099508e4-19ee-4e9b-93f3-0ef7a0d5b8b7","Type":"ContainerStarted","Data":"db3fd84c86046aa65d14b2238cf927ef241c031ac33aef5a027e6c3afbafed7b"}
Oct 30 00:44:20.903219 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:20.902846    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-lrlct" event={"ID":"099508e4-19ee-4e9b-93f3-0ef7a0d5b8b7","Type":"ContainerStarted","Data":"d46b75111b0a4ae94b0aa66ed1efc9a703e2543f928bd353e9a3a3987dcdb5a4"}
Oct 30 00:44:20.925824 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:20.925795    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-947lv"]
Oct 30 00:44:20.925944 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:44:20.925793    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee192122_5e45_439e_93b4_eb2a0bf03260.slice/crio-f1783e53b99a5291479a6c8dc84912812c677c7a3fab9b3c3ac62f86a43a5036 WatchSource:0}: Error finding container f1783e53b99a5291479a6c8dc84912812c677c7a3fab9b3c3ac62f86a43a5036: Status 404 returned error can't find the container with id f1783e53b99a5291479a6c8dc84912812c677c7a3fab9b3c3ac62f86a43a5036
Oct 30 00:44:20.928242 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:20.928216    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 00:44:20.928314 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:20.928275    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 00:44:20.928314 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:20.928302    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 00:44:20.954161 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:20.954101    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-lrlct" podStartSLOduration=0.954082606 podStartE2EDuration="954.082606ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:44:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:44:20.95347146 +0000 UTC m=+756.569904339" watchObservedRunningTime="2026-10-30 00:44:20.954082606 +0000 UTC m=+756.570515486"
Oct 30 00:44:21.505860 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:21.505827    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-lrlct"
Oct 30 00:44:21.511069 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:21.511033    5002 patch_prober.go:29] interesting pod/e2e-worker-w6-021d6d-openshift-default-65c58b6947-lrlct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body=
Oct 30 00:44:21.511319 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:21.511291    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-lrlct" podUID="099508e4-19ee-4e9b-93f3-0ef7a0d5b8b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused"
Oct 30 00:44:21.918497 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:21.918400    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-947lv" event={"ID":"ee192122-5e45-439e-93b4-eb2a0bf03260","Type":"ContainerStarted","Data":"7730166cee01474ea4a323a0a9ced68c13e88ead18eb88f6d8ebc207fc3c3ff1"}
Oct 30 00:44:21.918497 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:21.918458    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-947lv" event={"ID":"ee192122-5e45-439e-93b4-eb2a0bf03260","Type":"ContainerStarted","Data":"f1783e53b99a5291479a6c8dc84912812c677c7a3fab9b3c3ac62f86a43a5036"}
Oct 30 00:44:22.510905 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:22.510477    5002 patch_prober.go:29] interesting pod/e2e-worker-w6-021d6d-openshift-default-65c58b6947-lrlct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body=
Oct 30 00:44:22.510905 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:22.510538    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-lrlct" podUID="099508e4-19ee-4e9b-93f3-0ef7a0d5b8b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused"
Oct 30 00:44:22.620797 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:22.620714    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-947lv"
Oct 30 00:44:22.625441 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:22.625076    5002 patch_prober.go:29] interesting pod/e2e-worker-w2-1a11cf-openshift-default-748ddd859-947lv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body=
Oct 30 00:44:22.625441 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:22.625145    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-947lv" podUID="ee192122-5e45-439e-93b4-eb2a0bf03260" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused"
Oct 30 00:44:23.505053 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:23.504986    5002 patch_prober.go:29] interesting pod/e2e-worker-w6-021d6d-openshift-default-65c58b6947-lrlct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body=
Oct 30 00:44:23.505237 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:23.505106    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-lrlct" podUID="099508e4-19ee-4e9b-93f3-0ef7a0d5b8b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused"
Oct 30 00:44:23.620692 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:23.620653    5002 patch_prober.go:29] interesting pod/e2e-worker-w2-1a11cf-openshift-default-748ddd859-947lv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body=
Oct 30 00:44:23.621080 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:23.620703    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-947lv" podUID="ee192122-5e45-439e-93b4-eb2a0bf03260" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused"
Oct 30 00:44:24.507490 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:24.507096    5002 patch_prober.go:29] interesting pod/e2e-worker-w6-021d6d-openshift-default-65c58b6947-lrlct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body=
Oct 30 00:44:24.507490 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:24.507167    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-lrlct" podUID="099508e4-19ee-4e9b-93f3-0ef7a0d5b8b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused"
Oct 30 00:44:24.621362 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:24.621322    5002 patch_prober.go:29] interesting pod/e2e-worker-w2-1a11cf-openshift-default-748ddd859-947lv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body=
Oct 30 00:44:24.621865 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:24.621404    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-947lv" podUID="ee192122-5e45-439e-93b4-eb2a0bf03260" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused"
Oct 30 00:44:25.508781 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:25.508738    5002 patch_prober.go:29] interesting pod/e2e-worker-w6-021d6d-openshift-default-65c58b6947-lrlct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body=
Oct 30 00:44:25.509056 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:25.509026    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-lrlct" podUID="099508e4-19ee-4e9b-93f3-0ef7a0d5b8b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused"
Oct 30 00:44:25.621167 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:25.621130    5002 patch_prober.go:29] interesting pod/e2e-worker-w2-1a11cf-openshift-default-748ddd859-947lv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body=
Oct 30 00:44:25.621364 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:25.621190    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-947lv" podUID="ee192122-5e45-439e-93b4-eb2a0bf03260" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused"
Oct 30 00:44:26.507775 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:26.507738    5002 patch_prober.go:29] interesting pod/e2e-worker-w6-021d6d-openshift-default-65c58b6947-lrlct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body=
Oct 30 00:44:26.508325 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:26.508296    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-lrlct" podUID="099508e4-19ee-4e9b-93f3-0ef7a0d5b8b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused"
Oct 30 00:44:26.620912 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:26.620876    5002 patch_prober.go:29] interesting pod/e2e-worker-w2-1a11cf-openshift-default-748ddd859-947lv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body=
Oct 30 00:44:26.621093 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:26.620943    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-947lv" podUID="ee192122-5e45-439e-93b4-eb2a0bf03260" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused"
Oct 30 00:44:27.181450 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:27.181366    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-947lv" podStartSLOduration=7.181344659 podStartE2EDuration="7.181344659s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:44:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:44:21.986988851 +0000 UTC m=+757.603421731" watchObservedRunningTime="2026-10-30 00:44:27.181344659 +0000 UTC m=+762.797777541"
Oct 30 00:44:27.183997 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:27.183964    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-29be73/premium-w1-29be73-kserve-7b777cfc8c-7kgp5"]
Oct 30 00:44:27.188560 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:27.188536    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-29be73/premium-w1-29be73-kserve-7b777cfc8c-7kgp5"
Oct 30 00:44:27.208014 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:27.207974    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-29be73/premium-w1-29be73-kserve-7b777cfc8c-7kgp5"]
Oct 30 00:44:27.219927 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:27.219901    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-29be73\"/\"premium-w1-29be73-kserve-self-signed-certs\""
Oct 30 00:44:27.239774 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:27.239741    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6fabbdd-86ca-4cbf-85be-ef250575a355-home\") pod \"premium-w1-29be73-kserve-7b777cfc8c-7kgp5\" (UID: \"e6fabbdd-86ca-4cbf-85be-ef250575a355\") " pod="e2e-models-e2e-worker-w1-29be73/premium-w1-29be73-kserve-7b777cfc8c-7kgp5"
Oct 30 00:44:27.240061 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:27.240029    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhw4w\" (UniqueName: \"kubernetes.io/projected/e6fabbdd-86ca-4cbf-85be-ef250575a355-kube-api-access-vhw4w\") pod \"premium-w1-29be73-kserve-7b777cfc8c-7kgp5\" (UID: \"e6fabbdd-86ca-4cbf-85be-ef250575a355\") " pod="e2e-models-e2e-worker-w1-29be73/premium-w1-29be73-kserve-7b777cfc8c-7kgp5"
Oct 30 00:44:27.240191 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:27.240084    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6fabbdd-86ca-4cbf-85be-ef250575a355-tmp-dir\") pod \"premium-w1-29be73-kserve-7b777cfc8c-7kgp5\" (UID: \"e6fabbdd-86ca-4cbf-85be-ef250575a355\") " pod="e2e-models-e2e-worker-w1-29be73/premium-w1-29be73-kserve-7b777cfc8c-7kgp5"
Oct 30 00:44:27.240191 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:27.240125    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6fabbdd-86ca-4cbf-85be-ef250575a355-tls-certs\") pod \"premium-w1-29be73-kserve-7b777cfc8c-7kgp5\" (UID: \"e6fabbdd-86ca-4cbf-85be-ef250575a355\") " pod="e2e-models-e2e-worker-w1-29be73/premium-w1-29be73-kserve-7b777cfc8c-7kgp5"
Oct 30 00:44:27.240191 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:27.240156    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6fabbdd-86ca-4cbf-85be-ef250575a355-model-cache\") pod \"premium-w1-29be73-kserve-7b777cfc8c-7kgp5\" (UID: \"e6fabbdd-86ca-4cbf-85be-ef250575a355\") " pod="e2e-models-e2e-worker-w1-29be73/premium-w1-29be73-kserve-7b777cfc8c-7kgp5"
Oct 30 00:44:27.240395 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:27.240211    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6fabbdd-86ca-4cbf-85be-ef250575a355-dshm\") pod \"premium-w1-29be73-kserve-7b777cfc8c-7kgp5\" (UID: \"e6fabbdd-86ca-4cbf-85be-ef250575a355\") " pod="e2e-models-e2e-worker-w1-29be73/premium-w1-29be73-kserve-7b777cfc8c-7kgp5"
Oct 30 00:44:27.343371 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:27.343331    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6fabbdd-86ca-4cbf-85be-ef250575a355-home\") pod \"premium-w1-29be73-kserve-7b777cfc8c-7kgp5\" (UID: \"e6fabbdd-86ca-4cbf-85be-ef250575a355\") " pod="e2e-models-e2e-worker-w1-29be73/premium-w1-29be73-kserve-7b777cfc8c-7kgp5"
Oct 30 00:44:27.344159 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:27.344138    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhw4w\" (UniqueName: \"kubernetes.io/projected/e6fabbdd-86ca-4cbf-85be-ef250575a355-kube-api-access-vhw4w\") pod \"premium-w1-29be73-kserve-7b777cfc8c-7kgp5\" (UID: \"e6fabbdd-86ca-4cbf-85be-ef250575a355\") " pod="e2e-models-e2e-worker-w1-29be73/premium-w1-29be73-kserve-7b777cfc8c-7kgp5"
Oct 30 00:44:27.344702 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:27.344676    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6fabbdd-86ca-4cbf-85be-ef250575a355-tmp-dir\") pod \"premium-w1-29be73-kserve-7b777cfc8c-7kgp5\" (UID: \"e6fabbdd-86ca-4cbf-85be-ef250575a355\") " pod="e2e-models-e2e-worker-w1-29be73/premium-w1-29be73-kserve-7b777cfc8c-7kgp5"
Oct 30 00:44:27.345198 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:27.345181    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6fabbdd-86ca-4cbf-85be-ef250575a355-tls-certs\") pod \"premium-w1-29be73-kserve-7b777cfc8c-7kgp5\" (UID: \"e6fabbdd-86ca-4cbf-85be-ef250575a355\") " pod="e2e-models-e2e-worker-w1-29be73/premium-w1-29be73-kserve-7b777cfc8c-7kgp5"
Oct 30 00:44:27.345736 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:27.345718    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6fabbdd-86ca-4cbf-85be-ef250575a355-model-cache\") pod \"premium-w1-29be73-kserve-7b777cfc8c-7kgp5\" (UID: \"e6fabbdd-86ca-4cbf-85be-ef250575a355\") " pod="e2e-models-e2e-worker-w1-29be73/premium-w1-29be73-kserve-7b777cfc8c-7kgp5"
Oct 30 00:44:27.346235 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:27.346219    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6fabbdd-86ca-4cbf-85be-ef250575a355-dshm\") pod \"premium-w1-29be73-kserve-7b777cfc8c-7kgp5\" (UID: \"e6fabbdd-86ca-4cbf-85be-ef250575a355\") " pod="e2e-models-e2e-worker-w1-29be73/premium-w1-29be73-kserve-7b777cfc8c-7kgp5"
Oct 30 00:44:27.346724 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:27.346129    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6fabbdd-86ca-4cbf-85be-ef250575a355-model-cache\") pod \"premium-w1-29be73-kserve-7b777cfc8c-7kgp5\" (UID: \"e6fabbdd-86ca-4cbf-85be-ef250575a355\") " pod="e2e-models-e2e-worker-w1-29be73/premium-w1-29be73-kserve-7b777cfc8c-7kgp5"
Oct 30 00:44:27.346851 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:27.345116    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6fabbdd-86ca-4cbf-85be-ef250575a355-tmp-dir\") pod \"premium-w1-29be73-kserve-7b777cfc8c-7kgp5\" (UID: \"e6fabbdd-86ca-4cbf-85be-ef250575a355\") " pod="e2e-models-e2e-worker-w1-29be73/premium-w1-29be73-kserve-7b777cfc8c-7kgp5"
Oct 30 00:44:27.346953 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:27.344059    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6fabbdd-86ca-4cbf-85be-ef250575a355-home\") pod \"premium-w1-29be73-kserve-7b777cfc8c-7kgp5\" (UID: \"e6fabbdd-86ca-4cbf-85be-ef250575a355\") " pod="e2e-models-e2e-worker-w1-29be73/premium-w1-29be73-kserve-7b777cfc8c-7kgp5"
Oct 30 00:44:27.376323 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:27.375740    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6fabbdd-86ca-4cbf-85be-ef250575a355-dshm\") pod \"premium-w1-29be73-kserve-7b777cfc8c-7kgp5\" (UID: \"e6fabbdd-86ca-4cbf-85be-ef250575a355\") " pod="e2e-models-e2e-worker-w1-29be73/premium-w1-29be73-kserve-7b777cfc8c-7kgp5"
Oct 30 00:44:27.377647 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:27.377622    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6fabbdd-86ca-4cbf-85be-ef250575a355-tls-certs\") pod \"premium-w1-29be73-kserve-7b777cfc8c-7kgp5\" (UID: \"e6fabbdd-86ca-4cbf-85be-ef250575a355\") " pod="e2e-models-e2e-worker-w1-29be73/premium-w1-29be73-kserve-7b777cfc8c-7kgp5"
Oct 30 00:44:27.391407 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:27.386065    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhw4w\" (UniqueName: \"kubernetes.io/projected/e6fabbdd-86ca-4cbf-85be-ef250575a355-kube-api-access-vhw4w\") pod \"premium-w1-29be73-kserve-7b777cfc8c-7kgp5\" (UID: \"e6fabbdd-86ca-4cbf-85be-ef250575a355\") " pod="e2e-models-e2e-worker-w1-29be73/premium-w1-29be73-kserve-7b777cfc8c-7kgp5"
Oct 30 00:44:27.504616 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:27.504582    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-29be73/premium-w1-29be73-kserve-7b777cfc8c-7kgp5"
Oct 30 00:44:27.580243 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:27.580038    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-lrlct"
Oct 30 00:44:27.581080 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:27.581003    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-lrlct"
Oct 30 00:44:27.583907 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:27.583859    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-lrlct"
Oct 30 00:44:27.671314 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:27.670240    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-947lv"
Oct 30 00:44:27.671314 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:27.671188    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-947lv"
Oct 30 00:44:27.672021 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:27.672000    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-947lv"
Oct 30 00:44:27.833709 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:27.833682    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-29be73/premium-w1-29be73-kserve-7b777cfc8c-7kgp5"]
Oct 30 00:44:27.833827 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:44:27.833719    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode6fabbdd_86ca_4cbf_85be_ef250575a355.slice/crio-4287fa47a7aa917fca5fc19383794f5170a2eb3349930c8720ccccd5a4561e41 WatchSource:0}: Error finding container 4287fa47a7aa917fca5fc19383794f5170a2eb3349930c8720ccccd5a4561e41: Status 404 returned error can't find the container with id 4287fa47a7aa917fca5fc19383794f5170a2eb3349930c8720ccccd5a4561e41
Oct 30 00:44:27.950017 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:27.949982    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-29be73/premium-w1-29be73-kserve-7b777cfc8c-7kgp5" event={"ID":"e6fabbdd-86ca-4cbf-85be-ef250575a355","Type":"ContainerStarted","Data":"c00c22ee1139f3eb9a72f3f8ed39348c061219c29a43527a488d35fd30f85075"}
Oct 30 00:44:27.950185 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:27.950041    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-29be73/premium-w1-29be73-kserve-7b777cfc8c-7kgp5" event={"ID":"e6fabbdd-86ca-4cbf-85be-ef250575a355","Type":"ContainerStarted","Data":"4287fa47a7aa917fca5fc19383794f5170a2eb3349930c8720ccccd5a4561e41"}
Oct 30 00:44:27.996731 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:27.996664    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-29be73/premium-w1-29be73-kserve-7b777cfc8c-7kgp5" podStartSLOduration=0.996644975 podStartE2EDuration="996.644975ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:44:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:44:27.995072281 +0000 UTC m=+763.611505161" watchObservedRunningTime="2026-10-30 00:44:27.996644975 +0000 UTC m=+763.613077855"
Oct 30 00:44:28.505728 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:28.505689    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-29be73/premium-w1-29be73-kserve-7b777cfc8c-7kgp5"
Oct 30 00:44:30.630441 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:30.630407    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-52650f/premium-w0-52650f-kserve-c5bf66f9d-qq5dm"]
Oct 30 00:44:30.634298 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:30.634273    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-52650f/premium-w0-52650f-kserve-c5bf66f9d-qq5dm"
Oct 30 00:44:30.654249 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:30.654218    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-52650f/premium-w0-52650f-kserve-c5bf66f9d-qq5dm"]
Oct 30 00:44:30.666171 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:30.666121    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-52650f\"/\"premium-w0-52650f-kserve-self-signed-certs\""
Oct 30 00:44:30.682054 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:30.682023    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3455d983-b11b-4cee-bfce-d1f399cf9679-tmp-dir\") pod \"premium-w0-52650f-kserve-c5bf66f9d-qq5dm\" (UID: \"3455d983-b11b-4cee-bfce-d1f399cf9679\") " pod="e2e-models-e2e-worker-w0-52650f/premium-w0-52650f-kserve-c5bf66f9d-qq5dm"
Oct 30 00:44:30.682222 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:30.682068    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3455d983-b11b-4cee-bfce-d1f399cf9679-dshm\") pod \"premium-w0-52650f-kserve-c5bf66f9d-qq5dm\" (UID: \"3455d983-b11b-4cee-bfce-d1f399cf9679\") " pod="e2e-models-e2e-worker-w0-52650f/premium-w0-52650f-kserve-c5bf66f9d-qq5dm"
Oct 30 00:44:30.682287 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:30.682218    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3455d983-b11b-4cee-bfce-d1f399cf9679-model-cache\") pod \"premium-w0-52650f-kserve-c5bf66f9d-qq5dm\" (UID: \"3455d983-b11b-4cee-bfce-d1f399cf9679\") " pod="e2e-models-e2e-worker-w0-52650f/premium-w0-52650f-kserve-c5bf66f9d-qq5dm"
Oct 30 00:44:30.682352 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:30.682299    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3455d983-b11b-4cee-bfce-d1f399cf9679-home\") pod \"premium-w0-52650f-kserve-c5bf66f9d-qq5dm\" (UID: \"3455d983-b11b-4cee-bfce-d1f399cf9679\") " pod="e2e-models-e2e-worker-w0-52650f/premium-w0-52650f-kserve-c5bf66f9d-qq5dm"
Oct 30 00:44:30.682352 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:30.682338    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82sbm\" (UniqueName: \"kubernetes.io/projected/3455d983-b11b-4cee-bfce-d1f399cf9679-kube-api-access-82sbm\") pod \"premium-w0-52650f-kserve-c5bf66f9d-qq5dm\" (UID: \"3455d983-b11b-4cee-bfce-d1f399cf9679\") " pod="e2e-models-e2e-worker-w0-52650f/premium-w0-52650f-kserve-c5bf66f9d-qq5dm"
Oct 30 00:44:30.682490 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:30.682445    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3455d983-b11b-4cee-bfce-d1f399cf9679-tls-certs\") pod \"premium-w0-52650f-kserve-c5bf66f9d-qq5dm\" (UID: \"3455d983-b11b-4cee-bfce-d1f399cf9679\") " pod="e2e-models-e2e-worker-w0-52650f/premium-w0-52650f-kserve-c5bf66f9d-qq5dm"
Oct 30 00:44:30.783292 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:30.783254    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3455d983-b11b-4cee-bfce-d1f399cf9679-model-cache\") pod \"premium-w0-52650f-kserve-c5bf66f9d-qq5dm\" (UID: \"3455d983-b11b-4cee-bfce-d1f399cf9679\") " pod="e2e-models-e2e-worker-w0-52650f/premium-w0-52650f-kserve-c5bf66f9d-qq5dm"
Oct 30 00:44:30.783494 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:30.783327    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3455d983-b11b-4cee-bfce-d1f399cf9679-home\") pod \"premium-w0-52650f-kserve-c5bf66f9d-qq5dm\" (UID: \"3455d983-b11b-4cee-bfce-d1f399cf9679\") " pod="e2e-models-e2e-worker-w0-52650f/premium-w0-52650f-kserve-c5bf66f9d-qq5dm"
Oct 30 00:44:30.783494 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:30.783364    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82sbm\" (UniqueName: \"kubernetes.io/projected/3455d983-b11b-4cee-bfce-d1f399cf9679-kube-api-access-82sbm\") pod \"premium-w0-52650f-kserve-c5bf66f9d-qq5dm\" (UID: \"3455d983-b11b-4cee-bfce-d1f399cf9679\") " pod="e2e-models-e2e-worker-w0-52650f/premium-w0-52650f-kserve-c5bf66f9d-qq5dm"
Oct 30 00:44:30.783494 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:30.783417    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3455d983-b11b-4cee-bfce-d1f399cf9679-tls-certs\") pod \"premium-w0-52650f-kserve-c5bf66f9d-qq5dm\" (UID: \"3455d983-b11b-4cee-bfce-d1f399cf9679\") " pod="e2e-models-e2e-worker-w0-52650f/premium-w0-52650f-kserve-c5bf66f9d-qq5dm"
Oct 30 00:44:30.783613 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:30.783499    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3455d983-b11b-4cee-bfce-d1f399cf9679-tmp-dir\") pod \"premium-w0-52650f-kserve-c5bf66f9d-qq5dm\" (UID: \"3455d983-b11b-4cee-bfce-d1f399cf9679\") " pod="e2e-models-e2e-worker-w0-52650f/premium-w0-52650f-kserve-c5bf66f9d-qq5dm"
Oct 30 00:44:30.783613 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:30.783525    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3455d983-b11b-4cee-bfce-d1f399cf9679-dshm\") pod \"premium-w0-52650f-kserve-c5bf66f9d-qq5dm\" (UID: \"3455d983-b11b-4cee-bfce-d1f399cf9679\") " pod="e2e-models-e2e-worker-w0-52650f/premium-w0-52650f-kserve-c5bf66f9d-qq5dm"
Oct 30 00:44:30.783811 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:30.783779    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3455d983-b11b-4cee-bfce-d1f399cf9679-model-cache\") pod \"premium-w0-52650f-kserve-c5bf66f9d-qq5dm\" (UID: \"3455d983-b11b-4cee-bfce-d1f399cf9679\") " pod="e2e-models-e2e-worker-w0-52650f/premium-w0-52650f-kserve-c5bf66f9d-qq5dm"
Oct 30 00:44:30.784068 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:30.784043    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3455d983-b11b-4cee-bfce-d1f399cf9679-home\") pod \"premium-w0-52650f-kserve-c5bf66f9d-qq5dm\" (UID: \"3455d983-b11b-4cee-bfce-d1f399cf9679\") " pod="e2e-models-e2e-worker-w0-52650f/premium-w0-52650f-kserve-c5bf66f9d-qq5dm"
Oct 30 00:44:30.784318 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:30.784295    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3455d983-b11b-4cee-bfce-d1f399cf9679-tmp-dir\") pod \"premium-w0-52650f-kserve-c5bf66f9d-qq5dm\" (UID: \"3455d983-b11b-4cee-bfce-d1f399cf9679\") " pod="e2e-models-e2e-worker-w0-52650f/premium-w0-52650f-kserve-c5bf66f9d-qq5dm"
Oct 30 00:44:30.810402 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:30.802139    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3455d983-b11b-4cee-bfce-d1f399cf9679-tls-certs\") pod \"premium-w0-52650f-kserve-c5bf66f9d-qq5dm\" (UID: \"3455d983-b11b-4cee-bfce-d1f399cf9679\") " pod="e2e-models-e2e-worker-w0-52650f/premium-w0-52650f-kserve-c5bf66f9d-qq5dm"
Oct 30 00:44:30.814783 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:30.814750    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3455d983-b11b-4cee-bfce-d1f399cf9679-dshm\") pod \"premium-w0-52650f-kserve-c5bf66f9d-qq5dm\" (UID: \"3455d983-b11b-4cee-bfce-d1f399cf9679\") " pod="e2e-models-e2e-worker-w0-52650f/premium-w0-52650f-kserve-c5bf66f9d-qq5dm"
Oct 30 00:44:30.817634 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:30.817563    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-82sbm\" (UniqueName: \"kubernetes.io/projected/3455d983-b11b-4cee-bfce-d1f399cf9679-kube-api-access-82sbm\") pod \"premium-w0-52650f-kserve-c5bf66f9d-qq5dm\" (UID: \"3455d983-b11b-4cee-bfce-d1f399cf9679\") " pod="e2e-models-e2e-worker-w0-52650f/premium-w0-52650f-kserve-c5bf66f9d-qq5dm"
Oct 30 00:44:30.948561 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:30.948529    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-52650f/premium-w0-52650f-kserve-c5bf66f9d-qq5dm"
Oct 30 00:44:31.252424 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:31.252364    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-52650f/premium-w0-52650f-kserve-c5bf66f9d-qq5dm"]
Oct 30 00:44:31.253102 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:44:31.253075    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3455d983_b11b_4cee_bfce_d1f399cf9679.slice/crio-3e66fa365375faab474fa4216f5ff0b2982a4671ca16caa4c456af5ddcb7df64 WatchSource:0}: Error finding container 3e66fa365375faab474fa4216f5ff0b2982a4671ca16caa4c456af5ddcb7df64: Status 404 returned error can't find the container with id 3e66fa365375faab474fa4216f5ff0b2982a4671ca16caa4c456af5ddcb7df64
Oct 30 00:44:31.974323 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:31.974280    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-52650f/premium-w0-52650f-kserve-c5bf66f9d-qq5dm" event={"ID":"3455d983-b11b-4cee-bfce-d1f399cf9679","Type":"ContainerStarted","Data":"1ecfd3f6f656034f89b9864941c49a69ad3a47309acb1fc74491cc34c84cf548"}
Oct 30 00:44:31.974323 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:31.974321    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-52650f/premium-w0-52650f-kserve-c5bf66f9d-qq5dm" event={"ID":"3455d983-b11b-4cee-bfce-d1f399cf9679","Type":"ContainerStarted","Data":"3e66fa365375faab474fa4216f5ff0b2982a4671ca16caa4c456af5ddcb7df64"}
Oct 30 00:44:32.021373 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:32.021322    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-52650f/premium-w0-52650f-kserve-c5bf66f9d-qq5dm" podStartSLOduration=2.021307925 podStartE2EDuration="2.021307925s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:44:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:44:32.019104891 +0000 UTC m=+767.635537770" watchObservedRunningTime="2026-10-30 00:44:32.021307925 +0000 UTC m=+767.637740803"
Oct 30 00:44:32.949633 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:32.949597    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-52650f/premium-w0-52650f-kserve-c5bf66f9d-qq5dm"
Oct 30 00:44:34.637200 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:34.637160    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-6m5cj"]
Oct 30 00:44:34.641162 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:34.641137    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-6m5cj"
Oct 30 00:44:34.657634 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:34.657609    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-cefb27-openshift-default-dockercfg-jwt4c\""
Oct 30 00:44:34.673794 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:34.673769    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-6m5cj"]
Oct 30 00:44:34.725151 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:34.725114    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ff20380-75e5-4089-9f54-50d1917eff55-workload-socket\") pod \"e2e-worker-w3-cefb27-openshift-default-577b649d97-6m5cj\" (UID: \"0ff20380-75e5-4089-9f54-50d1917eff55\") " pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-6m5cj"
Oct 30 00:44:34.725346 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:34.725169    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ff20380-75e5-4089-9f54-50d1917eff55-credential-socket\") pod \"e2e-worker-w3-cefb27-openshift-default-577b649d97-6m5cj\" (UID: \"0ff20380-75e5-4089-9f54-50d1917eff55\") " pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-6m5cj"
Oct 30 00:44:34.725346 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:34.725278    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ff20380-75e5-4089-9f54-50d1917eff55-workload-certs\") pod \"e2e-worker-w3-cefb27-openshift-default-577b649d97-6m5cj\" (UID: \"0ff20380-75e5-4089-9f54-50d1917eff55\") " pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-6m5cj"
Oct 30 00:44:34.725346 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:34.725339    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ff20380-75e5-4089-9f54-50d1917eff55-istio-podinfo\") pod \"e2e-worker-w3-cefb27-openshift-default-577b649d97-6m5cj\" (UID: \"0ff20380-75e5-4089-9f54-50d1917eff55\") " pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-6m5cj"
Oct 30 00:44:34.725553 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:34.725371    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzg9n\" (UniqueName: \"kubernetes.io/projected/0ff20380-75e5-4089-9f54-50d1917eff55-kube-api-access-fzg9n\") pod \"e2e-worker-w3-cefb27-openshift-default-577b649d97-6m5cj\" (UID: \"0ff20380-75e5-4089-9f54-50d1917eff55\") " pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-6m5cj"
Oct 30 00:44:34.725553 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:34.725489    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ff20380-75e5-4089-9f54-50d1917eff55-istio-envoy\") pod \"e2e-worker-w3-cefb27-openshift-default-577b649d97-6m5cj\" (UID: \"0ff20380-75e5-4089-9f54-50d1917eff55\") " pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-6m5cj"
Oct 30 00:44:34.725553 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:34.725509    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ff20380-75e5-4089-9f54-50d1917eff55-istio-data\") pod \"e2e-worker-w3-cefb27-openshift-default-577b649d97-6m5cj\" (UID: \"0ff20380-75e5-4089-9f54-50d1917eff55\") " pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-6m5cj"
Oct 30 00:44:34.725553 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:34.725532    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ff20380-75e5-4089-9f54-50d1917eff55-istio-token\") pod \"e2e-worker-w3-cefb27-openshift-default-577b649d97-6m5cj\" (UID: \"0ff20380-75e5-4089-9f54-50d1917eff55\") " pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-6m5cj"
Oct 30 00:44:34.725744 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:34.725591    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ff20380-75e5-4089-9f54-50d1917eff55-istiod-ca-cert\") pod \"e2e-worker-w3-cefb27-openshift-default-577b649d97-6m5cj\" (UID: \"0ff20380-75e5-4089-9f54-50d1917eff55\") " pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-6m5cj"
Oct 30 00:44:34.826262 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:34.826209    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ff20380-75e5-4089-9f54-50d1917eff55-workload-certs\") pod \"e2e-worker-w3-cefb27-openshift-default-577b649d97-6m5cj\" (UID: \"0ff20380-75e5-4089-9f54-50d1917eff55\") " pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-6m5cj"
Oct 30 00:44:34.826262 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:34.826259    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ff20380-75e5-4089-9f54-50d1917eff55-istio-podinfo\") pod \"e2e-worker-w3-cefb27-openshift-default-577b649d97-6m5cj\" (UID: \"0ff20380-75e5-4089-9f54-50d1917eff55\") " pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-6m5cj"
Oct 30 00:44:34.826490 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:34.826289    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzg9n\" (UniqueName: \"kubernetes.io/projected/0ff20380-75e5-4089-9f54-50d1917eff55-kube-api-access-fzg9n\") pod \"e2e-worker-w3-cefb27-openshift-default-577b649d97-6m5cj\" (UID: \"0ff20380-75e5-4089-9f54-50d1917eff55\") " pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-6m5cj"
Oct 30 00:44:34.826490 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:34.826370    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ff20380-75e5-4089-9f54-50d1917eff55-istio-envoy\") pod \"e2e-worker-w3-cefb27-openshift-default-577b649d97-6m5cj\" (UID: \"0ff20380-75e5-4089-9f54-50d1917eff55\") " pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-6m5cj"
Oct 30 00:44:34.826490 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:34.826415    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ff20380-75e5-4089-9f54-50d1917eff55-istio-data\") pod \"e2e-worker-w3-cefb27-openshift-default-577b649d97-6m5cj\" (UID: \"0ff20380-75e5-4089-9f54-50d1917eff55\") " pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-6m5cj"
Oct 30 00:44:34.826490 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:34.826445    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ff20380-75e5-4089-9f54-50d1917eff55-istio-token\") pod \"e2e-worker-w3-cefb27-openshift-default-577b649d97-6m5cj\" (UID: \"0ff20380-75e5-4089-9f54-50d1917eff55\") " pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-6m5cj"
Oct 30 00:44:34.826490 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:34.826488    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ff20380-75e5-4089-9f54-50d1917eff55-istiod-ca-cert\") pod \"e2e-worker-w3-cefb27-openshift-default-577b649d97-6m5cj\" (UID: \"0ff20380-75e5-4089-9f54-50d1917eff55\") " pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-6m5cj"
Oct 30 00:44:34.826645 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:34.826570    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ff20380-75e5-4089-9f54-50d1917eff55-workload-socket\") pod \"e2e-worker-w3-cefb27-openshift-default-577b649d97-6m5cj\" (UID: \"0ff20380-75e5-4089-9f54-50d1917eff55\") " pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-6m5cj"
Oct 30 00:44:34.826645 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:34.826618    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ff20380-75e5-4089-9f54-50d1917eff55-credential-socket\") pod \"e2e-worker-w3-cefb27-openshift-default-577b649d97-6m5cj\" (UID: \"0ff20380-75e5-4089-9f54-50d1917eff55\") " pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-6m5cj"
Oct 30 00:44:34.827084 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:34.827058    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ff20380-75e5-4089-9f54-50d1917eff55-credential-socket\") pod \"e2e-worker-w3-cefb27-openshift-default-577b649d97-6m5cj\" (UID: \"0ff20380-75e5-4089-9f54-50d1917eff55\") " pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-6m5cj"
Oct 30 00:44:34.827970 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:34.827122    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ff20380-75e5-4089-9f54-50d1917eff55-workload-socket\") pod \"e2e-worker-w3-cefb27-openshift-default-577b649d97-6m5cj\" (UID: \"0ff20380-75e5-4089-9f54-50d1917eff55\") " pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-6m5cj"
Oct 30 00:44:34.827970 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:34.827370    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ff20380-75e5-4089-9f54-50d1917eff55-istio-data\") pod \"e2e-worker-w3-cefb27-openshift-default-577b649d97-6m5cj\" (UID: \"0ff20380-75e5-4089-9f54-50d1917eff55\") " pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-6m5cj"
Oct 30 00:44:34.828176 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:34.827897    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ff20380-75e5-4089-9f54-50d1917eff55-istiod-ca-cert\") pod \"e2e-worker-w3-cefb27-openshift-default-577b649d97-6m5cj\" (UID: \"0ff20380-75e5-4089-9f54-50d1917eff55\") " pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-6m5cj"
Oct 30 00:44:34.828252 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:34.828219    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ff20380-75e5-4089-9f54-50d1917eff55-workload-certs\") pod \"e2e-worker-w3-cefb27-openshift-default-577b649d97-6m5cj\" (UID: \"0ff20380-75e5-4089-9f54-50d1917eff55\") " pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-6m5cj"
Oct 30 00:44:34.829831 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:34.829804    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ff20380-75e5-4089-9f54-50d1917eff55-istio-envoy\") pod \"e2e-worker-w3-cefb27-openshift-default-577b649d97-6m5cj\" (UID: \"0ff20380-75e5-4089-9f54-50d1917eff55\") " pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-6m5cj"
Oct 30 00:44:34.830734 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:34.830711    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ff20380-75e5-4089-9f54-50d1917eff55-istio-podinfo\") pod \"e2e-worker-w3-cefb27-openshift-default-577b649d97-6m5cj\" (UID: \"0ff20380-75e5-4089-9f54-50d1917eff55\") " pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-6m5cj"
Oct 30 00:44:34.852835 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:34.852801    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ff20380-75e5-4089-9f54-50d1917eff55-istio-token\") pod \"e2e-worker-w3-cefb27-openshift-default-577b649d97-6m5cj\" (UID: \"0ff20380-75e5-4089-9f54-50d1917eff55\") " pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-6m5cj"
Oct 30 00:44:34.856036 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:34.855997    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzg9n\" (UniqueName: \"kubernetes.io/projected/0ff20380-75e5-4089-9f54-50d1917eff55-kube-api-access-fzg9n\") pod \"e2e-worker-w3-cefb27-openshift-default-577b649d97-6m5cj\" (UID: \"0ff20380-75e5-4089-9f54-50d1917eff55\") " pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-6m5cj"
Oct 30 00:44:34.955261 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:34.955228    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-6m5cj"
Oct 30 00:44:35.274245 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:35.274071    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-6m5cj"]
Oct 30 00:44:35.275862 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:44:35.275813    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ff20380_75e5_4089_9f54_50d1917eff55.slice/crio-0c5f5e8465542b9904c5ba4e50c2d86c908df4d9b764789f8868e04b4ef247b3 WatchSource:0}: Error finding container 0c5f5e8465542b9904c5ba4e50c2d86c908df4d9b764789f8868e04b4ef247b3: Status 404 returned error can't find the container with id 0c5f5e8465542b9904c5ba4e50c2d86c908df4d9b764789f8868e04b4ef247b3
Oct 30 00:44:35.278927 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:35.278887    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 00:44:35.279013 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:35.278971    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 00:44:35.279056 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:35.279016    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 00:44:35.637791 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:35.637703    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7d8z4"]
Oct 30 00:44:35.999502 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:35.999456    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-6m5cj" event={"ID":"0ff20380-75e5-4089-9f54-50d1917eff55","Type":"ContainerStarted","Data":"2298517812d470df4b182cf6e39d29cb2d02de7612a0c0ab201dda442fa3b9cb"}
Oct 30 00:44:35.999712 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:35.999526    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-6m5cj" event={"ID":"0ff20380-75e5-4089-9f54-50d1917eff55","Type":"ContainerStarted","Data":"0c5f5e8465542b9904c5ba4e50c2d86c908df4d9b764789f8868e04b4ef247b3"}
Oct 30 00:44:36.097774 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:36.097707    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-6m5cj" podStartSLOduration=2.097685521 podStartE2EDuration="2.097685521s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:44:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:44:36.083886257 +0000 UTC m=+771.700319155" watchObservedRunningTime="2026-10-30 00:44:36.097685521 +0000 UTC m=+771.714118379"
Oct 30 00:44:36.972510 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:36.969686    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-6m5cj"
Oct 30 00:44:36.972993 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:36.972509    5002 patch_prober.go:29] interesting pod/e2e-worker-w3-cefb27-openshift-default-577b649d97-6m5cj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body=
Oct 30 00:44:36.972993 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:36.972564    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-6m5cj" podUID="0ff20380-75e5-4089-9f54-50d1917eff55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused"
Oct 30 00:44:37.505870 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:37.505835    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-29be73/premium-w1-29be73-kserve-7b777cfc8c-7kgp5"
Oct 30 00:44:37.518875 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:37.518843    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-29be73/premium-w1-29be73-kserve-7b777cfc8c-7kgp5"
Oct 30 00:44:37.971078 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:37.971045    5002 patch_prober.go:29] interesting pod/e2e-worker-w3-cefb27-openshift-default-577b649d97-6m5cj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body=
Oct 30 00:44:37.971362 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:37.971333    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-6m5cj" podUID="0ff20380-75e5-4089-9f54-50d1917eff55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused"
Oct 30 00:44:38.026931 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:38.026901    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-29be73/premium-w1-29be73-kserve-7b777cfc8c-7kgp5"
Oct 30 00:44:38.955909 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:38.955871    5002 patch_prober.go:29] interesting pod/e2e-worker-w3-cefb27-openshift-default-577b649d97-6m5cj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body=
Oct 30 00:44:38.956242 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:38.956207    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-6m5cj" podUID="0ff20380-75e5-4089-9f54-50d1917eff55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused"
Oct 30 00:44:39.955762 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:39.955707    5002 patch_prober.go:29] interesting pod/e2e-worker-w3-cefb27-openshift-default-577b649d97-6m5cj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body=
Oct 30 00:44:39.956397 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:39.956334    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-6m5cj" podUID="0ff20380-75e5-4089-9f54-50d1917eff55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused"
Oct 30 00:44:40.948862 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:40.948828    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-52650f/premium-w0-52650f-kserve-c5bf66f9d-qq5dm"
Oct 30 00:44:40.975405 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:40.962498    5002 patch_prober.go:29] interesting pod/e2e-worker-w3-cefb27-openshift-default-577b649d97-6m5cj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body=
Oct 30 00:44:40.975405 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:40.962556    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-6m5cj" podUID="0ff20380-75e5-4089-9f54-50d1917eff55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused"
Oct 30 00:44:41.004743 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:41.004706    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-52650f/premium-w0-52650f-kserve-c5bf66f9d-qq5dm"
Oct 30 00:44:41.042375 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:41.042335    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-52650f/premium-w0-52650f-kserve-c5bf66f9d-qq5dm"
Oct 30 00:44:42.628523 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:42.627593    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-6m5cj"
Oct 30 00:44:42.628523 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:42.628454    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-6m5cj"
Oct 30 00:44:42.631964 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:42.629873    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-6m5cj"
Oct 30 00:44:42.870991 ip-10-0-128-7 kubenswrapper[5002]: E1030 00:44:42.870952    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbjdq" containerName="extractor"
Oct 30 00:44:49.775792 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:49.775750    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-thfqc"]
Oct 30 00:44:49.781315 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:49.781292    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-thfqc"
Oct 30 00:44:49.797620 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:49.797596    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-6f4fba-openshift-default-dockercfg-l52jt\""
Oct 30 00:44:49.813742 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:49.813714    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-thfqc"]
Oct 30 00:44:49.924895 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:49.924859    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40800c99-ad9a-4219-8e6e-f9aa7fd77f76-credential-socket\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-thfqc\" (UID: \"40800c99-ad9a-4219-8e6e-f9aa7fd77f76\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-thfqc"
Oct 30 00:44:49.925084 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:49.924976    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40800c99-ad9a-4219-8e6e-f9aa7fd77f76-workload-certs\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-thfqc\" (UID: \"40800c99-ad9a-4219-8e6e-f9aa7fd77f76\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-thfqc"
Oct 30 00:44:49.925084 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:49.925058    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40800c99-ad9a-4219-8e6e-f9aa7fd77f76-istiod-ca-cert\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-thfqc\" (UID: \"40800c99-ad9a-4219-8e6e-f9aa7fd77f76\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-thfqc"
Oct 30 00:44:49.925183 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:49.925121    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdmnf\" (UniqueName: \"kubernetes.io/projected/40800c99-ad9a-4219-8e6e-f9aa7fd77f76-kube-api-access-kdmnf\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-thfqc\" (UID: \"40800c99-ad9a-4219-8e6e-f9aa7fd77f76\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-thfqc"
Oct 30 00:44:49.925286 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:49.925270    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40800c99-ad9a-4219-8e6e-f9aa7fd77f76-istio-envoy\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-thfqc\" (UID: \"40800c99-ad9a-4219-8e6e-f9aa7fd77f76\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-thfqc"
Oct 30 00:44:49.925330 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:49.925316    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40800c99-ad9a-4219-8e6e-f9aa7fd77f76-istio-token\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-thfqc\" (UID: \"40800c99-ad9a-4219-8e6e-f9aa7fd77f76\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-thfqc"
Oct 30 00:44:49.925364 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:49.925341    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40800c99-ad9a-4219-8e6e-f9aa7fd77f76-istio-data\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-thfqc\" (UID: \"40800c99-ad9a-4219-8e6e-f9aa7fd77f76\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-thfqc"
Oct 30 00:44:49.925443 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:49.925364    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40800c99-ad9a-4219-8e6e-f9aa7fd77f76-workload-socket\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-thfqc\" (UID: \"40800c99-ad9a-4219-8e6e-f9aa7fd77f76\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-thfqc"
Oct 30 00:44:49.925443 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:49.925403    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40800c99-ad9a-4219-8e6e-f9aa7fd77f76-istio-podinfo\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-thfqc\" (UID: \"40800c99-ad9a-4219-8e6e-f9aa7fd77f76\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-thfqc"
Oct 30 00:44:50.027195 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:50.027108    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40800c99-ad9a-4219-8e6e-f9aa7fd77f76-workload-certs\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-thfqc\" (UID: \"40800c99-ad9a-4219-8e6e-f9aa7fd77f76\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-thfqc"
Oct 30 00:44:50.027195 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:50.027153    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40800c99-ad9a-4219-8e6e-f9aa7fd77f76-istiod-ca-cert\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-thfqc\" (UID: \"40800c99-ad9a-4219-8e6e-f9aa7fd77f76\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-thfqc"
Oct 30 00:44:50.027375 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:50.027198    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdmnf\" (UniqueName: \"kubernetes.io/projected/40800c99-ad9a-4219-8e6e-f9aa7fd77f76-kube-api-access-kdmnf\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-thfqc\" (UID: \"40800c99-ad9a-4219-8e6e-f9aa7fd77f76\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-thfqc"
Oct 30 00:44:50.027375 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:50.027338    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40800c99-ad9a-4219-8e6e-f9aa7fd77f76-istio-envoy\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-thfqc\" (UID: \"40800c99-ad9a-4219-8e6e-f9aa7fd77f76\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-thfqc"
Oct 30 00:44:50.027508 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:50.027407    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40800c99-ad9a-4219-8e6e-f9aa7fd77f76-istio-token\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-thfqc\" (UID: \"40800c99-ad9a-4219-8e6e-f9aa7fd77f76\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-thfqc"
Oct 30 00:44:50.027508 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:50.027441    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40800c99-ad9a-4219-8e6e-f9aa7fd77f76-istio-data\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-thfqc\" (UID: \"40800c99-ad9a-4219-8e6e-f9aa7fd77f76\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-thfqc"
Oct 30 00:44:50.027508 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:50.027486    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40800c99-ad9a-4219-8e6e-f9aa7fd77f76-workload-socket\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-thfqc\" (UID: \"40800c99-ad9a-4219-8e6e-f9aa7fd77f76\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-thfqc"
Oct 30 00:44:50.027667 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:50.027547    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40800c99-ad9a-4219-8e6e-f9aa7fd77f76-workload-certs\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-thfqc\" (UID: \"40800c99-ad9a-4219-8e6e-f9aa7fd77f76\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-thfqc"
Oct 30 00:44:50.027667 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:50.027659    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40800c99-ad9a-4219-8e6e-f9aa7fd77f76-istio-podinfo\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-thfqc\" (UID: \"40800c99-ad9a-4219-8e6e-f9aa7fd77f76\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-thfqc"
Oct 30 00:44:50.027775 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:50.027706    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40800c99-ad9a-4219-8e6e-f9aa7fd77f76-credential-socket\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-thfqc\" (UID: \"40800c99-ad9a-4219-8e6e-f9aa7fd77f76\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-thfqc"
Oct 30 00:44:50.027876 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:50.027852    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40800c99-ad9a-4219-8e6e-f9aa7fd77f76-istio-data\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-thfqc\" (UID: \"40800c99-ad9a-4219-8e6e-f9aa7fd77f76\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-thfqc"
Oct 30 00:44:50.027937 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:50.027914    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40800c99-ad9a-4219-8e6e-f9aa7fd77f76-workload-socket\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-thfqc\" (UID: \"40800c99-ad9a-4219-8e6e-f9aa7fd77f76\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-thfqc"
Oct 30 00:44:50.028069 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:50.028050    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40800c99-ad9a-4219-8e6e-f9aa7fd77f76-credential-socket\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-thfqc\" (UID: \"40800c99-ad9a-4219-8e6e-f9aa7fd77f76\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-thfqc"
Oct 30 00:44:50.028603 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:50.028578    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40800c99-ad9a-4219-8e6e-f9aa7fd77f76-istiod-ca-cert\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-thfqc\" (UID: \"40800c99-ad9a-4219-8e6e-f9aa7fd77f76\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-thfqc"
Oct 30 00:44:50.029850 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:50.029824    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40800c99-ad9a-4219-8e6e-f9aa7fd77f76-istio-envoy\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-thfqc\" (UID: \"40800c99-ad9a-4219-8e6e-f9aa7fd77f76\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-thfqc"
Oct 30 00:44:50.030161 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:50.030140    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40800c99-ad9a-4219-8e6e-f9aa7fd77f76-istio-podinfo\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-thfqc\" (UID: \"40800c99-ad9a-4219-8e6e-f9aa7fd77f76\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-thfqc"
Oct 30 00:44:50.054102 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:50.054067    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40800c99-ad9a-4219-8e6e-f9aa7fd77f76-istio-token\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-thfqc\" (UID: \"40800c99-ad9a-4219-8e6e-f9aa7fd77f76\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-thfqc"
Oct 30 00:44:50.054249 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:50.054228    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdmnf\" (UniqueName: \"kubernetes.io/projected/40800c99-ad9a-4219-8e6e-f9aa7fd77f76-kube-api-access-kdmnf\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-thfqc\" (UID: \"40800c99-ad9a-4219-8e6e-f9aa7fd77f76\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-thfqc"
Oct 30 00:44:50.094813 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:50.094784    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-thfqc"
Oct 30 00:44:50.416327 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:50.416295    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-thfqc"]
Oct 30 00:44:50.416701 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:44:50.416663    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40800c99_ad9a_4219_8e6e_f9aa7fd77f76.slice/crio-7d9d2e9078a95b347ccfa21d588532640246d97b649694fc03fb5f339522b15e WatchSource:0}: Error finding container 7d9d2e9078a95b347ccfa21d588532640246d97b649694fc03fb5f339522b15e: Status 404 returned error can't find the container with id 7d9d2e9078a95b347ccfa21d588532640246d97b649694fc03fb5f339522b15e
Oct 30 00:44:50.419248 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:50.419205    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 00:44:50.419395 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:50.419286    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 00:44:50.419395 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:50.419318    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 00:44:51.095319 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:51.093671    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-thfqc" event={"ID":"40800c99-ad9a-4219-8e6e-f9aa7fd77f76","Type":"ContainerStarted","Data":"2a0a4d1434b7cd0d24d70350b5d1ad6d6529cec0c484fbc73c215b8fb04b8fac"}
Oct 30 00:44:51.095319 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:51.093817    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-thfqc" event={"ID":"40800c99-ad9a-4219-8e6e-f9aa7fd77f76","Type":"ContainerStarted","Data":"7d9d2e9078a95b347ccfa21d588532640246d97b649694fc03fb5f339522b15e"}
Oct 30 00:44:51.095996 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:51.095362    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-thfqc"
Oct 30 00:44:51.096802 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:51.096773    5002 patch_prober.go:29] interesting pod/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-thfqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body=
Oct 30 00:44:51.103111 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:51.103072    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-thfqc" podUID="40800c99-ad9a-4219-8e6e-f9aa7fd77f76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused"
Oct 30 00:44:51.180089 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:51.180022    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-thfqc" podStartSLOduration=2.179999359 podStartE2EDuration="2.179999359s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:44:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:44:51.167612442 +0000 UTC m=+786.784045324" watchObservedRunningTime="2026-10-30 00:44:51.179999359 +0000 UTC m=+786.796432240"
Oct 30 00:44:52.095193 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:52.095152    5002 patch_prober.go:29] interesting pod/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-thfqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body=
Oct 30 00:44:52.095424 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:52.095252    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-thfqc" podUID="40800c99-ad9a-4219-8e6e-f9aa7fd77f76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused"
Oct 30 00:44:53.095835 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:53.095793    5002 patch_prober.go:29] interesting pod/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-thfqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body=
Oct 30 00:44:53.096279 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:53.095894    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-thfqc" podUID="40800c99-ad9a-4219-8e6e-f9aa7fd77f76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused"
Oct 30 00:44:54.095840 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:54.095803    5002 patch_prober.go:29] interesting pod/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-thfqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body=
Oct 30 00:44:54.096229 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:54.095871    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-thfqc" podUID="40800c99-ad9a-4219-8e6e-f9aa7fd77f76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused"
Oct 30 00:44:55.095945 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:55.095904    5002 patch_prober.go:29] interesting pod/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-thfqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body=
Oct 30 00:44:55.096400 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:55.095964    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-thfqc" podUID="40800c99-ad9a-4219-8e6e-f9aa7fd77f76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused"
Oct 30 00:44:56.095698 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:56.095658    5002 patch_prober.go:29] interesting pod/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-thfqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body=
Oct 30 00:44:56.095880 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:56.095721    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-thfqc" podUID="40800c99-ad9a-4219-8e6e-f9aa7fd77f76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused"
Oct 30 00:44:57.225968 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:57.225929    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-thfqc"
Oct 30 00:44:57.226842 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:57.226552    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-thfqc"
Oct 30 00:44:57.227677 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:57.227654    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-thfqc"
Oct 30 00:44:59.471143 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:44:59.471097    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7d8z4"]
Oct 30 00:45:00.152165 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:00.152129    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-021d6d-29888685-nftp2"]
Oct 30 00:45:00.156633 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:00.156604    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-021d6d-29888685-nftp2"
Oct 30 00:45:00.163964 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:00.163939    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1a11cf-29888685-pm6kg"]
Oct 30 00:45:00.168583 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:00.168544    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1a11cf-29888685-pm6kg"
Oct 30 00:45:00.173266 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:00.173242    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888685-m7pp6"]
Oct 30 00:45:00.182778 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:00.180739    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888685-m7pp6"
Oct 30 00:45:00.193436 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:00.185956    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5phsh\""
Oct 30 00:45:00.193436 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:00.190273    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-021d6d-29888685-nftp2"]
Oct 30 00:45:00.194617 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:00.194073    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888685-m7pp6"]
Oct 30 00:45:00.194617 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:00.194236    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 00:45:00.205666 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:00.205520    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 00:45:00.205981 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:00.205950    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6f4fba-29888685-rrxtq"]
Oct 30 00:45:00.217950 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:00.217920    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6f4fba-29888685-rrxtq"
Oct 30 00:45:00.245512 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:00.245408    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6f4fba-29888685-rrxtq"]
Oct 30 00:45:00.247756 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:00.247728    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1a11cf-29888685-pm6kg"]
Oct 30 00:45:00.256099 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:00.256066    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e453da6f-d3ef-4d30-9461-08952eb347fd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-021d6d-29888685-nftp2\" (UID: \"e453da6f-d3ef-4d30-9461-08952eb347fd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-021d6d-29888685-nftp2"
Oct 30 00:45:00.256220 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:00.256134    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9sgw\" (UniqueName: \"kubernetes.io/projected/7941ea9e-5393-4ab4-ad00-a4b1cd42cd38-kube-api-access-l9sgw\") pod \"maas-api-key-cleanup-e2e-worker-w2-1a11cf-29888685-pm6kg\" (UID: \"7941ea9e-5393-4ab4-ad00-a4b1cd42cd38\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1a11cf-29888685-pm6kg"
Oct 30 00:45:00.256220 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:00.256203    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klnsp\" (UniqueName: \"kubernetes.io/projected/e453da6f-d3ef-4d30-9461-08952eb347fd-kube-api-access-klnsp\") pod \"maas-api-key-cleanup-e2e-worker-w6-021d6d-29888685-nftp2\" (UID: \"e453da6f-d3ef-4d30-9461-08952eb347fd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-021d6d-29888685-nftp2"
Oct 30 00:45:00.256306 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:00.256252    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrptf\" (UniqueName: \"kubernetes.io/projected/18a26407-78ee-4e2d-a863-f99a13c924e5-kube-api-access-zrptf\") pod \"maas-api-key-cleanup-29888685-m7pp6\" (UID: \"18a26407-78ee-4e2d-a863-f99a13c924e5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888685-m7pp6"
Oct 30 00:45:00.256306 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:00.256289    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/db996a04-ded3-4343-bc10-236b38d60130-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-6f4fba-29888685-rrxtq\" (UID: \"db996a04-ded3-4343-bc10-236b38d60130\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6f4fba-29888685-rrxtq"
Oct 30 00:45:00.256441 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:00.256319    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/18a26407-78ee-4e2d-a863-f99a13c924e5-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888685-m7pp6\" (UID: \"18a26407-78ee-4e2d-a863-f99a13c924e5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888685-m7pp6"
Oct 30 00:45:00.256441 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:00.256372    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5jjm\" (UniqueName: \"kubernetes.io/projected/db996a04-ded3-4343-bc10-236b38d60130-kube-api-access-t5jjm\") pod \"maas-api-key-cleanup-e2e-worker-w5-6f4fba-29888685-rrxtq\" (UID: \"db996a04-ded3-4343-bc10-236b38d60130\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6f4fba-29888685-rrxtq"
Oct 30 00:45:00.256441 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:00.256423    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7941ea9e-5393-4ab4-ad00-a4b1cd42cd38-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-1a11cf-29888685-pm6kg\" (UID: \"7941ea9e-5393-4ab4-ad00-a4b1cd42cd38\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1a11cf-29888685-pm6kg"
Oct 30 00:45:00.357118 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:00.357079    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e453da6f-d3ef-4d30-9461-08952eb347fd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-021d6d-29888685-nftp2\" (UID: \"e453da6f-d3ef-4d30-9461-08952eb347fd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-021d6d-29888685-nftp2"
Oct 30 00:45:00.357419 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:00.357392    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9sgw\" (UniqueName: \"kubernetes.io/projected/7941ea9e-5393-4ab4-ad00-a4b1cd42cd38-kube-api-access-l9sgw\") pod \"maas-api-key-cleanup-e2e-worker-w2-1a11cf-29888685-pm6kg\" (UID: \"7941ea9e-5393-4ab4-ad00-a4b1cd42cd38\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1a11cf-29888685-pm6kg"
Oct 30 00:45:00.357588 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:00.357574    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klnsp\" (UniqueName: \"kubernetes.io/projected/e453da6f-d3ef-4d30-9461-08952eb347fd-kube-api-access-klnsp\") pod \"maas-api-key-cleanup-e2e-worker-w6-021d6d-29888685-nftp2\" (UID: \"e453da6f-d3ef-4d30-9461-08952eb347fd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-021d6d-29888685-nftp2"
Oct 30 00:45:00.357701 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:00.357687    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrptf\" (UniqueName: \"kubernetes.io/projected/18a26407-78ee-4e2d-a863-f99a13c924e5-kube-api-access-zrptf\") pod \"maas-api-key-cleanup-29888685-m7pp6\" (UID: \"18a26407-78ee-4e2d-a863-f99a13c924e5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888685-m7pp6"
Oct 30 00:45:00.357803 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:00.357790    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/db996a04-ded3-4343-bc10-236b38d60130-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-6f4fba-29888685-rrxtq\" (UID: \"db996a04-ded3-4343-bc10-236b38d60130\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6f4fba-29888685-rrxtq"
Oct 30 00:45:00.357894 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:00.357881    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/18a26407-78ee-4e2d-a863-f99a13c924e5-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888685-m7pp6\" (UID: \"18a26407-78ee-4e2d-a863-f99a13c924e5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888685-m7pp6"
Oct 30 00:45:00.358008 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:00.357996    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5jjm\" (UniqueName: \"kubernetes.io/projected/db996a04-ded3-4343-bc10-236b38d60130-kube-api-access-t5jjm\") pod \"maas-api-key-cleanup-e2e-worker-w5-6f4fba-29888685-rrxtq\" (UID: \"db996a04-ded3-4343-bc10-236b38d60130\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6f4fba-29888685-rrxtq"
Oct 30 00:45:00.358091 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:00.358079    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7941ea9e-5393-4ab4-ad00-a4b1cd42cd38-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-1a11cf-29888685-pm6kg\" (UID: \"7941ea9e-5393-4ab4-ad00-a4b1cd42cd38\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1a11cf-29888685-pm6kg"
Oct 30 00:45:00.358252 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:00.358079    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e453da6f-d3ef-4d30-9461-08952eb347fd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-021d6d-29888685-nftp2\" (UID: \"e453da6f-d3ef-4d30-9461-08952eb347fd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-021d6d-29888685-nftp2"
Oct 30 00:45:00.359098 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:00.359074    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/db996a04-ded3-4343-bc10-236b38d60130-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-6f4fba-29888685-rrxtq\" (UID: \"db996a04-ded3-4343-bc10-236b38d60130\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6f4fba-29888685-rrxtq"
Oct 30 00:45:00.359507 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:00.359479    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/18a26407-78ee-4e2d-a863-f99a13c924e5-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888685-m7pp6\" (UID: \"18a26407-78ee-4e2d-a863-f99a13c924e5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888685-m7pp6"
Oct 30 00:45:00.360012 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:00.359985    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7941ea9e-5393-4ab4-ad00-a4b1cd42cd38-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-1a11cf-29888685-pm6kg\" (UID: \"7941ea9e-5393-4ab4-ad00-a4b1cd42cd38\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1a11cf-29888685-pm6kg"
Oct 30 00:45:00.384869 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:00.384777    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-klnsp\" (UniqueName: \"kubernetes.io/projected/e453da6f-d3ef-4d30-9461-08952eb347fd-kube-api-access-klnsp\") pod \"maas-api-key-cleanup-e2e-worker-w6-021d6d-29888685-nftp2\" (UID: \"e453da6f-d3ef-4d30-9461-08952eb347fd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-021d6d-29888685-nftp2"
Oct 30 00:45:00.385741 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:00.385688    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9sgw\" (UniqueName: \"kubernetes.io/projected/7941ea9e-5393-4ab4-ad00-a4b1cd42cd38-kube-api-access-l9sgw\") pod \"maas-api-key-cleanup-e2e-worker-w2-1a11cf-29888685-pm6kg\" (UID: \"7941ea9e-5393-4ab4-ad00-a4b1cd42cd38\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1a11cf-29888685-pm6kg"
Oct 30 00:45:00.388439 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:00.388359    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5jjm\" (UniqueName: \"kubernetes.io/projected/db996a04-ded3-4343-bc10-236b38d60130-kube-api-access-t5jjm\") pod \"maas-api-key-cleanup-e2e-worker-w5-6f4fba-29888685-rrxtq\" (UID: \"db996a04-ded3-4343-bc10-236b38d60130\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6f4fba-29888685-rrxtq"
Oct 30 00:45:00.388892 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:00.388845    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrptf\" (UniqueName: \"kubernetes.io/projected/18a26407-78ee-4e2d-a863-f99a13c924e5-kube-api-access-zrptf\") pod \"maas-api-key-cleanup-29888685-m7pp6\" (UID: \"18a26407-78ee-4e2d-a863-f99a13c924e5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888685-m7pp6"
Oct 30 00:45:00.517673 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:00.517627    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-021d6d-29888685-nftp2"
Oct 30 00:45:00.520754 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:00.520725    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1a11cf-29888685-pm6kg"
Oct 30 00:45:00.541748 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:00.541445    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888685-m7pp6"
Oct 30 00:45:00.554083 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:00.553844    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6f4fba-29888685-rrxtq"
Oct 30 00:45:01.220439 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:01.214779    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1a11cf-29888685-pm6kg"]
Oct 30 00:45:01.220439 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:01.220243    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6f4fba-29888685-rrxtq"]
Oct 30 00:45:01.313410 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:01.309081    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-021d6d-29888685-nftp2"]
Oct 30 00:45:01.331491 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:45:01.331447    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode453da6f_d3ef_4d30_9461_08952eb347fd.slice/crio-b9432464800315a93c76f787ae671aba9d64234f6c5fbb7191a0aa18984aeb22 WatchSource:0}: Error finding container b9432464800315a93c76f787ae671aba9d64234f6c5fbb7191a0aa18984aeb22: Status 404 returned error can't find the container with id b9432464800315a93c76f787ae671aba9d64234f6c5fbb7191a0aa18984aeb22
Oct 30 00:45:01.361348 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:01.361322    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888685-m7pp6"]
Oct 30 00:45:02.220418 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:02.219952    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1a11cf-29888685-pm6kg" event={"ID":"7941ea9e-5393-4ab4-ad00-a4b1cd42cd38","Type":"ContainerStarted","Data":"747dbb4717dd1046638c0ba8af40952dff7fec4f16ef5d19ae0f349f5ce98ddc"}
Oct 30 00:45:02.223342 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:02.223280    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6f4fba-29888685-rrxtq" event={"ID":"db996a04-ded3-4343-bc10-236b38d60130","Type":"ContainerStarted","Data":"4e62a459ecfb90a9de59b5ddaac71774339df92d9037c129dd47d53a714feeb8"}
Oct 30 00:45:02.227989 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:02.227956    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888685-m7pp6" event={"ID":"18a26407-78ee-4e2d-a863-f99a13c924e5","Type":"ContainerStarted","Data":"009fcdce86982d124201d62a74de42fa8640c778fee04aefbd6b9e03fa2f12f2"}
Oct 30 00:45:02.231229 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:02.231202    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-021d6d-29888685-nftp2" event={"ID":"e453da6f-d3ef-4d30-9461-08952eb347fd","Type":"ContainerStarted","Data":"b9432464800315a93c76f787ae671aba9d64234f6c5fbb7191a0aa18984aeb22"}
Oct 30 00:45:06.265610 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:06.265570    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6f4fba-29888685-rrxtq" event={"ID":"db996a04-ded3-4343-bc10-236b38d60130","Type":"ContainerStarted","Data":"76d426c2955bf18094c229a2b6e6ed4fc25a0ecd42853fc7566dde0495b60325"}
Oct 30 00:45:06.267235 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:06.267210    5002 generic.go:356] "Generic (PLEG): container finished" podID="18a26407-78ee-4e2d-a863-f99a13c924e5" containerID="7956d736cb91f05476e22a0157da659f0796a1ea563c7e12f4edcf08f2943f90" exitCode=60
Oct 30 00:45:06.267371 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:06.267285    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888685-m7pp6" event={"ID":"18a26407-78ee-4e2d-a863-f99a13c924e5","Type":"ContainerDied","Data":"7956d736cb91f05476e22a0157da659f0796a1ea563c7e12f4edcf08f2943f90"}
Oct 30 00:45:06.267554 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:06.267534    5002 scope.go:122] "RemoveContainer" containerID="7956d736cb91f05476e22a0157da659f0796a1ea563c7e12f4edcf08f2943f90"
Oct 30 00:45:06.268944 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:06.268918    5002 generic.go:356] "Generic (PLEG): container finished" podID="e453da6f-d3ef-4d30-9461-08952eb347fd" containerID="2b2532f2754c23db5a82e07c1991ea197b66330e8c3ab8071254c20d50fb0ad3" exitCode=60
Oct 30 00:45:06.269040 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:06.268998    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-021d6d-29888685-nftp2" event={"ID":"e453da6f-d3ef-4d30-9461-08952eb347fd","Type":"ContainerDied","Data":"2b2532f2754c23db5a82e07c1991ea197b66330e8c3ab8071254c20d50fb0ad3"}
Oct 30 00:45:06.269240 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:06.269224    5002 scope.go:122] "RemoveContainer" containerID="2b2532f2754c23db5a82e07c1991ea197b66330e8c3ab8071254c20d50fb0ad3"
Oct 30 00:45:06.271677 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:06.271639    5002 generic.go:356] "Generic (PLEG): container finished" podID="7941ea9e-5393-4ab4-ad00-a4b1cd42cd38" containerID="ca191d7870218fe4879b0be8fb94e0be4e17a3cc15eb9d9648bd07e633562dcb" exitCode=60
Oct 30 00:45:06.271677 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:06.271669    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1a11cf-29888685-pm6kg" event={"ID":"7941ea9e-5393-4ab4-ad00-a4b1cd42cd38","Type":"ContainerDied","Data":"ca191d7870218fe4879b0be8fb94e0be4e17a3cc15eb9d9648bd07e633562dcb"}
Oct 30 00:45:06.271969 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:06.271921    5002 scope.go:122] "RemoveContainer" containerID="ca191d7870218fe4879b0be8fb94e0be4e17a3cc15eb9d9648bd07e633562dcb"
Oct 30 00:45:06.309891 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:06.309846    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6f4fba-29888685-rrxtq" podStartSLOduration=2.084085973 podStartE2EDuration="6.309832168s" totalImagesPullingTime="4.225746195s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:45:06.307488512 +0000 UTC m=+801.923921373" watchObservedRunningTime="2026-10-30 00:45:06.309832168 +0000 UTC m=+801.926265046"
Oct 30 00:45:06.572635 ip-10-0-128-7 kubenswrapper[5002]: E1030 00:45:06.572605    5002 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb996a04_ded3_4343_bc10_236b38d60130.slice/crio-76d426c2955bf18094c229a2b6e6ed4fc25a0ecd42853fc7566dde0495b60325.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb996a04_ded3_4343_bc10_236b38d60130.slice/crio-conmon-76d426c2955bf18094c229a2b6e6ed4fc25a0ecd42853fc7566dde0495b60325.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 00:45:06.572635 ip-10-0-128-7 kubenswrapper[5002]: E1030 00:45:06.572626    5002 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb996a04_ded3_4343_bc10_236b38d60130.slice/crio-conmon-76d426c2955bf18094c229a2b6e6ed4fc25a0ecd42853fc7566dde0495b60325.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 00:45:07.288093 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:07.288053    5002 generic.go:356] "Generic (PLEG): container finished" podID="7941ea9e-5393-4ab4-ad00-a4b1cd42cd38" containerID="fe517db43b76f4a82b327bad5fc6d2bf68784f14d5bc5f448490e00895b45bf6" exitCode=60
Oct 30 00:45:07.288530 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:07.288129    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1a11cf-29888685-pm6kg" event={"ID":"7941ea9e-5393-4ab4-ad00-a4b1cd42cd38","Type":"ContainerDied","Data":"fe517db43b76f4a82b327bad5fc6d2bf68784f14d5bc5f448490e00895b45bf6"}
Oct 30 00:45:07.288530 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:07.288216    5002 scope.go:122] "RemoveContainer" containerID="ca191d7870218fe4879b0be8fb94e0be4e17a3cc15eb9d9648bd07e633562dcb"
Oct 30 00:45:07.288530 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:07.288483    5002 scope.go:122] "RemoveContainer" containerID="fe517db43b76f4a82b327bad5fc6d2bf68784f14d5bc5f448490e00895b45bf6"
Oct 30 00:45:07.288794 ip-10-0-128-7 kubenswrapper[5002]: E1030 00:45:07.288770    5002 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-1a11cf-29888685-pm6kg_odh-ai-gateway-infra(7941ea9e-5393-4ab4-ad00-a4b1cd42cd38)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1a11cf-29888685-pm6kg" podUID="7941ea9e-5393-4ab4-ad00-a4b1cd42cd38"
Oct 30 00:45:07.290014 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:07.289984    5002 generic.go:356] "Generic (PLEG): container finished" podID="db996a04-ded3-4343-bc10-236b38d60130" containerID="76d426c2955bf18094c229a2b6e6ed4fc25a0ecd42853fc7566dde0495b60325" exitCode=60
Oct 30 00:45:07.290109 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:07.290020    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6f4fba-29888685-rrxtq" event={"ID":"db996a04-ded3-4343-bc10-236b38d60130","Type":"ContainerDied","Data":"76d426c2955bf18094c229a2b6e6ed4fc25a0ecd42853fc7566dde0495b60325"}
Oct 30 00:45:07.290299 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:07.290283    5002 scope.go:122] "RemoveContainer" containerID="76d426c2955bf18094c229a2b6e6ed4fc25a0ecd42853fc7566dde0495b60325"
Oct 30 00:45:07.292205 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:07.292182    5002 generic.go:356] "Generic (PLEG): container finished" podID="18a26407-78ee-4e2d-a863-f99a13c924e5" containerID="11a35d7cc59885e4ef289938fd76618c64c1c627ebcab5cf1e51d28239f8e3d2" exitCode=60
Oct 30 00:45:07.292287 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:07.292259    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888685-m7pp6" event={"ID":"18a26407-78ee-4e2d-a863-f99a13c924e5","Type":"ContainerDied","Data":"11a35d7cc59885e4ef289938fd76618c64c1c627ebcab5cf1e51d28239f8e3d2"}
Oct 30 00:45:07.292497 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:07.292482    5002 scope.go:122] "RemoveContainer" containerID="11a35d7cc59885e4ef289938fd76618c64c1c627ebcab5cf1e51d28239f8e3d2"
Oct 30 00:45:07.292700 ip-10-0-128-7 kubenswrapper[5002]: E1030 00:45:07.292683    5002 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29888685-m7pp6_odh-ai-gateway-infra(18a26407-78ee-4e2d-a863-f99a13c924e5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888685-m7pp6" podUID="18a26407-78ee-4e2d-a863-f99a13c924e5"
Oct 30 00:45:07.294395 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:07.294308    5002 generic.go:356] "Generic (PLEG): container finished" podID="e453da6f-d3ef-4d30-9461-08952eb347fd" containerID="858462796e410842d4853278939ad3581f7ca5def6fa250dda6ddfd1631edc52" exitCode=60
Oct 30 00:45:07.294395 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:07.294366    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-021d6d-29888685-nftp2" event={"ID":"e453da6f-d3ef-4d30-9461-08952eb347fd","Type":"ContainerDied","Data":"858462796e410842d4853278939ad3581f7ca5def6fa250dda6ddfd1631edc52"}
Oct 30 00:45:07.294658 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:07.294640    5002 scope.go:122] "RemoveContainer" containerID="858462796e410842d4853278939ad3581f7ca5def6fa250dda6ddfd1631edc52"
Oct 30 00:45:07.294894 ip-10-0-128-7 kubenswrapper[5002]: E1030 00:45:07.294873    5002 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-021d6d-29888685-nftp2_odh-ai-gateway-infra(e453da6f-d3ef-4d30-9461-08952eb347fd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-021d6d-29888685-nftp2" podUID="e453da6f-d3ef-4d30-9461-08952eb347fd"
Oct 30 00:45:07.351091 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:07.351062    5002 scope.go:122] "RemoveContainer" containerID="7956d736cb91f05476e22a0157da659f0796a1ea563c7e12f4edcf08f2943f90"
Oct 30 00:45:07.363994 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:07.363967    5002 scope.go:122] "RemoveContainer" containerID="2b2532f2754c23db5a82e07c1991ea197b66330e8c3ab8071254c20d50fb0ad3"
Oct 30 00:45:08.237712 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:08.237683    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98f2e7aa-7vs62"]
Oct 30 00:45:08.241312 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:08.241291    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98f2e7aa-7vs62"
Oct 30 00:45:08.258888 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:08.258865    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 00:45:08.274581 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:08.274550    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98f2e7aa-7vs62"]
Oct 30 00:45:08.302354 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:08.302315    5002 scope.go:122] "RemoveContainer" containerID="11a35d7cc59885e4ef289938fd76618c64c1c627ebcab5cf1e51d28239f8e3d2"
Oct 30 00:45:08.302808 ip-10-0-128-7 kubenswrapper[5002]: E1030 00:45:08.302625    5002 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29888685-m7pp6_odh-ai-gateway-infra(18a26407-78ee-4e2d-a863-f99a13c924e5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888685-m7pp6" podUID="18a26407-78ee-4e2d-a863-f99a13c924e5"
Oct 30 00:45:08.304091 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:08.304070    5002 scope.go:122] "RemoveContainer" containerID="858462796e410842d4853278939ad3581f7ca5def6fa250dda6ddfd1631edc52"
Oct 30 00:45:08.304337 ip-10-0-128-7 kubenswrapper[5002]: E1030 00:45:08.304318    5002 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-021d6d-29888685-nftp2_odh-ai-gateway-infra(e453da6f-d3ef-4d30-9461-08952eb347fd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-021d6d-29888685-nftp2" podUID="e453da6f-d3ef-4d30-9461-08952eb347fd"
Oct 30 00:45:08.305891 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:08.305872    5002 scope.go:122] "RemoveContainer" containerID="fe517db43b76f4a82b327bad5fc6d2bf68784f14d5bc5f448490e00895b45bf6"
Oct 30 00:45:08.306121 ip-10-0-128-7 kubenswrapper[5002]: E1030 00:45:08.306100    5002 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-1a11cf-29888685-pm6kg_odh-ai-gateway-infra(7941ea9e-5393-4ab4-ad00-a4b1cd42cd38)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1a11cf-29888685-pm6kg" podUID="7941ea9e-5393-4ab4-ad00-a4b1cd42cd38"
Oct 30 00:45:08.307592 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:08.307571    5002 generic.go:356] "Generic (PLEG): container finished" podID="db996a04-ded3-4343-bc10-236b38d60130" containerID="32d819c065745e9d39a48e958bcba5031fd0371d85a1525636580556bf030907" exitCode=60
Oct 30 00:45:08.307696 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:08.307601    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6f4fba-29888685-rrxtq" event={"ID":"db996a04-ded3-4343-bc10-236b38d60130","Type":"ContainerDied","Data":"32d819c065745e9d39a48e958bcba5031fd0371d85a1525636580556bf030907"}
Oct 30 00:45:08.307696 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:08.307647    5002 scope.go:122] "RemoveContainer" containerID="76d426c2955bf18094c229a2b6e6ed4fc25a0ecd42853fc7566dde0495b60325"
Oct 30 00:45:08.307843 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:08.307825    5002 scope.go:122] "RemoveContainer" containerID="32d819c065745e9d39a48e958bcba5031fd0371d85a1525636580556bf030907"
Oct 30 00:45:08.308024 ip-10-0-128-7 kubenswrapper[5002]: E1030 00:45:08.308003    5002 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-6f4fba-29888685-rrxtq_odh-ai-gateway-infra(db996a04-ded3-4343-bc10-236b38d60130)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6f4fba-29888685-rrxtq" podUID="db996a04-ded3-4343-bc10-236b38d60130"
Oct 30 00:45:08.318886 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:08.318866    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b76dd9d-45bc-41f8-b2a9-1ecbae554e94-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-98f2e7aa-7vs62\" (UID: \"4b76dd9d-45bc-41f8-b2a9-1ecbae554e94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98f2e7aa-7vs62"
Oct 30 00:45:08.319009 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:08.318992    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b76dd9d-45bc-41f8-b2a9-1ecbae554e94-maas-api-xks-ca\") pod \"maas-api-revoke-sub-98f2e7aa-7vs62\" (UID: \"4b76dd9d-45bc-41f8-b2a9-1ecbae554e94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98f2e7aa-7vs62"
Oct 30 00:45:08.319120 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:08.319097    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dm68f\" (UniqueName: \"kubernetes.io/projected/4b76dd9d-45bc-41f8-b2a9-1ecbae554e94-kube-api-access-dm68f\") pod \"maas-api-revoke-sub-98f2e7aa-7vs62\" (UID: \"4b76dd9d-45bc-41f8-b2a9-1ecbae554e94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98f2e7aa-7vs62"
Oct 30 00:45:08.420263 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:08.420167    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b76dd9d-45bc-41f8-b2a9-1ecbae554e94-maas-api-xks-ca\") pod \"maas-api-revoke-sub-98f2e7aa-7vs62\" (UID: \"4b76dd9d-45bc-41f8-b2a9-1ecbae554e94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98f2e7aa-7vs62"
Oct 30 00:45:08.420466 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:08.420434    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dm68f\" (UniqueName: \"kubernetes.io/projected/4b76dd9d-45bc-41f8-b2a9-1ecbae554e94-kube-api-access-dm68f\") pod \"maas-api-revoke-sub-98f2e7aa-7vs62\" (UID: \"4b76dd9d-45bc-41f8-b2a9-1ecbae554e94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98f2e7aa-7vs62"
Oct 30 00:45:08.421078 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:08.420583    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b76dd9d-45bc-41f8-b2a9-1ecbae554e94-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-98f2e7aa-7vs62\" (UID: \"4b76dd9d-45bc-41f8-b2a9-1ecbae554e94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98f2e7aa-7vs62"
Oct 30 00:45:08.421368 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:08.421343    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b76dd9d-45bc-41f8-b2a9-1ecbae554e94-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-98f2e7aa-7vs62\" (UID: \"4b76dd9d-45bc-41f8-b2a9-1ecbae554e94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98f2e7aa-7vs62"
Oct 30 00:45:08.422738 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:08.422716    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b76dd9d-45bc-41f8-b2a9-1ecbae554e94-maas-api-xks-ca\") pod \"maas-api-revoke-sub-98f2e7aa-7vs62\" (UID: \"4b76dd9d-45bc-41f8-b2a9-1ecbae554e94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98f2e7aa-7vs62"
Oct 30 00:45:08.445629 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:08.445604    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dm68f\" (UniqueName: \"kubernetes.io/projected/4b76dd9d-45bc-41f8-b2a9-1ecbae554e94-kube-api-access-dm68f\") pod \"maas-api-revoke-sub-98f2e7aa-7vs62\" (UID: \"4b76dd9d-45bc-41f8-b2a9-1ecbae554e94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98f2e7aa-7vs62"
Oct 30 00:45:08.553934 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:08.553889    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98f2e7aa-7vs62"
Oct 30 00:45:08.857568 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:08.857537    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98f2e7aa-7vs62"]
Oct 30 00:45:08.858491 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:45:08.858466    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b76dd9d_45bc_41f8_b2a9_1ecbae554e94.slice/crio-64bb72e5eaccbf25eee75cc36e34a2a504bf1aa1abf147dc0d902933732637b7 WatchSource:0}: Error finding container 64bb72e5eaccbf25eee75cc36e34a2a504bf1aa1abf147dc0d902933732637b7: Status 404 returned error can't find the container with id 64bb72e5eaccbf25eee75cc36e34a2a504bf1aa1abf147dc0d902933732637b7
Oct 30 00:45:09.315252 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:09.315220    5002 scope.go:122] "RemoveContainer" containerID="32d819c065745e9d39a48e958bcba5031fd0371d85a1525636580556bf030907"
Oct 30 00:45:09.315734 ip-10-0-128-7 kubenswrapper[5002]: E1030 00:45:09.315583    5002 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-6f4fba-29888685-rrxtq_odh-ai-gateway-infra(db996a04-ded3-4343-bc10-236b38d60130)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6f4fba-29888685-rrxtq" podUID="db996a04-ded3-4343-bc10-236b38d60130"
Oct 30 00:45:09.316484 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:09.316454    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98f2e7aa-7vs62" event={"ID":"4b76dd9d-45bc-41f8-b2a9-1ecbae554e94","Type":"ContainerStarted","Data":"9c2f81cea168f013fcefe1836fbcc2ea77a48bd01c278fe3f9d8aee78e7dc972"}
Oct 30 00:45:09.316586 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:09.316497    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98f2e7aa-7vs62" event={"ID":"4b76dd9d-45bc-41f8-b2a9-1ecbae554e94","Type":"ContainerStarted","Data":"64bb72e5eaccbf25eee75cc36e34a2a504bf1aa1abf147dc0d902933732637b7"}
Oct 30 00:45:09.403748 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:09.403694    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98f2e7aa-7vs62" podStartSLOduration=1.4036785090000001 podStartE2EDuration="1.403678509s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:45:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:45:09.401649849 +0000 UTC m=+805.018082728" watchObservedRunningTime="2026-10-30 00:45:09.403678509 +0000 UTC m=+805.020111409"
Oct 30 00:45:10.324557 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:10.324517    5002 generic.go:356] "Generic (PLEG): container finished" podID="4b76dd9d-45bc-41f8-b2a9-1ecbae554e94" containerID="9c2f81cea168f013fcefe1836fbcc2ea77a48bd01c278fe3f9d8aee78e7dc972" exitCode=0
Oct 30 00:45:10.324943 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:10.324587    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98f2e7aa-7vs62" event={"ID":"4b76dd9d-45bc-41f8-b2a9-1ecbae554e94","Type":"ContainerDied","Data":"9c2f81cea168f013fcefe1836fbcc2ea77a48bd01c278fe3f9d8aee78e7dc972"}
Oct 30 00:45:11.478789 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:11.478757    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98f2e7aa-7vs62"
Oct 30 00:45:11.553050 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:11.553019    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b76dd9d-45bc-41f8-b2a9-1ecbae554e94-kube-api-access-dm68f\" (UniqueName: \"kubernetes.io/projected/4b76dd9d-45bc-41f8-b2a9-1ecbae554e94-kube-api-access-dm68f\") pod \"4b76dd9d-45bc-41f8-b2a9-1ecbae554e94\" (UID: \"4b76dd9d-45bc-41f8-b2a9-1ecbae554e94\") "
Oct 30 00:45:11.553217 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:11.553154    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4b76dd9d-45bc-41f8-b2a9-1ecbae554e94-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b76dd9d-45bc-41f8-b2a9-1ecbae554e94-maas-api-xks-ca\") pod \"4b76dd9d-45bc-41f8-b2a9-1ecbae554e94\" (UID: \"4b76dd9d-45bc-41f8-b2a9-1ecbae554e94\") "
Oct 30 00:45:11.553217 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:11.553202    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b76dd9d-45bc-41f8-b2a9-1ecbae554e94-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b76dd9d-45bc-41f8-b2a9-1ecbae554e94-maas-api-openshift-ca\") pod \"4b76dd9d-45bc-41f8-b2a9-1ecbae554e94\" (UID: \"4b76dd9d-45bc-41f8-b2a9-1ecbae554e94\") "
Oct 30 00:45:11.554273 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:11.554245    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b76dd9d-45bc-41f8-b2a9-1ecbae554e94-maas-api-openshift-ca" pod "4b76dd9d-45bc-41f8-b2a9-1ecbae554e94" (UID: "4b76dd9d-45bc-41f8-b2a9-1ecbae554e94"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:45:11.555916 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:11.555884    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b76dd9d-45bc-41f8-b2a9-1ecbae554e94-maas-api-xks-ca" pod "4b76dd9d-45bc-41f8-b2a9-1ecbae554e94" (UID: "4b76dd9d-45bc-41f8-b2a9-1ecbae554e94"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:45:11.556040 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:11.555983    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b76dd9d-45bc-41f8-b2a9-1ecbae554e94-kube-api-access-dm68f" pod "4b76dd9d-45bc-41f8-b2a9-1ecbae554e94" (UID: "4b76dd9d-45bc-41f8-b2a9-1ecbae554e94"). InnerVolumeSpecName "kube-api-access-dm68f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:45:11.655407 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:11.655300    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b76dd9d-45bc-41f8-b2a9-1ecbae554e94-maas-api-xks-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:45:11.655407 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:11.655334    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b76dd9d-45bc-41f8-b2a9-1ecbae554e94-maas-api-openshift-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:45:11.655407 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:11.655351    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dm68f\" (UniqueName: \"kubernetes.io/projected/4b76dd9d-45bc-41f8-b2a9-1ecbae554e94-kube-api-access-dm68f\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:45:12.337004 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:12.336971    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98f2e7aa-7vs62"
Oct 30 00:45:12.337196 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:12.337006    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98f2e7aa-7vs62" event={"ID":"4b76dd9d-45bc-41f8-b2a9-1ecbae554e94","Type":"ContainerDied","Data":"64bb72e5eaccbf25eee75cc36e34a2a504bf1aa1abf147dc0d902933732637b7"}
Oct 30 00:45:12.337196 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:12.337065    5002 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="64bb72e5eaccbf25eee75cc36e34a2a504bf1aa1abf147dc0d902933732637b7"
Oct 30 00:45:12.862165 ip-10-0-128-7 kubenswrapper[5002]: E1030 00:45:12.862117    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbjdq" containerName="extractor"
Oct 30 00:45:18.487655 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:18.487616    5002 scope.go:122] "RemoveContainer" containerID="858462796e410842d4853278939ad3581f7ca5def6fa250dda6ddfd1631edc52"
Oct 30 00:45:18.488296 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:18.487669    5002 scope.go:122] "RemoveContainer" containerID="fe517db43b76f4a82b327bad5fc6d2bf68784f14d5bc5f448490e00895b45bf6"
Oct 30 00:45:19.378541 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:19.378502    5002 generic.go:356] "Generic (PLEG): container finished" podID="7941ea9e-5393-4ab4-ad00-a4b1cd42cd38" containerID="597d87c5cb8a58fe5423d12b9b4f0721f2e1f0710415ef3e3140e646c2269d02" exitCode=60
Oct 30 00:45:19.378719 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:19.378570    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1a11cf-29888685-pm6kg" event={"ID":"7941ea9e-5393-4ab4-ad00-a4b1cd42cd38","Type":"ContainerDied","Data":"597d87c5cb8a58fe5423d12b9b4f0721f2e1f0710415ef3e3140e646c2269d02"}
Oct 30 00:45:19.378719 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:19.378631    5002 scope.go:122] "RemoveContainer" containerID="fe517db43b76f4a82b327bad5fc6d2bf68784f14d5bc5f448490e00895b45bf6"
Oct 30 00:45:19.379076 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:19.379052    5002 scope.go:122] "RemoveContainer" containerID="597d87c5cb8a58fe5423d12b9b4f0721f2e1f0710415ef3e3140e646c2269d02"
Oct 30 00:45:19.379409 ip-10-0-128-7 kubenswrapper[5002]: E1030 00:45:19.379371    5002 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-1a11cf-29888685-pm6kg_odh-ai-gateway-infra(7941ea9e-5393-4ab4-ad00-a4b1cd42cd38)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1a11cf-29888685-pm6kg" podUID="7941ea9e-5393-4ab4-ad00-a4b1cd42cd38"
Oct 30 00:45:19.381185 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:19.381160    5002 generic.go:356] "Generic (PLEG): container finished" podID="e453da6f-d3ef-4d30-9461-08952eb347fd" containerID="74f53e7a437e3984f9a60541435cbae2576f6fd8cde9c5b91e71419a2846841a" exitCode=60
Oct 30 00:45:19.381275 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:19.381192    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-021d6d-29888685-nftp2" event={"ID":"e453da6f-d3ef-4d30-9461-08952eb347fd","Type":"ContainerDied","Data":"74f53e7a437e3984f9a60541435cbae2576f6fd8cde9c5b91e71419a2846841a"}
Oct 30 00:45:19.381645 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:19.381628    5002 scope.go:122] "RemoveContainer" containerID="74f53e7a437e3984f9a60541435cbae2576f6fd8cde9c5b91e71419a2846841a"
Oct 30 00:45:19.381905 ip-10-0-128-7 kubenswrapper[5002]: E1030 00:45:19.381886    5002 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-021d6d-29888685-nftp2_odh-ai-gateway-infra(e453da6f-d3ef-4d30-9461-08952eb347fd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-021d6d-29888685-nftp2" podUID="e453da6f-d3ef-4d30-9461-08952eb347fd"
Oct 30 00:45:19.401262 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:19.401232    5002 scope.go:122] "RemoveContainer" containerID="858462796e410842d4853278939ad3581f7ca5def6fa250dda6ddfd1631edc52"
Oct 30 00:45:19.487191 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:19.487163    5002 scope.go:122] "RemoveContainer" containerID="11a35d7cc59885e4ef289938fd76618c64c1c627ebcab5cf1e51d28239f8e3d2"
Oct 30 00:45:20.147389 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:20.147341    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-lhfxq"]
Oct 30 00:45:20.148063 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:20.148044    5002 cpu_manager.go:414] "containerMap: removing container" podUID="4b76dd9d-45bc-41f8-b2a9-1ecbae554e94" containerName="revoke-keys"
Oct 30 00:45:20.148109 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:20.148066    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b76dd9d-45bc-41f8-b2a9-1ecbae554e94" containerName="revoke-keys"
Oct 30 00:45:20.148247 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:20.148235    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b76dd9d-45bc-41f8-b2a9-1ecbae554e94" containerName="revoke-keys"
Oct 30 00:45:20.152016 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:20.151991    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-lhfxq"
Oct 30 00:45:20.196285 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:20.196248    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-lhfxq"]
Oct 30 00:45:20.197312 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:20.197279    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-ca5363-openshift-default-dockercfg-m4tkh\""
Oct 30 00:45:20.257838 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:20.257804    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/543ddf6f-c221-48c0-b5d5-881c23397832-istio-token\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-lhfxq\" (UID: \"543ddf6f-c221-48c0-b5d5-881c23397832\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-lhfxq"
Oct 30 00:45:20.257988 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:20.257855    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/543ddf6f-c221-48c0-b5d5-881c23397832-istiod-ca-cert\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-lhfxq\" (UID: \"543ddf6f-c221-48c0-b5d5-881c23397832\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-lhfxq"
Oct 30 00:45:20.257988 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:20.257886    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/543ddf6f-c221-48c0-b5d5-881c23397832-credential-socket\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-lhfxq\" (UID: \"543ddf6f-c221-48c0-b5d5-881c23397832\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-lhfxq"
Oct 30 00:45:20.257988 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:20.257977    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/543ddf6f-c221-48c0-b5d5-881c23397832-istio-data\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-lhfxq\" (UID: \"543ddf6f-c221-48c0-b5d5-881c23397832\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-lhfxq"
Oct 30 00:45:20.258143 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:20.258125    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/543ddf6f-c221-48c0-b5d5-881c23397832-istio-envoy\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-lhfxq\" (UID: \"543ddf6f-c221-48c0-b5d5-881c23397832\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-lhfxq"
Oct 30 00:45:20.258189 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:20.258150    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/543ddf6f-c221-48c0-b5d5-881c23397832-istio-podinfo\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-lhfxq\" (UID: \"543ddf6f-c221-48c0-b5d5-881c23397832\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-lhfxq"
Oct 30 00:45:20.258189 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:20.258176    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/543ddf6f-c221-48c0-b5d5-881c23397832-workload-socket\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-lhfxq\" (UID: \"543ddf6f-c221-48c0-b5d5-881c23397832\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-lhfxq"
Oct 30 00:45:20.258264 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:20.258195    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/543ddf6f-c221-48c0-b5d5-881c23397832-workload-certs\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-lhfxq\" (UID: \"543ddf6f-c221-48c0-b5d5-881c23397832\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-lhfxq"
Oct 30 00:45:20.258353 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:20.258329    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8n4rl\" (UniqueName: \"kubernetes.io/projected/543ddf6f-c221-48c0-b5d5-881c23397832-kube-api-access-8n4rl\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-lhfxq\" (UID: \"543ddf6f-c221-48c0-b5d5-881c23397832\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-lhfxq"
Oct 30 00:45:20.359119 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:20.359082    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/543ddf6f-c221-48c0-b5d5-881c23397832-istio-envoy\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-lhfxq\" (UID: \"543ddf6f-c221-48c0-b5d5-881c23397832\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-lhfxq"
Oct 30 00:45:20.359119 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:20.359120    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/543ddf6f-c221-48c0-b5d5-881c23397832-istio-podinfo\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-lhfxq\" (UID: \"543ddf6f-c221-48c0-b5d5-881c23397832\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-lhfxq"
Oct 30 00:45:20.359401 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:20.359143    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/543ddf6f-c221-48c0-b5d5-881c23397832-workload-socket\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-lhfxq\" (UID: \"543ddf6f-c221-48c0-b5d5-881c23397832\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-lhfxq"
Oct 30 00:45:20.359401 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:20.359160    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/543ddf6f-c221-48c0-b5d5-881c23397832-workload-certs\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-lhfxq\" (UID: \"543ddf6f-c221-48c0-b5d5-881c23397832\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-lhfxq"
Oct 30 00:45:20.359401 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:20.359193    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8n4rl\" (UniqueName: \"kubernetes.io/projected/543ddf6f-c221-48c0-b5d5-881c23397832-kube-api-access-8n4rl\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-lhfxq\" (UID: \"543ddf6f-c221-48c0-b5d5-881c23397832\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-lhfxq"
Oct 30 00:45:20.359401 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:20.359244    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/543ddf6f-c221-48c0-b5d5-881c23397832-istio-token\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-lhfxq\" (UID: \"543ddf6f-c221-48c0-b5d5-881c23397832\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-lhfxq"
Oct 30 00:45:20.359401 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:20.359312    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/543ddf6f-c221-48c0-b5d5-881c23397832-istiod-ca-cert\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-lhfxq\" (UID: \"543ddf6f-c221-48c0-b5d5-881c23397832\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-lhfxq"
Oct 30 00:45:20.359401 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:20.359363    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/543ddf6f-c221-48c0-b5d5-881c23397832-credential-socket\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-lhfxq\" (UID: \"543ddf6f-c221-48c0-b5d5-881c23397832\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-lhfxq"
Oct 30 00:45:20.359853 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:20.359414    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/543ddf6f-c221-48c0-b5d5-881c23397832-istio-data\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-lhfxq\" (UID: \"543ddf6f-c221-48c0-b5d5-881c23397832\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-lhfxq"
Oct 30 00:45:20.359853 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:20.359614    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/543ddf6f-c221-48c0-b5d5-881c23397832-workload-socket\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-lhfxq\" (UID: \"543ddf6f-c221-48c0-b5d5-881c23397832\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-lhfxq"
Oct 30 00:45:20.359853 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:20.359662    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/543ddf6f-c221-48c0-b5d5-881c23397832-workload-certs\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-lhfxq\" (UID: \"543ddf6f-c221-48c0-b5d5-881c23397832\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-lhfxq"
Oct 30 00:45:20.359853 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:20.359800    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/543ddf6f-c221-48c0-b5d5-881c23397832-istio-data\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-lhfxq\" (UID: \"543ddf6f-c221-48c0-b5d5-881c23397832\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-lhfxq"
Oct 30 00:45:20.360034 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:20.359885    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/543ddf6f-c221-48c0-b5d5-881c23397832-credential-socket\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-lhfxq\" (UID: \"543ddf6f-c221-48c0-b5d5-881c23397832\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-lhfxq"
Oct 30 00:45:20.360094 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:20.360074    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/543ddf6f-c221-48c0-b5d5-881c23397832-istiod-ca-cert\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-lhfxq\" (UID: \"543ddf6f-c221-48c0-b5d5-881c23397832\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-lhfxq"
Oct 30 00:45:20.362010 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:20.361984    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/543ddf6f-c221-48c0-b5d5-881c23397832-istio-envoy\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-lhfxq\" (UID: \"543ddf6f-c221-48c0-b5d5-881c23397832\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-lhfxq"
Oct 30 00:45:20.362327 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:20.362308    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/543ddf6f-c221-48c0-b5d5-881c23397832-istio-podinfo\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-lhfxq\" (UID: \"543ddf6f-c221-48c0-b5d5-881c23397832\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-lhfxq"
Oct 30 00:45:20.384401 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:20.384341    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/543ddf6f-c221-48c0-b5d5-881c23397832-istio-token\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-lhfxq\" (UID: \"543ddf6f-c221-48c0-b5d5-881c23397832\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-lhfxq"
Oct 30 00:45:20.384584 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:20.384561    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8n4rl\" (UniqueName: \"kubernetes.io/projected/543ddf6f-c221-48c0-b5d5-881c23397832-kube-api-access-8n4rl\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-lhfxq\" (UID: \"543ddf6f-c221-48c0-b5d5-881c23397832\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-lhfxq"
Oct 30 00:45:20.398667 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:20.391700    5002 generic.go:356] "Generic (PLEG): container finished" podID="18a26407-78ee-4e2d-a863-f99a13c924e5" containerID="1619b4d1ae9f5628c5fabbb394fd971425326080d4783f20f852ff03871d71ef" exitCode=60
Oct 30 00:45:20.398667 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:20.391806    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888685-m7pp6" event={"ID":"18a26407-78ee-4e2d-a863-f99a13c924e5","Type":"ContainerDied","Data":"1619b4d1ae9f5628c5fabbb394fd971425326080d4783f20f852ff03871d71ef"}
Oct 30 00:45:20.398667 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:20.391906    5002 scope.go:122] "RemoveContainer" containerID="11a35d7cc59885e4ef289938fd76618c64c1c627ebcab5cf1e51d28239f8e3d2"
Oct 30 00:45:20.398667 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:20.398209    5002 scope.go:122] "RemoveContainer" containerID="1619b4d1ae9f5628c5fabbb394fd971425326080d4783f20f852ff03871d71ef"
Oct 30 00:45:20.398667 ip-10-0-128-7 kubenswrapper[5002]: E1030 00:45:20.398635    5002 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29888685-m7pp6_odh-ai-gateway-infra(18a26407-78ee-4e2d-a863-f99a13c924e5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888685-m7pp6" podUID="18a26407-78ee-4e2d-a863-f99a13c924e5"
Oct 30 00:45:20.439530 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:20.439494    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1a11cf-29888685-pm6kg"]
Oct 30 00:45:20.466630 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:20.466591    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-lhfxq"
Oct 30 00:45:20.481338 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:20.481306    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-021d6d-29888685-nftp2"]
Oct 30 00:45:20.672175 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:20.672144    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1a11cf-29888685-pm6kg"
Oct 30 00:45:20.725597 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:20.725563    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-021d6d-29888685-nftp2"
Oct 30 00:45:20.764817 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:20.764788    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7941ea9e-5393-4ab4-ad00-a4b1cd42cd38-kube-api-access-l9sgw\" (UniqueName: \"kubernetes.io/projected/7941ea9e-5393-4ab4-ad00-a4b1cd42cd38-kube-api-access-l9sgw\") pod \"7941ea9e-5393-4ab4-ad00-a4b1cd42cd38\" (UID: \"7941ea9e-5393-4ab4-ad00-a4b1cd42cd38\") "
Oct 30 00:45:20.765007 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:20.764867    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7941ea9e-5393-4ab4-ad00-a4b1cd42cd38-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7941ea9e-5393-4ab4-ad00-a4b1cd42cd38-maas-api-service-ca\") pod \"7941ea9e-5393-4ab4-ad00-a4b1cd42cd38\" (UID: \"7941ea9e-5393-4ab4-ad00-a4b1cd42cd38\") "
Oct 30 00:45:20.765210 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:20.765186    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7941ea9e-5393-4ab4-ad00-a4b1cd42cd38-maas-api-service-ca" pod "7941ea9e-5393-4ab4-ad00-a4b1cd42cd38" (UID: "7941ea9e-5393-4ab4-ad00-a4b1cd42cd38"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:45:20.767053 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:20.767023    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7941ea9e-5393-4ab4-ad00-a4b1cd42cd38-kube-api-access-l9sgw" pod "7941ea9e-5393-4ab4-ad00-a4b1cd42cd38" (UID: "7941ea9e-5393-4ab4-ad00-a4b1cd42cd38"). InnerVolumeSpecName "kube-api-access-l9sgw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:45:20.825867 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:45:20.825832    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod543ddf6f_c221_48c0_b5d5_881c23397832.slice/crio-a262e7019ae4dfbaa0a3390837256a9dc8189e36c1871caff7df0323f2844f75 WatchSource:0}: Error finding container a262e7019ae4dfbaa0a3390837256a9dc8189e36c1871caff7df0323f2844f75: Status 404 returned error can't find the container with id a262e7019ae4dfbaa0a3390837256a9dc8189e36c1871caff7df0323f2844f75
Oct 30 00:45:20.826705 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:20.826685    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-lhfxq"]
Oct 30 00:45:20.828530 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:20.828501    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 00:45:20.828598 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:20.828566    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 00:45:20.828638 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:20.828595    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 00:45:20.865462 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:20.865443    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e453da6f-d3ef-4d30-9461-08952eb347fd-kube-api-access-klnsp\" (UniqueName: \"kubernetes.io/projected/e453da6f-d3ef-4d30-9461-08952eb347fd-kube-api-access-klnsp\") pod \"e453da6f-d3ef-4d30-9461-08952eb347fd\" (UID: \"e453da6f-d3ef-4d30-9461-08952eb347fd\") "
Oct 30 00:45:20.865598 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:20.865583    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e453da6f-d3ef-4d30-9461-08952eb347fd-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e453da6f-d3ef-4d30-9461-08952eb347fd-maas-api-service-ca\") pod \"e453da6f-d3ef-4d30-9461-08952eb347fd\" (UID: \"e453da6f-d3ef-4d30-9461-08952eb347fd\") "
Oct 30 00:45:20.865887 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:20.865872    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l9sgw\" (UniqueName: \"kubernetes.io/projected/7941ea9e-5393-4ab4-ad00-a4b1cd42cd38-kube-api-access-l9sgw\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:45:20.865954 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:20.865892    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7941ea9e-5393-4ab4-ad00-a4b1cd42cd38-maas-api-service-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:45:20.865954 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:20.865897    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e453da6f-d3ef-4d30-9461-08952eb347fd-maas-api-service-ca" pod "e453da6f-d3ef-4d30-9461-08952eb347fd" (UID: "e453da6f-d3ef-4d30-9461-08952eb347fd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:45:20.867537 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:20.867519    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e453da6f-d3ef-4d30-9461-08952eb347fd-kube-api-access-klnsp" pod "e453da6f-d3ef-4d30-9461-08952eb347fd" (UID: "e453da6f-d3ef-4d30-9461-08952eb347fd"). InnerVolumeSpecName "kube-api-access-klnsp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:45:20.966795 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:20.966753    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-klnsp\" (UniqueName: \"kubernetes.io/projected/e453da6f-d3ef-4d30-9461-08952eb347fd-kube-api-access-klnsp\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:45:20.966795 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:20.966792    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e453da6f-d3ef-4d30-9461-08952eb347fd-maas-api-service-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:45:21.414095 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:21.414009    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-021d6d-29888685-nftp2"
Oct 30 00:45:21.414095 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:21.414043    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-021d6d-29888685-nftp2" event={"ID":"e453da6f-d3ef-4d30-9461-08952eb347fd","Type":"ContainerDied","Data":"b9432464800315a93c76f787ae671aba9d64234f6c5fbb7191a0aa18984aeb22"}
Oct 30 00:45:21.415051 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:21.414106    5002 scope.go:122] "RemoveContainer" containerID="74f53e7a437e3984f9a60541435cbae2576f6fd8cde9c5b91e71419a2846841a"
Oct 30 00:45:21.415635 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:21.415609    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1a11cf-29888685-pm6kg"
Oct 30 00:45:21.415635 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:21.415624    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1a11cf-29888685-pm6kg" event={"ID":"7941ea9e-5393-4ab4-ad00-a4b1cd42cd38","Type":"ContainerDied","Data":"747dbb4717dd1046638c0ba8af40952dff7fec4f16ef5d19ae0f349f5ce98ddc"}
Oct 30 00:45:21.417496 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:21.417469    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-lhfxq" event={"ID":"543ddf6f-c221-48c0-b5d5-881c23397832","Type":"ContainerStarted","Data":"c2e515361511c699a057ce13a30e41a1bba07e19a8d1ea26054712ab170ff601"}
Oct 30 00:45:21.417603 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:21.417519    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-lhfxq" event={"ID":"543ddf6f-c221-48c0-b5d5-881c23397832","Type":"ContainerStarted","Data":"a262e7019ae4dfbaa0a3390837256a9dc8189e36c1871caff7df0323f2844f75"}
Oct 30 00:45:21.423835 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:21.423818    5002 scope.go:122] "RemoveContainer" containerID="597d87c5cb8a58fe5423d12b9b4f0721f2e1f0710415ef3e3140e646c2269d02"
Oct 30 00:45:21.458345 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:21.458314    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888685-m7pp6"]
Oct 30 00:45:21.476436 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:21.466971    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-lhfxq"
Oct 30 00:45:21.476436 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:21.468966    5002 patch_prober.go:29] interesting pod/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-lhfxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body=
Oct 30 00:45:21.476436 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:21.469015    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-lhfxq" podUID="543ddf6f-c221-48c0-b5d5-881c23397832" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused"
Oct 30 00:45:21.476436 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:21.474109    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1a11cf-29888685-pm6kg"]
Oct 30 00:45:21.477886 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:21.477859    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1a11cf-29888685-pm6kg"]
Oct 30 00:45:21.524318 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:21.524256    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-lhfxq" podStartSLOduration=1.524234546 podStartE2EDuration="1.524234546s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:45:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:45:21.520407467 +0000 UTC m=+817.136840347" watchObservedRunningTime="2026-10-30 00:45:21.524234546 +0000 UTC m=+817.140667427"
Oct 30 00:45:21.563778 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:21.563743    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-021d6d-29888685-nftp2"]
Oct 30 00:45:21.566162 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:21.566133    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-021d6d-29888685-nftp2"]
Oct 30 00:45:21.636658 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:21.636634    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888685-m7pp6"
Oct 30 00:45:21.779808 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:21.779769    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18a26407-78ee-4e2d-a863-f99a13c924e5-kube-api-access-zrptf\" (UniqueName: \"kubernetes.io/projected/18a26407-78ee-4e2d-a863-f99a13c924e5-kube-api-access-zrptf\") pod \"18a26407-78ee-4e2d-a863-f99a13c924e5\" (UID: \"18a26407-78ee-4e2d-a863-f99a13c924e5\") "
Oct 30 00:45:21.780646 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:21.780607    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18a26407-78ee-4e2d-a863-f99a13c924e5-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/18a26407-78ee-4e2d-a863-f99a13c924e5-maas-api-service-ca\") pod \"18a26407-78ee-4e2d-a863-f99a13c924e5\" (UID: \"18a26407-78ee-4e2d-a863-f99a13c924e5\") "
Oct 30 00:45:21.789403 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:21.786071    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18a26407-78ee-4e2d-a863-f99a13c924e5-maas-api-service-ca" pod "18a26407-78ee-4e2d-a863-f99a13c924e5" (UID: "18a26407-78ee-4e2d-a863-f99a13c924e5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:45:21.789559 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:21.789517    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18a26407-78ee-4e2d-a863-f99a13c924e5-kube-api-access-zrptf" pod "18a26407-78ee-4e2d-a863-f99a13c924e5" (UID: "18a26407-78ee-4e2d-a863-f99a13c924e5"). InnerVolumeSpecName "kube-api-access-zrptf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:45:21.888208 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:21.888167    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zrptf\" (UniqueName: \"kubernetes.io/projected/18a26407-78ee-4e2d-a863-f99a13c924e5-kube-api-access-zrptf\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:45:21.888208 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:21.888204    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/18a26407-78ee-4e2d-a863-f99a13c924e5-maas-api-service-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:45:22.424199 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:22.424173    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888685-m7pp6"
Oct 30 00:45:22.424199 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:22.424198    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888685-m7pp6" event={"ID":"18a26407-78ee-4e2d-a863-f99a13c924e5","Type":"ContainerDied","Data":"009fcdce86982d124201d62a74de42fa8640c778fee04aefbd6b9e03fa2f12f2"}
Oct 30 00:45:22.424702 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:22.424267    5002 scope.go:122] "RemoveContainer" containerID="1619b4d1ae9f5628c5fabbb394fd971425326080d4783f20f852ff03871d71ef"
Oct 30 00:45:22.479437 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:22.470753    5002 patch_prober.go:29] interesting pod/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-lhfxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body=
Oct 30 00:45:22.479437 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:22.470814    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-lhfxq" podUID="543ddf6f-c221-48c0-b5d5-881c23397832" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused"
Oct 30 00:45:22.487623 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:22.487595    5002 scope.go:122] "RemoveContainer" containerID="32d819c065745e9d39a48e958bcba5031fd0371d85a1525636580556bf030907"
Oct 30 00:45:22.496776 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:22.496745    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7941ea9e-5393-4ab4-ad00-a4b1cd42cd38" path="/var/lib/kubelet/pods/7941ea9e-5393-4ab4-ad00-a4b1cd42cd38/volumes"
Oct 30 00:45:22.497157 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:22.497139    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e453da6f-d3ef-4d30-9461-08952eb347fd" path="/var/lib/kubelet/pods/e453da6f-d3ef-4d30-9461-08952eb347fd/volumes"
Oct 30 00:45:22.497484 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:22.497467    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888685-m7pp6"]
Oct 30 00:45:22.498671 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:22.498639    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888685-m7pp6"]
Oct 30 00:45:23.433557 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:23.433525    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6f4fba-29888685-rrxtq" event={"ID":"db996a04-ded3-4343-bc10-236b38d60130","Type":"ContainerStarted","Data":"0b67cba57010eae3f9f93d90d0662c4866b6c4103eec45bfb6140c8e9a7be082"}
Oct 30 00:45:23.474167 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:23.467805    5002 patch_prober.go:29] interesting pod/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-lhfxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body=
Oct 30 00:45:23.474167 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:23.467879    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-lhfxq" podUID="543ddf6f-c221-48c0-b5d5-881c23397832" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused"
Oct 30 00:45:24.053406 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:24.050482    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7d8z4"]
Oct 30 00:45:24.442917 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:24.442877    5002 generic.go:356] "Generic (PLEG): container finished" podID="db996a04-ded3-4343-bc10-236b38d60130" containerID="0b67cba57010eae3f9f93d90d0662c4866b6c4103eec45bfb6140c8e9a7be082" exitCode=60
Oct 30 00:45:24.443375 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:24.442941    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6f4fba-29888685-rrxtq" event={"ID":"db996a04-ded3-4343-bc10-236b38d60130","Type":"ContainerDied","Data":"0b67cba57010eae3f9f93d90d0662c4866b6c4103eec45bfb6140c8e9a7be082"}
Oct 30 00:45:24.443375 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:24.443003    5002 scope.go:122] "RemoveContainer" containerID="32d819c065745e9d39a48e958bcba5031fd0371d85a1525636580556bf030907"
Oct 30 00:45:24.443530 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:24.443368    5002 scope.go:122] "RemoveContainer" containerID="0b67cba57010eae3f9f93d90d0662c4866b6c4103eec45bfb6140c8e9a7be082"
Oct 30 00:45:24.443716 ip-10-0-128-7 kubenswrapper[5002]: E1030 00:45:24.443690    5002 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-6f4fba-29888685-rrxtq_odh-ai-gateway-infra(db996a04-ded3-4343-bc10-236b38d60130)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6f4fba-29888685-rrxtq" podUID="db996a04-ded3-4343-bc10-236b38d60130"
Oct 30 00:45:24.475526 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:24.475491    5002 patch_prober.go:29] interesting pod/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-lhfxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body=
Oct 30 00:45:24.475891 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:24.475858    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-lhfxq" podUID="543ddf6f-c221-48c0-b5d5-881c23397832" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused"
Oct 30 00:45:24.497020 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:24.496976    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18a26407-78ee-4e2d-a863-f99a13c924e5" path="/var/lib/kubelet/pods/18a26407-78ee-4e2d-a863-f99a13c924e5/volumes"
Oct 30 00:45:24.499504 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:24.499480    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6f4fba-29888685-rrxtq"]
Oct 30 00:45:25.481403 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:25.467120    5002 patch_prober.go:29] interesting pod/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-lhfxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body=
Oct 30 00:45:25.481403 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:25.467196    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-lhfxq" podUID="543ddf6f-c221-48c0-b5d5-881c23397832" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused"
Oct 30 00:45:26.065616 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:26.065580    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6f4fba-29888685-rrxtq"
Oct 30 00:45:26.175988 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:26.175898    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db996a04-ded3-4343-bc10-236b38d60130-kube-api-access-t5jjm\" (UniqueName: \"kubernetes.io/projected/db996a04-ded3-4343-bc10-236b38d60130-kube-api-access-t5jjm\") pod \"db996a04-ded3-4343-bc10-236b38d60130\" (UID: \"db996a04-ded3-4343-bc10-236b38d60130\") "
Oct 30 00:45:26.176335 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:26.176296    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db996a04-ded3-4343-bc10-236b38d60130-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/db996a04-ded3-4343-bc10-236b38d60130-maas-api-service-ca\") pod \"db996a04-ded3-4343-bc10-236b38d60130\" (UID: \"db996a04-ded3-4343-bc10-236b38d60130\") "
Oct 30 00:45:26.177375 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:26.177341    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db996a04-ded3-4343-bc10-236b38d60130-maas-api-service-ca" pod "db996a04-ded3-4343-bc10-236b38d60130" (UID: "db996a04-ded3-4343-bc10-236b38d60130"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:45:26.251732 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:26.251678    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db996a04-ded3-4343-bc10-236b38d60130-kube-api-access-t5jjm" pod "db996a04-ded3-4343-bc10-236b38d60130" (UID: "db996a04-ded3-4343-bc10-236b38d60130"). InnerVolumeSpecName "kube-api-access-t5jjm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:45:26.401569 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:26.363319    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/db996a04-ded3-4343-bc10-236b38d60130-maas-api-service-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:45:26.478976 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:26.472124    5002 patch_prober.go:29] interesting pod/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-lhfxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body=
Oct 30 00:45:26.478976 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:26.472180    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-lhfxq" podUID="543ddf6f-c221-48c0-b5d5-881c23397832" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused"
Oct 30 00:45:26.504453 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:26.504420    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t5jjm\" (UniqueName: \"kubernetes.io/projected/db996a04-ded3-4343-bc10-236b38d60130-kube-api-access-t5jjm\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:45:26.513116 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:26.513079    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6f4fba-29888685-rrxtq" event={"ID":"db996a04-ded3-4343-bc10-236b38d60130","Type":"ContainerDied","Data":"4e62a459ecfb90a9de59b5ddaac71774339df92d9037c129dd47d53a714feeb8"}
Oct 30 00:45:26.513440 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:26.513422    5002 scope.go:122] "RemoveContainer" containerID="0b67cba57010eae3f9f93d90d0662c4866b6c4103eec45bfb6140c8e9a7be082"
Oct 30 00:45:26.513598 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:26.513342    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6f4fba-29888685-rrxtq"
Oct 30 00:45:26.607538 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:26.607464    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6f4fba-29888685-rrxtq"]
Oct 30 00:45:26.611087 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:26.611023    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6f4fba-29888685-rrxtq"]
Oct 30 00:45:27.469801 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:27.469762    5002 patch_prober.go:29] interesting pod/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-lhfxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body=
Oct 30 00:45:27.470083 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:27.470050    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-lhfxq" podUID="543ddf6f-c221-48c0-b5d5-881c23397832" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused"
Oct 30 00:45:28.510461 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:28.510421    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db996a04-ded3-4343-bc10-236b38d60130" path="/var/lib/kubelet/pods/db996a04-ded3-4343-bc10-236b38d60130/volumes"
Oct 30 00:45:29.240705 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:29.240663    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-lhfxq"
Oct 30 00:45:29.241224 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:29.241189    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-lhfxq"
Oct 30 00:45:29.250342 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:29.250277    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-lhfxq"
Oct 30 00:45:33.427390 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:33.427345    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0aed46c7-xvzsh"]
Oct 30 00:45:33.427844 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:33.427828    5002 cpu_manager.go:414] "containerMap: removing container" podUID="18a26407-78ee-4e2d-a863-f99a13c924e5" containerName="cleanup"
Oct 30 00:45:33.427898 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:33.427844    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18a26407-78ee-4e2d-a863-f99a13c924e5" containerName="cleanup"
Oct 30 00:45:33.427898 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:33.427855    5002 cpu_manager.go:414] "containerMap: removing container" podUID="7941ea9e-5393-4ab4-ad00-a4b1cd42cd38" containerName="cleanup"
Oct 30 00:45:33.427898 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:33.427860    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7941ea9e-5393-4ab4-ad00-a4b1cd42cd38" containerName="cleanup"
Oct 30 00:45:33.427898 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:33.427868    5002 cpu_manager.go:414] "containerMap: removing container" podUID="7941ea9e-5393-4ab4-ad00-a4b1cd42cd38" containerName="cleanup"
Oct 30 00:45:33.427898 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:33.427872    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7941ea9e-5393-4ab4-ad00-a4b1cd42cd38" containerName="cleanup"
Oct 30 00:45:33.427898 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:33.427885    5002 cpu_manager.go:414] "containerMap: removing container" podUID="e453da6f-d3ef-4d30-9461-08952eb347fd" containerName="cleanup"
Oct 30 00:45:33.427898 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:33.427891    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e453da6f-d3ef-4d30-9461-08952eb347fd" containerName="cleanup"
Oct 30 00:45:33.427898 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:33.427897    5002 cpu_manager.go:414] "containerMap: removing container" podUID="18a26407-78ee-4e2d-a863-f99a13c924e5" containerName="cleanup"
Oct 30 00:45:33.427898 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:33.427902    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18a26407-78ee-4e2d-a863-f99a13c924e5" containerName="cleanup"
Oct 30 00:45:33.428293 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:33.427908    5002 cpu_manager.go:414] "containerMap: removing container" podUID="db996a04-ded3-4343-bc10-236b38d60130" containerName="cleanup"
Oct 30 00:45:33.428293 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:33.427912    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db996a04-ded3-4343-bc10-236b38d60130" containerName="cleanup"
Oct 30 00:45:33.428293 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:33.427922    5002 cpu_manager.go:414] "containerMap: removing container" podUID="e453da6f-d3ef-4d30-9461-08952eb347fd" containerName="cleanup"
Oct 30 00:45:33.428293 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:33.427926    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e453da6f-d3ef-4d30-9461-08952eb347fd" containerName="cleanup"
Oct 30 00:45:33.428293 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:33.427934    5002 cpu_manager.go:414] "containerMap: removing container" podUID="7941ea9e-5393-4ab4-ad00-a4b1cd42cd38" containerName="cleanup"
Oct 30 00:45:33.428293 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:33.427937    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7941ea9e-5393-4ab4-ad00-a4b1cd42cd38" containerName="cleanup"
Oct 30 00:45:33.428293 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:33.427944    5002 cpu_manager.go:414] "containerMap: removing container" podUID="db996a04-ded3-4343-bc10-236b38d60130" containerName="cleanup"
Oct 30 00:45:33.428293 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:33.427948    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db996a04-ded3-4343-bc10-236b38d60130" containerName="cleanup"
Oct 30 00:45:33.428293 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:33.427954    5002 cpu_manager.go:414] "containerMap: removing container" podUID="db996a04-ded3-4343-bc10-236b38d60130" containerName="cleanup"
Oct 30 00:45:33.428293 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:33.427959    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db996a04-ded3-4343-bc10-236b38d60130" containerName="cleanup"
Oct 30 00:45:33.428293 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:33.427967    5002 cpu_manager.go:414] "containerMap: removing container" podUID="e453da6f-d3ef-4d30-9461-08952eb347fd" containerName="cleanup"
Oct 30 00:45:33.428293 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:33.427973    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e453da6f-d3ef-4d30-9461-08952eb347fd" containerName="cleanup"
Oct 30 00:45:33.428293 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:33.428040    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="e453da6f-d3ef-4d30-9461-08952eb347fd" containerName="cleanup"
Oct 30 00:45:33.428293 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:33.428047    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="18a26407-78ee-4e2d-a863-f99a13c924e5" containerName="cleanup"
Oct 30 00:45:33.428293 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:33.428052    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="db996a04-ded3-4343-bc10-236b38d60130" containerName="cleanup"
Oct 30 00:45:33.428293 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:33.428059    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="7941ea9e-5393-4ab4-ad00-a4b1cd42cd38" containerName="cleanup"
Oct 30 00:45:33.428293 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:33.428064    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="db996a04-ded3-4343-bc10-236b38d60130" containerName="cleanup"
Oct 30 00:45:33.428293 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:33.428069    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="e453da6f-d3ef-4d30-9461-08952eb347fd" containerName="cleanup"
Oct 30 00:45:33.428293 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:33.428079    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="18a26407-78ee-4e2d-a863-f99a13c924e5" containerName="cleanup"
Oct 30 00:45:33.428293 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:33.428084    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="7941ea9e-5393-4ab4-ad00-a4b1cd42cd38" containerName="cleanup"
Oct 30 00:45:33.431310 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:33.431288    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0aed46c7-xvzsh"
Oct 30 00:45:33.462559 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:33.462525    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 00:45:33.462777 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:33.462538    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5phsh\""
Oct 30 00:45:33.462884 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:33.462737    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 00:45:33.462884 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:33.462786    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 00:45:33.465056 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:33.465031    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0aed46c7-xvzsh"]
Oct 30 00:45:33.487979 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:33.487948    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flf22\" (UniqueName: \"kubernetes.io/projected/c7c497d9-dbad-44f0-8e38-8f275d3f0410-kube-api-access-flf22\") pod \"maas-api-revoke-sub-0aed46c7-xvzsh\" (UID: \"c7c497d9-dbad-44f0-8e38-8f275d3f0410\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0aed46c7-xvzsh"
Oct 30 00:45:33.488102 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:33.488001    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7c497d9-dbad-44f0-8e38-8f275d3f0410-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0aed46c7-xvzsh\" (UID: \"c7c497d9-dbad-44f0-8e38-8f275d3f0410\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0aed46c7-xvzsh"
Oct 30 00:45:33.488275 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:33.488247    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7c497d9-dbad-44f0-8e38-8f275d3f0410-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0aed46c7-xvzsh\" (UID: \"c7c497d9-dbad-44f0-8e38-8f275d3f0410\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0aed46c7-xvzsh"
Oct 30 00:45:33.589792 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:33.589755    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flf22\" (UniqueName: \"kubernetes.io/projected/c7c497d9-dbad-44f0-8e38-8f275d3f0410-kube-api-access-flf22\") pod \"maas-api-revoke-sub-0aed46c7-xvzsh\" (UID: \"c7c497d9-dbad-44f0-8e38-8f275d3f0410\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0aed46c7-xvzsh"
Oct 30 00:45:33.589984 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:33.589809    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7c497d9-dbad-44f0-8e38-8f275d3f0410-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0aed46c7-xvzsh\" (UID: \"c7c497d9-dbad-44f0-8e38-8f275d3f0410\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0aed46c7-xvzsh"
Oct 30 00:45:33.590040 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:33.589982    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7c497d9-dbad-44f0-8e38-8f275d3f0410-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0aed46c7-xvzsh\" (UID: \"c7c497d9-dbad-44f0-8e38-8f275d3f0410\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0aed46c7-xvzsh"
Oct 30 00:45:33.590637 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:33.590614    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7c497d9-dbad-44f0-8e38-8f275d3f0410-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0aed46c7-xvzsh\" (UID: \"c7c497d9-dbad-44f0-8e38-8f275d3f0410\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0aed46c7-xvzsh"
Oct 30 00:45:33.592952 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:33.592923    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7c497d9-dbad-44f0-8e38-8f275d3f0410-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0aed46c7-xvzsh\" (UID: \"c7c497d9-dbad-44f0-8e38-8f275d3f0410\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0aed46c7-xvzsh"
Oct 30 00:45:33.614481 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:33.614452    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-flf22\" (UniqueName: \"kubernetes.io/projected/c7c497d9-dbad-44f0-8e38-8f275d3f0410-kube-api-access-flf22\") pod \"maas-api-revoke-sub-0aed46c7-xvzsh\" (UID: \"c7c497d9-dbad-44f0-8e38-8f275d3f0410\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0aed46c7-xvzsh"
Oct 30 00:45:33.745241 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:33.745209    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0aed46c7-xvzsh"
Oct 30 00:45:34.046021 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:34.045990    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0aed46c7-xvzsh"]
Oct 30 00:45:34.046815 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:45:34.046777    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7c497d9_dbad_44f0_8e38_8f275d3f0410.slice/crio-f9118a74804f003d23853d77b9bb45040be489ad25b15c1b1baf40660ac6bc31 WatchSource:0}: Error finding container f9118a74804f003d23853d77b9bb45040be489ad25b15c1b1baf40660ac6bc31: Status 404 returned error can't find the container with id f9118a74804f003d23853d77b9bb45040be489ad25b15c1b1baf40660ac6bc31
Oct 30 00:45:34.568977 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:34.568937    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0aed46c7-xvzsh" event={"ID":"c7c497d9-dbad-44f0-8e38-8f275d3f0410","Type":"ContainerStarted","Data":"50ff46e8a04b45d0ba9be413e88b04df2509ff644b741adb24bd447c5d17a344"}
Oct 30 00:45:34.569353 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:34.569004    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0aed46c7-xvzsh" event={"ID":"c7c497d9-dbad-44f0-8e38-8f275d3f0410","Type":"ContainerStarted","Data":"f9118a74804f003d23853d77b9bb45040be489ad25b15c1b1baf40660ac6bc31"}
Oct 30 00:45:34.613643 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:34.613533    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0aed46c7-xvzsh" podStartSLOduration=1.61351461 podStartE2EDuration="1.61351461s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:45:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:45:34.61164052 +0000 UTC m=+830.228073425" watchObservedRunningTime="2026-10-30 00:45:34.61351461 +0000 UTC m=+830.229947489"
Oct 30 00:45:35.577097 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:35.577059    5002 generic.go:356] "Generic (PLEG): container finished" podID="c7c497d9-dbad-44f0-8e38-8f275d3f0410" containerID="50ff46e8a04b45d0ba9be413e88b04df2509ff644b741adb24bd447c5d17a344" exitCode=0
Oct 30 00:45:35.577541 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:35.577129    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0aed46c7-xvzsh" event={"ID":"c7c497d9-dbad-44f0-8e38-8f275d3f0410","Type":"ContainerDied","Data":"50ff46e8a04b45d0ba9be413e88b04df2509ff644b741adb24bd447c5d17a344"}
Oct 30 00:45:36.744852 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:36.744824    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0aed46c7-xvzsh"
Oct 30 00:45:36.826584 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:36.826538    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7c497d9-dbad-44f0-8e38-8f275d3f0410-kube-api-access-flf22\" (UniqueName: \"kubernetes.io/projected/c7c497d9-dbad-44f0-8e38-8f275d3f0410-kube-api-access-flf22\") pod \"c7c497d9-dbad-44f0-8e38-8f275d3f0410\" (UID: \"c7c497d9-dbad-44f0-8e38-8f275d3f0410\") "
Oct 30 00:45:36.826770 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:36.826686    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c7c497d9-dbad-44f0-8e38-8f275d3f0410-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7c497d9-dbad-44f0-8e38-8f275d3f0410-maas-api-xks-ca\") pod \"c7c497d9-dbad-44f0-8e38-8f275d3f0410\" (UID: \"c7c497d9-dbad-44f0-8e38-8f275d3f0410\") "
Oct 30 00:45:36.826770 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:36.826742    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7c497d9-dbad-44f0-8e38-8f275d3f0410-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7c497d9-dbad-44f0-8e38-8f275d3f0410-maas-api-openshift-ca\") pod \"c7c497d9-dbad-44f0-8e38-8f275d3f0410\" (UID: \"c7c497d9-dbad-44f0-8e38-8f275d3f0410\") "
Oct 30 00:45:36.827135 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:36.827110    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7c497d9-dbad-44f0-8e38-8f275d3f0410-maas-api-openshift-ca" pod "c7c497d9-dbad-44f0-8e38-8f275d3f0410" (UID: "c7c497d9-dbad-44f0-8e38-8f275d3f0410"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:45:36.829334 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:36.829311    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7c497d9-dbad-44f0-8e38-8f275d3f0410-kube-api-access-flf22" pod "c7c497d9-dbad-44f0-8e38-8f275d3f0410" (UID: "c7c497d9-dbad-44f0-8e38-8f275d3f0410"). InnerVolumeSpecName "kube-api-access-flf22". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:45:36.829519 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:36.829393    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7c497d9-dbad-44f0-8e38-8f275d3f0410-maas-api-xks-ca" pod "c7c497d9-dbad-44f0-8e38-8f275d3f0410" (UID: "c7c497d9-dbad-44f0-8e38-8f275d3f0410"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:45:36.928456 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:36.928411    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7c497d9-dbad-44f0-8e38-8f275d3f0410-maas-api-xks-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:45:36.928456 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:36.928451    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7c497d9-dbad-44f0-8e38-8f275d3f0410-maas-api-openshift-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:45:36.928657 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:36.928465    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-flf22\" (UniqueName: \"kubernetes.io/projected/c7c497d9-dbad-44f0-8e38-8f275d3f0410-kube-api-access-flf22\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:45:37.592365 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:37.592335    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0aed46c7-xvzsh"
Oct 30 00:45:37.592548 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:37.592338    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0aed46c7-xvzsh" event={"ID":"c7c497d9-dbad-44f0-8e38-8f275d3f0410","Type":"ContainerDied","Data":"f9118a74804f003d23853d77b9bb45040be489ad25b15c1b1baf40660ac6bc31"}
Oct 30 00:45:37.592548 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:37.592443    5002 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f9118a74804f003d23853d77b9bb45040be489ad25b15c1b1baf40660ac6bc31"
Oct 30 00:45:42.866361 ip-10-0-128-7 kubenswrapper[5002]: E1030 00:45:42.866327    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbjdq" containerName="extractor"
Oct 30 00:45:51.089029 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:51.088994    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-52650f/distinct-w0-52650f-kserve-d85899b5-vlhps"]
Oct 30 00:45:51.089677 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:51.089654    5002 cpu_manager.go:414] "containerMap: removing container" podUID="18a26407-78ee-4e2d-a863-f99a13c924e5" containerName="cleanup"
Oct 30 00:45:51.089764 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:51.089679    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18a26407-78ee-4e2d-a863-f99a13c924e5" containerName="cleanup"
Oct 30 00:45:51.089764 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:51.089744    5002 cpu_manager.go:414] "containerMap: removing container" podUID="c7c497d9-dbad-44f0-8e38-8f275d3f0410" containerName="revoke-keys"
Oct 30 00:45:51.089764 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:51.089753    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7c497d9-dbad-44f0-8e38-8f275d3f0410" containerName="revoke-keys"
Oct 30 00:45:51.089913 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:51.089878    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="18a26407-78ee-4e2d-a863-f99a13c924e5" containerName="cleanup"
Oct 30 00:45:51.089913 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:51.089893    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7c497d9-dbad-44f0-8e38-8f275d3f0410" containerName="revoke-keys"
Oct 30 00:45:51.089913 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:51.089907    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="7941ea9e-5393-4ab4-ad00-a4b1cd42cd38" containerName="cleanup"
Oct 30 00:45:51.090056 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:51.089917    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="db996a04-ded3-4343-bc10-236b38d60130" containerName="cleanup"
Oct 30 00:45:51.090056 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:51.089925    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="e453da6f-d3ef-4d30-9461-08952eb347fd" containerName="cleanup"
Oct 30 00:45:51.093000 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:51.092976    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-52650f/distinct-w0-52650f-kserve-d85899b5-vlhps"
Oct 30 00:45:51.117529 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:51.117496    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-52650f/distinct-w0-52650f-kserve-d85899b5-vlhps"]
Oct 30 00:45:51.123118 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:51.123090    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea1850d9-abba-4977-a540-f6e0eaefb17d-home\") pod \"distinct-w0-52650f-kserve-d85899b5-vlhps\" (UID: \"ea1850d9-abba-4977-a540-f6e0eaefb17d\") " pod="e2e-models-e2e-worker-w0-52650f/distinct-w0-52650f-kserve-d85899b5-vlhps"
Oct 30 00:45:51.123663 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:51.123153    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea1850d9-abba-4977-a540-f6e0eaefb17d-tls-certs\") pod \"distinct-w0-52650f-kserve-d85899b5-vlhps\" (UID: \"ea1850d9-abba-4977-a540-f6e0eaefb17d\") " pod="e2e-models-e2e-worker-w0-52650f/distinct-w0-52650f-kserve-d85899b5-vlhps"
Oct 30 00:45:51.123663 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:51.123630    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea1850d9-abba-4977-a540-f6e0eaefb17d-tmp-dir\") pod \"distinct-w0-52650f-kserve-d85899b5-vlhps\" (UID: \"ea1850d9-abba-4977-a540-f6e0eaefb17d\") " pod="e2e-models-e2e-worker-w0-52650f/distinct-w0-52650f-kserve-d85899b5-vlhps"
Oct 30 00:45:51.123791 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:51.123698    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hf95\" (UniqueName: \"kubernetes.io/projected/ea1850d9-abba-4977-a540-f6e0eaefb17d-kube-api-access-8hf95\") pod \"distinct-w0-52650f-kserve-d85899b5-vlhps\" (UID: \"ea1850d9-abba-4977-a540-f6e0eaefb17d\") " pod="e2e-models-e2e-worker-w0-52650f/distinct-w0-52650f-kserve-d85899b5-vlhps"
Oct 30 00:45:51.123791 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:51.123736    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea1850d9-abba-4977-a540-f6e0eaefb17d-model-cache\") pod \"distinct-w0-52650f-kserve-d85899b5-vlhps\" (UID: \"ea1850d9-abba-4977-a540-f6e0eaefb17d\") " pod="e2e-models-e2e-worker-w0-52650f/distinct-w0-52650f-kserve-d85899b5-vlhps"
Oct 30 00:45:51.123791 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:51.123779    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea1850d9-abba-4977-a540-f6e0eaefb17d-dshm\") pod \"distinct-w0-52650f-kserve-d85899b5-vlhps\" (UID: \"ea1850d9-abba-4977-a540-f6e0eaefb17d\") " pod="e2e-models-e2e-worker-w0-52650f/distinct-w0-52650f-kserve-d85899b5-vlhps"
Oct 30 00:45:51.127550 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:51.127529    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-52650f\"/\"distinct-w0-52650f-kserve-self-signed-certs\""
Oct 30 00:45:51.224742 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:51.224706    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea1850d9-abba-4977-a540-f6e0eaefb17d-home\") pod \"distinct-w0-52650f-kserve-d85899b5-vlhps\" (UID: \"ea1850d9-abba-4977-a540-f6e0eaefb17d\") " pod="e2e-models-e2e-worker-w0-52650f/distinct-w0-52650f-kserve-d85899b5-vlhps"
Oct 30 00:45:51.224942 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:51.224761    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea1850d9-abba-4977-a540-f6e0eaefb17d-tls-certs\") pod \"distinct-w0-52650f-kserve-d85899b5-vlhps\" (UID: \"ea1850d9-abba-4977-a540-f6e0eaefb17d\") " pod="e2e-models-e2e-worker-w0-52650f/distinct-w0-52650f-kserve-d85899b5-vlhps"
Oct 30 00:45:51.224942 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:51.224814    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea1850d9-abba-4977-a540-f6e0eaefb17d-tmp-dir\") pod \"distinct-w0-52650f-kserve-d85899b5-vlhps\" (UID: \"ea1850d9-abba-4977-a540-f6e0eaefb17d\") " pod="e2e-models-e2e-worker-w0-52650f/distinct-w0-52650f-kserve-d85899b5-vlhps"
Oct 30 00:45:51.224942 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:51.224843    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hf95\" (UniqueName: \"kubernetes.io/projected/ea1850d9-abba-4977-a540-f6e0eaefb17d-kube-api-access-8hf95\") pod \"distinct-w0-52650f-kserve-d85899b5-vlhps\" (UID: \"ea1850d9-abba-4977-a540-f6e0eaefb17d\") " pod="e2e-models-e2e-worker-w0-52650f/distinct-w0-52650f-kserve-d85899b5-vlhps"
Oct 30 00:45:51.225168 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:51.225130    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea1850d9-abba-4977-a540-f6e0eaefb17d-model-cache\") pod \"distinct-w0-52650f-kserve-d85899b5-vlhps\" (UID: \"ea1850d9-abba-4977-a540-f6e0eaefb17d\") " pod="e2e-models-e2e-worker-w0-52650f/distinct-w0-52650f-kserve-d85899b5-vlhps"
Oct 30 00:45:51.225580 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:51.225556    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea1850d9-abba-4977-a540-f6e0eaefb17d-dshm\") pod \"distinct-w0-52650f-kserve-d85899b5-vlhps\" (UID: \"ea1850d9-abba-4977-a540-f6e0eaefb17d\") " pod="e2e-models-e2e-worker-w0-52650f/distinct-w0-52650f-kserve-d85899b5-vlhps"
Oct 30 00:45:51.225685 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:51.225182    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea1850d9-abba-4977-a540-f6e0eaefb17d-home\") pod \"distinct-w0-52650f-kserve-d85899b5-vlhps\" (UID: \"ea1850d9-abba-4977-a540-f6e0eaefb17d\") " pod="e2e-models-e2e-worker-w0-52650f/distinct-w0-52650f-kserve-d85899b5-vlhps"
Oct 30 00:45:51.225685 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:51.225488    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea1850d9-abba-4977-a540-f6e0eaefb17d-model-cache\") pod \"distinct-w0-52650f-kserve-d85899b5-vlhps\" (UID: \"ea1850d9-abba-4977-a540-f6e0eaefb17d\") " pod="e2e-models-e2e-worker-w0-52650f/distinct-w0-52650f-kserve-d85899b5-vlhps"
Oct 30 00:45:51.225685 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:51.225190    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea1850d9-abba-4977-a540-f6e0eaefb17d-tmp-dir\") pod \"distinct-w0-52650f-kserve-d85899b5-vlhps\" (UID: \"ea1850d9-abba-4977-a540-f6e0eaefb17d\") " pod="e2e-models-e2e-worker-w0-52650f/distinct-w0-52650f-kserve-d85899b5-vlhps"
Oct 30 00:45:51.227569 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:51.227544    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea1850d9-abba-4977-a540-f6e0eaefb17d-tls-certs\") pod \"distinct-w0-52650f-kserve-d85899b5-vlhps\" (UID: \"ea1850d9-abba-4977-a540-f6e0eaefb17d\") " pod="e2e-models-e2e-worker-w0-52650f/distinct-w0-52650f-kserve-d85899b5-vlhps"
Oct 30 00:45:51.227818 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:51.227799    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea1850d9-abba-4977-a540-f6e0eaefb17d-dshm\") pod \"distinct-w0-52650f-kserve-d85899b5-vlhps\" (UID: \"ea1850d9-abba-4977-a540-f6e0eaefb17d\") " pod="e2e-models-e2e-worker-w0-52650f/distinct-w0-52650f-kserve-d85899b5-vlhps"
Oct 30 00:45:51.250508 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:51.250474    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hf95\" (UniqueName: \"kubernetes.io/projected/ea1850d9-abba-4977-a540-f6e0eaefb17d-kube-api-access-8hf95\") pod \"distinct-w0-52650f-kserve-d85899b5-vlhps\" (UID: \"ea1850d9-abba-4977-a540-f6e0eaefb17d\") " pod="e2e-models-e2e-worker-w0-52650f/distinct-w0-52650f-kserve-d85899b5-vlhps"
Oct 30 00:45:51.427973 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:51.427886    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-52650f/distinct-w0-52650f-kserve-d85899b5-vlhps"
Oct 30 00:45:51.742388 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:51.742349    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-52650f/distinct-w0-52650f-kserve-d85899b5-vlhps"]
Oct 30 00:45:51.742507 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:45:51.742356    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podea1850d9_abba_4977_a540_f6e0eaefb17d.slice/crio-7344b1bfd00b82bbb1533f91c8b6f8a119d3010d2f8876fed4aacf2a5fd683c3 WatchSource:0}: Error finding container 7344b1bfd00b82bbb1533f91c8b6f8a119d3010d2f8876fed4aacf2a5fd683c3: Status 404 returned error can't find the container with id 7344b1bfd00b82bbb1533f91c8b6f8a119d3010d2f8876fed4aacf2a5fd683c3
Oct 30 00:45:52.693313 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:52.693280    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-52650f/distinct-w0-52650f-kserve-d85899b5-vlhps" event={"ID":"ea1850d9-abba-4977-a540-f6e0eaefb17d","Type":"ContainerStarted","Data":"11f466b38ab95d14798e8bdccca6fbc6dbe5350bec4ca0c79a57bf6d8942c6c4"}
Oct 30 00:45:52.693313 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:52.693323    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-52650f/distinct-w0-52650f-kserve-d85899b5-vlhps" event={"ID":"ea1850d9-abba-4977-a540-f6e0eaefb17d","Type":"ContainerStarted","Data":"7344b1bfd00b82bbb1533f91c8b6f8a119d3010d2f8876fed4aacf2a5fd683c3"}
Oct 30 00:45:52.738863 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:52.738811    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-52650f/distinct-w0-52650f-kserve-d85899b5-vlhps" podStartSLOduration=1.7387978849999999 podStartE2EDuration="1.738797885s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:45:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:45:52.737492818 +0000 UTC m=+848.353925697" watchObservedRunningTime="2026-10-30 00:45:52.738797885 +0000 UTC m=+848.355230764"
Oct 30 00:45:52.914262 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:52.914226    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7d8z4"]
Oct 30 00:45:53.428356 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:53.428324    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-52650f/distinct-w0-52650f-kserve-d85899b5-vlhps"
Oct 30 00:45:54.062767 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:54.062732    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-1a11cf/distinct-w2-1a11cf-kserve-854cf4cdf8-cddw6"]
Oct 30 00:45:54.066799 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:54.066776    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-1a11cf/distinct-w2-1a11cf-kserve-854cf4cdf8-cddw6"
Oct 30 00:45:54.091424 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:54.088046    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-1a11cf/distinct-w2-1a11cf-kserve-854cf4cdf8-cddw6"]
Oct 30 00:45:54.099602 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:54.099576    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-1a11cf\"/\"openshift-service-ca.crt\""
Oct 30 00:45:54.099747 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:54.099633    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-1a11cf\"/\"distinct-w2-1a11cf-kserve-self-signed-certs\""
Oct 30 00:45:54.099812 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:54.099694    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-1a11cf\"/\"kube-root-ca.crt\""
Oct 30 00:45:54.099812 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:54.099753    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-1a11cf\"/\"default-dockercfg-v7tcc\""
Oct 30 00:45:54.156072 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:54.156036    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32a03478-46eb-4363-97b6-51a32c4099c7-tls-certs\") pod \"distinct-w2-1a11cf-kserve-854cf4cdf8-cddw6\" (UID: \"32a03478-46eb-4363-97b6-51a32c4099c7\") " pod="e2e-models-e2e-worker-w2-1a11cf/distinct-w2-1a11cf-kserve-854cf4cdf8-cddw6"
Oct 30 00:45:54.156258 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:54.156089    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32a03478-46eb-4363-97b6-51a32c4099c7-home\") pod \"distinct-w2-1a11cf-kserve-854cf4cdf8-cddw6\" (UID: \"32a03478-46eb-4363-97b6-51a32c4099c7\") " pod="e2e-models-e2e-worker-w2-1a11cf/distinct-w2-1a11cf-kserve-854cf4cdf8-cddw6"
Oct 30 00:45:54.156258 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:54.156212    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32a03478-46eb-4363-97b6-51a32c4099c7-tmp-dir\") pod \"distinct-w2-1a11cf-kserve-854cf4cdf8-cddw6\" (UID: \"32a03478-46eb-4363-97b6-51a32c4099c7\") " pod="e2e-models-e2e-worker-w2-1a11cf/distinct-w2-1a11cf-kserve-854cf4cdf8-cddw6"
Oct 30 00:45:54.156371 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:54.156320    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8t8n9\" (UniqueName: \"kubernetes.io/projected/32a03478-46eb-4363-97b6-51a32c4099c7-kube-api-access-8t8n9\") pod \"distinct-w2-1a11cf-kserve-854cf4cdf8-cddw6\" (UID: \"32a03478-46eb-4363-97b6-51a32c4099c7\") " pod="e2e-models-e2e-worker-w2-1a11cf/distinct-w2-1a11cf-kserve-854cf4cdf8-cddw6"
Oct 30 00:45:54.156467 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:54.156438    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32a03478-46eb-4363-97b6-51a32c4099c7-dshm\") pod \"distinct-w2-1a11cf-kserve-854cf4cdf8-cddw6\" (UID: \"32a03478-46eb-4363-97b6-51a32c4099c7\") " pod="e2e-models-e2e-worker-w2-1a11cf/distinct-w2-1a11cf-kserve-854cf4cdf8-cddw6"
Oct 30 00:45:54.156528 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:54.156464    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32a03478-46eb-4363-97b6-51a32c4099c7-model-cache\") pod \"distinct-w2-1a11cf-kserve-854cf4cdf8-cddw6\" (UID: \"32a03478-46eb-4363-97b6-51a32c4099c7\") " pod="e2e-models-e2e-worker-w2-1a11cf/distinct-w2-1a11cf-kserve-854cf4cdf8-cddw6"
Oct 30 00:45:54.263293 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:54.263253    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32a03478-46eb-4363-97b6-51a32c4099c7-dshm\") pod \"distinct-w2-1a11cf-kserve-854cf4cdf8-cddw6\" (UID: \"32a03478-46eb-4363-97b6-51a32c4099c7\") " pod="e2e-models-e2e-worker-w2-1a11cf/distinct-w2-1a11cf-kserve-854cf4cdf8-cddw6"
Oct 30 00:45:54.264584 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:54.264553    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32a03478-46eb-4363-97b6-51a32c4099c7-model-cache\") pod \"distinct-w2-1a11cf-kserve-854cf4cdf8-cddw6\" (UID: \"32a03478-46eb-4363-97b6-51a32c4099c7\") " pod="e2e-models-e2e-worker-w2-1a11cf/distinct-w2-1a11cf-kserve-854cf4cdf8-cddw6"
Oct 30 00:45:54.264793 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:54.264779    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32a03478-46eb-4363-97b6-51a32c4099c7-tls-certs\") pod \"distinct-w2-1a11cf-kserve-854cf4cdf8-cddw6\" (UID: \"32a03478-46eb-4363-97b6-51a32c4099c7\") " pod="e2e-models-e2e-worker-w2-1a11cf/distinct-w2-1a11cf-kserve-854cf4cdf8-cddw6"
Oct 30 00:45:54.264897 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:54.264885    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32a03478-46eb-4363-97b6-51a32c4099c7-home\") pod \"distinct-w2-1a11cf-kserve-854cf4cdf8-cddw6\" (UID: \"32a03478-46eb-4363-97b6-51a32c4099c7\") " pod="e2e-models-e2e-worker-w2-1a11cf/distinct-w2-1a11cf-kserve-854cf4cdf8-cddw6"
Oct 30 00:45:54.265052 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:54.265039    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32a03478-46eb-4363-97b6-51a32c4099c7-tmp-dir\") pod \"distinct-w2-1a11cf-kserve-854cf4cdf8-cddw6\" (UID: \"32a03478-46eb-4363-97b6-51a32c4099c7\") " pod="e2e-models-e2e-worker-w2-1a11cf/distinct-w2-1a11cf-kserve-854cf4cdf8-cddw6"
Oct 30 00:45:54.265235 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:54.265219    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8t8n9\" (UniqueName: \"kubernetes.io/projected/32a03478-46eb-4363-97b6-51a32c4099c7-kube-api-access-8t8n9\") pod \"distinct-w2-1a11cf-kserve-854cf4cdf8-cddw6\" (UID: \"32a03478-46eb-4363-97b6-51a32c4099c7\") " pod="e2e-models-e2e-worker-w2-1a11cf/distinct-w2-1a11cf-kserve-854cf4cdf8-cddw6"
Oct 30 00:45:54.273486 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:54.273462    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32a03478-46eb-4363-97b6-51a32c4099c7-dshm\") pod \"distinct-w2-1a11cf-kserve-854cf4cdf8-cddw6\" (UID: \"32a03478-46eb-4363-97b6-51a32c4099c7\") " pod="e2e-models-e2e-worker-w2-1a11cf/distinct-w2-1a11cf-kserve-854cf4cdf8-cddw6"
Oct 30 00:45:54.273923 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:54.273904    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32a03478-46eb-4363-97b6-51a32c4099c7-model-cache\") pod \"distinct-w2-1a11cf-kserve-854cf4cdf8-cddw6\" (UID: \"32a03478-46eb-4363-97b6-51a32c4099c7\") " pod="e2e-models-e2e-worker-w2-1a11cf/distinct-w2-1a11cf-kserve-854cf4cdf8-cddw6"
Oct 30 00:45:54.277967 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:54.277932    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32a03478-46eb-4363-97b6-51a32c4099c7-home\") pod \"distinct-w2-1a11cf-kserve-854cf4cdf8-cddw6\" (UID: \"32a03478-46eb-4363-97b6-51a32c4099c7\") " pod="e2e-models-e2e-worker-w2-1a11cf/distinct-w2-1a11cf-kserve-854cf4cdf8-cddw6"
Oct 30 00:45:54.278259 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:54.278236    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32a03478-46eb-4363-97b6-51a32c4099c7-tmp-dir\") pod \"distinct-w2-1a11cf-kserve-854cf4cdf8-cddw6\" (UID: \"32a03478-46eb-4363-97b6-51a32c4099c7\") " pod="e2e-models-e2e-worker-w2-1a11cf/distinct-w2-1a11cf-kserve-854cf4cdf8-cddw6"
Oct 30 00:45:54.293053 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:54.293025    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32a03478-46eb-4363-97b6-51a32c4099c7-tls-certs\") pod \"distinct-w2-1a11cf-kserve-854cf4cdf8-cddw6\" (UID: \"32a03478-46eb-4363-97b6-51a32c4099c7\") " pod="e2e-models-e2e-worker-w2-1a11cf/distinct-w2-1a11cf-kserve-854cf4cdf8-cddw6"
Oct 30 00:45:54.358456 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:54.354960    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8t8n9\" (UniqueName: \"kubernetes.io/projected/32a03478-46eb-4363-97b6-51a32c4099c7-kube-api-access-8t8n9\") pod \"distinct-w2-1a11cf-kserve-854cf4cdf8-cddw6\" (UID: \"32a03478-46eb-4363-97b6-51a32c4099c7\") " pod="e2e-models-e2e-worker-w2-1a11cf/distinct-w2-1a11cf-kserve-854cf4cdf8-cddw6"
Oct 30 00:45:54.383461 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:54.383058    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-1a11cf/distinct-w2-1a11cf-kserve-854cf4cdf8-cddw6"
Oct 30 00:45:54.765613 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:45:54.765582    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod32a03478_46eb_4363_97b6_51a32c4099c7.slice/crio-61371647b3505acee6dfc168db3f0c9fc10ec3cd0c104b317dd56c3c15ab2b1c WatchSource:0}: Error finding container 61371647b3505acee6dfc168db3f0c9fc10ec3cd0c104b317dd56c3c15ab2b1c: Status 404 returned error can't find the container with id 61371647b3505acee6dfc168db3f0c9fc10ec3cd0c104b317dd56c3c15ab2b1c
Oct 30 00:45:54.766003 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:54.765978    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-1a11cf/distinct-w2-1a11cf-kserve-854cf4cdf8-cddw6"]
Oct 30 00:45:55.713013 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:55.712958    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-1a11cf/distinct-w2-1a11cf-kserve-854cf4cdf8-cddw6" event={"ID":"32a03478-46eb-4363-97b6-51a32c4099c7","Type":"ContainerStarted","Data":"dcb0a41337b62c29232f8b4261c4e6d0adf60e2e9d0dd0073c743932690095a8"}
Oct 30 00:45:55.713489 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:55.713030    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-1a11cf/distinct-w2-1a11cf-kserve-854cf4cdf8-cddw6" event={"ID":"32a03478-46eb-4363-97b6-51a32c4099c7","Type":"ContainerStarted","Data":"61371647b3505acee6dfc168db3f0c9fc10ec3cd0c104b317dd56c3c15ab2b1c"}
Oct 30 00:45:55.759149 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:55.759057    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-1a11cf/distinct-w2-1a11cf-kserve-854cf4cdf8-cddw6" podStartSLOduration=1.759042821 podStartE2EDuration="1.759042821s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:45:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:45:55.756816926 +0000 UTC m=+851.373249802" watchObservedRunningTime="2026-10-30 00:45:55.759042821 +0000 UTC m=+851.375475696"
Oct 30 00:45:56.384474 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:56.384432    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-1a11cf/distinct-w2-1a11cf-kserve-854cf4cdf8-cddw6"
Oct 30 00:45:59.913551 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:59.913507    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-fwrpg"]
Oct 30 00:45:59.917488 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:59.917463    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-fwrpg"
Oct 30 00:45:59.956329 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:45:59.956290    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-fwrpg"]
Oct 30 00:46:00.031939 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:00.031902    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a498076a-ee51-4409-b7d4-a0fabd0f385e-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-fwrpg\" (UID: \"a498076a-ee51-4409-b7d4-a0fabd0f385e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-fwrpg"
Oct 30 00:46:00.032111 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:00.031961    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a498076a-ee51-4409-b7d4-a0fabd0f385e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-fwrpg\" (UID: \"a498076a-ee51-4409-b7d4-a0fabd0f385e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-fwrpg"
Oct 30 00:46:00.032111 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:00.032063    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a498076a-ee51-4409-b7d4-a0fabd0f385e-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-fwrpg\" (UID: \"a498076a-ee51-4409-b7d4-a0fabd0f385e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-fwrpg"
Oct 30 00:46:00.032111 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:00.032104    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a498076a-ee51-4409-b7d4-a0fabd0f385e-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-fwrpg\" (UID: \"a498076a-ee51-4409-b7d4-a0fabd0f385e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-fwrpg"
Oct 30 00:46:00.032233 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:00.032127    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a498076a-ee51-4409-b7d4-a0fabd0f385e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-fwrpg\" (UID: \"a498076a-ee51-4409-b7d4-a0fabd0f385e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-fwrpg"
Oct 30 00:46:00.032264 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:00.032243    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58b2n\" (UniqueName: \"kubernetes.io/projected/a498076a-ee51-4409-b7d4-a0fabd0f385e-kube-api-access-58b2n\") pod \"maas-default-gateway-openshift-default-7d9897b54c-fwrpg\" (UID: \"a498076a-ee51-4409-b7d4-a0fabd0f385e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-fwrpg"
Oct 30 00:46:00.032297 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:00.032271    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a498076a-ee51-4409-b7d4-a0fabd0f385e-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-fwrpg\" (UID: \"a498076a-ee51-4409-b7d4-a0fabd0f385e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-fwrpg"
Oct 30 00:46:00.032342 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:00.032303    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a498076a-ee51-4409-b7d4-a0fabd0f385e-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-fwrpg\" (UID: \"a498076a-ee51-4409-b7d4-a0fabd0f385e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-fwrpg"
Oct 30 00:46:00.032424 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:00.032410    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a498076a-ee51-4409-b7d4-a0fabd0f385e-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-fwrpg\" (UID: \"a498076a-ee51-4409-b7d4-a0fabd0f385e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-fwrpg"
Oct 30 00:46:00.133047 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:00.133014    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a498076a-ee51-4409-b7d4-a0fabd0f385e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-fwrpg\" (UID: \"a498076a-ee51-4409-b7d4-a0fabd0f385e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-fwrpg"
Oct 30 00:46:00.133221 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:00.133068    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a498076a-ee51-4409-b7d4-a0fabd0f385e-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-fwrpg\" (UID: \"a498076a-ee51-4409-b7d4-a0fabd0f385e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-fwrpg"
Oct 30 00:46:00.133221 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:00.133101    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a498076a-ee51-4409-b7d4-a0fabd0f385e-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-fwrpg\" (UID: \"a498076a-ee51-4409-b7d4-a0fabd0f385e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-fwrpg"
Oct 30 00:46:00.133221 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:00.133129    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a498076a-ee51-4409-b7d4-a0fabd0f385e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-fwrpg\" (UID: \"a498076a-ee51-4409-b7d4-a0fabd0f385e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-fwrpg"
Oct 30 00:46:00.133221 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:00.133201    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58b2n\" (UniqueName: \"kubernetes.io/projected/a498076a-ee51-4409-b7d4-a0fabd0f385e-kube-api-access-58b2n\") pod \"maas-default-gateway-openshift-default-7d9897b54c-fwrpg\" (UID: \"a498076a-ee51-4409-b7d4-a0fabd0f385e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-fwrpg"
Oct 30 00:46:00.133419 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:00.133227    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a498076a-ee51-4409-b7d4-a0fabd0f385e-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-fwrpg\" (UID: \"a498076a-ee51-4409-b7d4-a0fabd0f385e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-fwrpg"
Oct 30 00:46:00.133419 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:00.133286    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a498076a-ee51-4409-b7d4-a0fabd0f385e-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-fwrpg\" (UID: \"a498076a-ee51-4409-b7d4-a0fabd0f385e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-fwrpg"
Oct 30 00:46:00.133419 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:00.133317    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a498076a-ee51-4409-b7d4-a0fabd0f385e-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-fwrpg\" (UID: \"a498076a-ee51-4409-b7d4-a0fabd0f385e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-fwrpg"
Oct 30 00:46:00.133530 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:00.133432    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a498076a-ee51-4409-b7d4-a0fabd0f385e-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-fwrpg\" (UID: \"a498076a-ee51-4409-b7d4-a0fabd0f385e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-fwrpg"
Oct 30 00:46:00.134127 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:00.133771    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a498076a-ee51-4409-b7d4-a0fabd0f385e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-fwrpg\" (UID: \"a498076a-ee51-4409-b7d4-a0fabd0f385e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-fwrpg"
Oct 30 00:46:00.134127 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:00.133981    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a498076a-ee51-4409-b7d4-a0fabd0f385e-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-fwrpg\" (UID: \"a498076a-ee51-4409-b7d4-a0fabd0f385e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-fwrpg"
Oct 30 00:46:00.134127 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:00.134052    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a498076a-ee51-4409-b7d4-a0fabd0f385e-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-fwrpg\" (UID: \"a498076a-ee51-4409-b7d4-a0fabd0f385e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-fwrpg"
Oct 30 00:46:00.134127 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:00.134057    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a498076a-ee51-4409-b7d4-a0fabd0f385e-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-fwrpg\" (UID: \"a498076a-ee51-4409-b7d4-a0fabd0f385e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-fwrpg"
Oct 30 00:46:00.134127 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:00.134098    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a498076a-ee51-4409-b7d4-a0fabd0f385e-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-fwrpg\" (UID: \"a498076a-ee51-4409-b7d4-a0fabd0f385e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-fwrpg"
Oct 30 00:46:00.135918 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:00.135895    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a498076a-ee51-4409-b7d4-a0fabd0f385e-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-fwrpg\" (UID: \"a498076a-ee51-4409-b7d4-a0fabd0f385e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-fwrpg"
Oct 30 00:46:00.136332 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:00.136310    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a498076a-ee51-4409-b7d4-a0fabd0f385e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-fwrpg\" (UID: \"a498076a-ee51-4409-b7d4-a0fabd0f385e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-fwrpg"
Oct 30 00:46:00.160482 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:00.160440    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a498076a-ee51-4409-b7d4-a0fabd0f385e-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-fwrpg\" (UID: \"a498076a-ee51-4409-b7d4-a0fabd0f385e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-fwrpg"
Oct 30 00:46:00.160720 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:00.160696    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-58b2n\" (UniqueName: \"kubernetes.io/projected/a498076a-ee51-4409-b7d4-a0fabd0f385e-kube-api-access-58b2n\") pod \"maas-default-gateway-openshift-default-7d9897b54c-fwrpg\" (UID: \"a498076a-ee51-4409-b7d4-a0fabd0f385e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-fwrpg"
Oct 30 00:46:00.234469 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:00.234131    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-fwrpg"
Oct 30 00:46:00.561346 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:00.558522    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-fwrpg"]
Oct 30 00:46:00.561922 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:46:00.561886    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda498076a_ee51_4409_b7d4_a0fabd0f385e.slice/crio-737463eec608712aedd9bff6452b7662e206fc2278be7c75ce297b0ee2f0f9f5 WatchSource:0}: Error finding container 737463eec608712aedd9bff6452b7662e206fc2278be7c75ce297b0ee2f0f9f5: Status 404 returned error can't find the container with id 737463eec608712aedd9bff6452b7662e206fc2278be7c75ce297b0ee2f0f9f5
Oct 30 00:46:00.564859 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:00.564818    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 00:46:00.564958 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:00.564893    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 00:46:00.564958 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:00.564923    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 00:46:00.745653 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:00.745619    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-fwrpg" event={"ID":"a498076a-ee51-4409-b7d4-a0fabd0f385e","Type":"ContainerStarted","Data":"6a4bc3dbb2ca630ef750fb9f407640b45b1313836cc71723ba752f8e79dbb471"}
Oct 30 00:46:00.745851 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:00.745674    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-fwrpg" event={"ID":"a498076a-ee51-4409-b7d4-a0fabd0f385e","Type":"ContainerStarted","Data":"737463eec608712aedd9bff6452b7662e206fc2278be7c75ce297b0ee2f0f9f5"}
Oct 30 00:46:00.795013 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:00.794949    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-fwrpg" podStartSLOduration=1.7949324199999999 podStartE2EDuration="1.79493242s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:45:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:46:00.792666291 +0000 UTC m=+856.409099172" watchObservedRunningTime="2026-10-30 00:46:00.79493242 +0000 UTC m=+856.411365301"
Oct 30 00:46:01.236907 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:01.236868    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-fwrpg"
Oct 30 00:46:01.239523 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:01.239483    5002 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-fwrpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body=
Oct 30 00:46:01.239771 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:01.239744    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-fwrpg" podUID="a498076a-ee51-4409-b7d4-a0fabd0f385e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused"
Oct 30 00:46:01.428807 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:01.428768    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-52650f/distinct-w0-52650f-kserve-d85899b5-vlhps"
Oct 30 00:46:01.441878 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:01.441850    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-52650f/distinct-w0-52650f-kserve-d85899b5-vlhps"
Oct 30 00:46:01.797016 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:01.796984    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-52650f/distinct-w0-52650f-kserve-d85899b5-vlhps"
Oct 30 00:46:02.235343 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:02.235302    5002 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-fwrpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body=
Oct 30 00:46:02.235536 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:02.235366    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-fwrpg" podUID="a498076a-ee51-4409-b7d4-a0fabd0f385e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused"
Oct 30 00:46:03.235194 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:03.235153    5002 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-fwrpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body=
Oct 30 00:46:03.235620 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:03.235208    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-fwrpg" podUID="a498076a-ee51-4409-b7d4-a0fabd0f385e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused"
Oct 30 00:46:04.234941 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:04.234901    5002 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-fwrpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body=
Oct 30 00:46:04.235152 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:04.234960    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-fwrpg" podUID="a498076a-ee51-4409-b7d4-a0fabd0f385e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused"
Oct 30 00:46:04.386403 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:04.385738    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-1a11cf/distinct-w2-1a11cf-kserve-854cf4cdf8-cddw6"
Oct 30 00:46:04.403156 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:04.403129    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-1a11cf/distinct-w2-1a11cf-kserve-854cf4cdf8-cddw6"
Oct 30 00:46:04.812305 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:04.812271    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-1a11cf/distinct-w2-1a11cf-kserve-854cf4cdf8-cddw6"
Oct 30 00:46:05.087756 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:05.087662    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-gvb85"]
Oct 30 00:46:05.091795 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:05.091770    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-gvb85"
Oct 30 00:46:05.111119 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:05.111076    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-29be73-openshift-default-dockercfg-xqn9v\""
Oct 30 00:46:05.114197 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:05.114174    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-gvb85"]
Oct 30 00:46:05.228029 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:05.227991    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvv8q\" (UniqueName: \"kubernetes.io/projected/70f303a4-967c-4a2f-8378-f30e81101a66-kube-api-access-rvv8q\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-gvb85\" (UID: \"70f303a4-967c-4a2f-8378-f30e81101a66\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-gvb85"
Oct 30 00:46:05.228201 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:05.228058    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70f303a4-967c-4a2f-8378-f30e81101a66-istio-envoy\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-gvb85\" (UID: \"70f303a4-967c-4a2f-8378-f30e81101a66\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-gvb85"
Oct 30 00:46:05.228201 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:05.228137    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70f303a4-967c-4a2f-8378-f30e81101a66-istio-data\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-gvb85\" (UID: \"70f303a4-967c-4a2f-8378-f30e81101a66\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-gvb85"
Oct 30 00:46:05.228271 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:05.228249    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70f303a4-967c-4a2f-8378-f30e81101a66-workload-socket\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-gvb85\" (UID: \"70f303a4-967c-4a2f-8378-f30e81101a66\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-gvb85"
Oct 30 00:46:05.228302 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:05.228291    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70f303a4-967c-4a2f-8378-f30e81101a66-istiod-ca-cert\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-gvb85\" (UID: \"70f303a4-967c-4a2f-8378-f30e81101a66\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-gvb85"
Oct 30 00:46:05.228425 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:05.228409    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70f303a4-967c-4a2f-8378-f30e81101a66-istio-podinfo\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-gvb85\" (UID: \"70f303a4-967c-4a2f-8378-f30e81101a66\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-gvb85"
Oct 30 00:46:05.228473 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:05.228452    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70f303a4-967c-4a2f-8378-f30e81101a66-workload-certs\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-gvb85\" (UID: \"70f303a4-967c-4a2f-8378-f30e81101a66\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-gvb85"
Oct 30 00:46:05.228565 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:05.228548    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70f303a4-967c-4a2f-8378-f30e81101a66-credential-socket\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-gvb85\" (UID: \"70f303a4-967c-4a2f-8378-f30e81101a66\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-gvb85"
Oct 30 00:46:05.228608 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:05.228581    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70f303a4-967c-4a2f-8378-f30e81101a66-istio-token\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-gvb85\" (UID: \"70f303a4-967c-4a2f-8378-f30e81101a66\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-gvb85"
Oct 30 00:46:05.235025 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:05.235001    5002 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-fwrpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body=
Oct 30 00:46:05.235145 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:05.235066    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-fwrpg" podUID="a498076a-ee51-4409-b7d4-a0fabd0f385e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused"
Oct 30 00:46:05.330282 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:05.330242    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70f303a4-967c-4a2f-8378-f30e81101a66-workload-socket\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-gvb85\" (UID: \"70f303a4-967c-4a2f-8378-f30e81101a66\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-gvb85"
Oct 30 00:46:05.330282 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:05.330291    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70f303a4-967c-4a2f-8378-f30e81101a66-istiod-ca-cert\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-gvb85\" (UID: \"70f303a4-967c-4a2f-8378-f30e81101a66\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-gvb85"
Oct 30 00:46:05.330532 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:05.330326    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70f303a4-967c-4a2f-8378-f30e81101a66-istio-podinfo\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-gvb85\" (UID: \"70f303a4-967c-4a2f-8378-f30e81101a66\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-gvb85"
Oct 30 00:46:05.330532 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:05.330355    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70f303a4-967c-4a2f-8378-f30e81101a66-workload-certs\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-gvb85\" (UID: \"70f303a4-967c-4a2f-8378-f30e81101a66\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-gvb85"
Oct 30 00:46:05.330532 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:05.330415    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70f303a4-967c-4a2f-8378-f30e81101a66-credential-socket\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-gvb85\" (UID: \"70f303a4-967c-4a2f-8378-f30e81101a66\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-gvb85"
Oct 30 00:46:05.330532 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:05.330474    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70f303a4-967c-4a2f-8378-f30e81101a66-istio-token\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-gvb85\" (UID: \"70f303a4-967c-4a2f-8378-f30e81101a66\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-gvb85"
Oct 30 00:46:05.330663 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:05.330570    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvv8q\" (UniqueName: \"kubernetes.io/projected/70f303a4-967c-4a2f-8378-f30e81101a66-kube-api-access-rvv8q\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-gvb85\" (UID: \"70f303a4-967c-4a2f-8378-f30e81101a66\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-gvb85"
Oct 30 00:46:05.330663 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:05.330629    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70f303a4-967c-4a2f-8378-f30e81101a66-istio-envoy\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-gvb85\" (UID: \"70f303a4-967c-4a2f-8378-f30e81101a66\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-gvb85"
Oct 30 00:46:05.330663 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:05.330653    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70f303a4-967c-4a2f-8378-f30e81101a66-istio-data\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-gvb85\" (UID: \"70f303a4-967c-4a2f-8378-f30e81101a66\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-gvb85"
Oct 30 00:46:05.331125 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:05.330834    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70f303a4-967c-4a2f-8378-f30e81101a66-credential-socket\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-gvb85\" (UID: \"70f303a4-967c-4a2f-8378-f30e81101a66\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-gvb85"
Oct 30 00:46:05.331125 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:05.330875    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70f303a4-967c-4a2f-8378-f30e81101a66-workload-certs\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-gvb85\" (UID: \"70f303a4-967c-4a2f-8378-f30e81101a66\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-gvb85"
Oct 30 00:46:05.331125 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:05.331048    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70f303a4-967c-4a2f-8378-f30e81101a66-istio-data\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-gvb85\" (UID: \"70f303a4-967c-4a2f-8378-f30e81101a66\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-gvb85"
Oct 30 00:46:05.331335 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:05.331169    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70f303a4-967c-4a2f-8378-f30e81101a66-istiod-ca-cert\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-gvb85\" (UID: \"70f303a4-967c-4a2f-8378-f30e81101a66\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-gvb85"
Oct 30 00:46:05.331335 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:05.331276    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70f303a4-967c-4a2f-8378-f30e81101a66-workload-socket\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-gvb85\" (UID: \"70f303a4-967c-4a2f-8378-f30e81101a66\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-gvb85"
Oct 30 00:46:05.333061 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:05.333042    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70f303a4-967c-4a2f-8378-f30e81101a66-istio-envoy\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-gvb85\" (UID: \"70f303a4-967c-4a2f-8378-f30e81101a66\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-gvb85"
Oct 30 00:46:05.333188 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:05.333169    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70f303a4-967c-4a2f-8378-f30e81101a66-istio-podinfo\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-gvb85\" (UID: \"70f303a4-967c-4a2f-8378-f30e81101a66\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-gvb85"
Oct 30 00:46:05.359446 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:05.357167    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70f303a4-967c-4a2f-8378-f30e81101a66-istio-token\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-gvb85\" (UID: \"70f303a4-967c-4a2f-8378-f30e81101a66\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-gvb85"
Oct 30 00:46:05.371405 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:05.369065    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvv8q\" (UniqueName: \"kubernetes.io/projected/70f303a4-967c-4a2f-8378-f30e81101a66-kube-api-access-rvv8q\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-gvb85\" (UID: \"70f303a4-967c-4a2f-8378-f30e81101a66\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-gvb85"
Oct 30 00:46:05.407430 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:05.407397    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-gvb85"
Oct 30 00:46:05.779504 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:05.777710    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-gvb85"]
Oct 30 00:46:05.784028 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:05.783986    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 00:46:05.784157 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:05.784075    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 00:46:05.784157 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:05.784120    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 00:46:05.809444 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:05.809417    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-gvb85" event={"ID":"70f303a4-967c-4a2f-8378-f30e81101a66","Type":"ContainerStarted","Data":"93af0cfc599394d0ac6abed5cf670450a5206a86289f3a3d545c2d713158d339"}
Oct 30 00:46:06.235174 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:06.235128    5002 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-fwrpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body=
Oct 30 00:46:06.235336 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:06.235197    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-fwrpg" podUID="a498076a-ee51-4409-b7d4-a0fabd0f385e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused"
Oct 30 00:46:06.820872 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:06.820837    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-gvb85" event={"ID":"70f303a4-967c-4a2f-8378-f30e81101a66","Type":"ContainerStarted","Data":"fb9406b986c59d5abf58064fd61d7063bb19cf99e799e33f6964c3c28eea99a2"}
Oct 30 00:46:06.888346 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:06.888260    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-gvb85" podStartSLOduration=1.888240814 podStartE2EDuration="1.888240814s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:46:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:46:06.8868647 +0000 UTC m=+862.503297600" watchObservedRunningTime="2026-10-30 00:46:06.888240814 +0000 UTC m=+862.504673695"
Oct 30 00:46:07.235443 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:07.235406    5002 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-fwrpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body=
Oct 30 00:46:07.235673 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:07.235650    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-fwrpg" podUID="a498076a-ee51-4409-b7d4-a0fabd0f385e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused"
Oct 30 00:46:07.408714 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:07.408671    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-gvb85"
Oct 30 00:46:07.410619 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:07.410589    5002 patch_prober.go:29] interesting pod/e2e-worker-w1-29be73-openshift-default-66467f5dcf-gvb85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body=
Oct 30 00:46:07.410757 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:07.410645    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-gvb85" podUID="70f303a4-967c-4a2f-8378-f30e81101a66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused"
Oct 30 00:46:08.234774 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:08.234733    5002 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-fwrpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body=
Oct 30 00:46:08.235148 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:08.234794    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-fwrpg" podUID="a498076a-ee51-4409-b7d4-a0fabd0f385e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused"
Oct 30 00:46:08.408699 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:08.408652    5002 patch_prober.go:29] interesting pod/e2e-worker-w1-29be73-openshift-default-66467f5dcf-gvb85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body=
Oct 30 00:46:08.408883 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:08.408723    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-gvb85" podUID="70f303a4-967c-4a2f-8378-f30e81101a66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused"
Oct 30 00:46:08.826525 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:08.826493    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-ca5363/distinct-2-w4-ca5363-kserve-576576f9b5-xkkqc"]
Oct 30 00:46:08.830970 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:08.830939    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ca5363/distinct-2-w4-ca5363-kserve-576576f9b5-xkkqc"
Oct 30 00:46:08.883967 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:08.883930    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-ca5363\"/\"distinct-2-w4-ca5363-kserve-self-signed-certs\""
Oct 30 00:46:08.891520 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:08.888735    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-ca5363\"/\"default-dockercfg-lm88z\""
Oct 30 00:46:08.892976 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:08.892948    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-ca5363/distinct-2-w4-ca5363-kserve-576576f9b5-xkkqc"]
Oct 30 00:46:08.901643 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:08.901593    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-ca5363\"/\"openshift-service-ca.crt\""
Oct 30 00:46:08.901811 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:08.901639    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-ca5363\"/\"kube-root-ca.crt\""
Oct 30 00:46:09.006262 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:09.004003    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5de8bebf-cfad-49da-9a98-08aedde4442c-model-cache\") pod \"distinct-2-w4-ca5363-kserve-576576f9b5-xkkqc\" (UID: \"5de8bebf-cfad-49da-9a98-08aedde4442c\") " pod="e2e-models-e2e-worker-w4-ca5363/distinct-2-w4-ca5363-kserve-576576f9b5-xkkqc"
Oct 30 00:46:09.006262 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:09.004063    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5de8bebf-cfad-49da-9a98-08aedde4442c-dshm\") pod \"distinct-2-w4-ca5363-kserve-576576f9b5-xkkqc\" (UID: \"5de8bebf-cfad-49da-9a98-08aedde4442c\") " pod="e2e-models-e2e-worker-w4-ca5363/distinct-2-w4-ca5363-kserve-576576f9b5-xkkqc"
Oct 30 00:46:09.006262 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:09.004115    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5de8bebf-cfad-49da-9a98-08aedde4442c-tmp-dir\") pod \"distinct-2-w4-ca5363-kserve-576576f9b5-xkkqc\" (UID: \"5de8bebf-cfad-49da-9a98-08aedde4442c\") " pod="e2e-models-e2e-worker-w4-ca5363/distinct-2-w4-ca5363-kserve-576576f9b5-xkkqc"
Oct 30 00:46:09.006262 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:09.004183    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5de8bebf-cfad-49da-9a98-08aedde4442c-tls-certs\") pod \"distinct-2-w4-ca5363-kserve-576576f9b5-xkkqc\" (UID: \"5de8bebf-cfad-49da-9a98-08aedde4442c\") " pod="e2e-models-e2e-worker-w4-ca5363/distinct-2-w4-ca5363-kserve-576576f9b5-xkkqc"
Oct 30 00:46:09.006262 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:09.004256    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5v78h\" (UniqueName: \"kubernetes.io/projected/5de8bebf-cfad-49da-9a98-08aedde4442c-kube-api-access-5v78h\") pod \"distinct-2-w4-ca5363-kserve-576576f9b5-xkkqc\" (UID: \"5de8bebf-cfad-49da-9a98-08aedde4442c\") " pod="e2e-models-e2e-worker-w4-ca5363/distinct-2-w4-ca5363-kserve-576576f9b5-xkkqc"
Oct 30 00:46:09.006262 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:09.004332    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5de8bebf-cfad-49da-9a98-08aedde4442c-home\") pod \"distinct-2-w4-ca5363-kserve-576576f9b5-xkkqc\" (UID: \"5de8bebf-cfad-49da-9a98-08aedde4442c\") " pod="e2e-models-e2e-worker-w4-ca5363/distinct-2-w4-ca5363-kserve-576576f9b5-xkkqc"
Oct 30 00:46:09.107460 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:09.107335    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5de8bebf-cfad-49da-9a98-08aedde4442c-model-cache\") pod \"distinct-2-w4-ca5363-kserve-576576f9b5-xkkqc\" (UID: \"5de8bebf-cfad-49da-9a98-08aedde4442c\") " pod="e2e-models-e2e-worker-w4-ca5363/distinct-2-w4-ca5363-kserve-576576f9b5-xkkqc"
Oct 30 00:46:09.107763 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:09.107732    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5de8bebf-cfad-49da-9a98-08aedde4442c-dshm\") pod \"distinct-2-w4-ca5363-kserve-576576f9b5-xkkqc\" (UID: \"5de8bebf-cfad-49da-9a98-08aedde4442c\") " pod="e2e-models-e2e-worker-w4-ca5363/distinct-2-w4-ca5363-kserve-576576f9b5-xkkqc"
Oct 30 00:46:09.107898 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:09.107881    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5de8bebf-cfad-49da-9a98-08aedde4442c-tmp-dir\") pod \"distinct-2-w4-ca5363-kserve-576576f9b5-xkkqc\" (UID: \"5de8bebf-cfad-49da-9a98-08aedde4442c\") " pod="e2e-models-e2e-worker-w4-ca5363/distinct-2-w4-ca5363-kserve-576576f9b5-xkkqc"
Oct 30 00:46:09.108053 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:09.108037    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5de8bebf-cfad-49da-9a98-08aedde4442c-tls-certs\") pod \"distinct-2-w4-ca5363-kserve-576576f9b5-xkkqc\" (UID: \"5de8bebf-cfad-49da-9a98-08aedde4442c\") " pod="e2e-models-e2e-worker-w4-ca5363/distinct-2-w4-ca5363-kserve-576576f9b5-xkkqc"
Oct 30 00:46:09.108181 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:09.108163    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5v78h\" (UniqueName: \"kubernetes.io/projected/5de8bebf-cfad-49da-9a98-08aedde4442c-kube-api-access-5v78h\") pod \"distinct-2-w4-ca5363-kserve-576576f9b5-xkkqc\" (UID: \"5de8bebf-cfad-49da-9a98-08aedde4442c\") " pod="e2e-models-e2e-worker-w4-ca5363/distinct-2-w4-ca5363-kserve-576576f9b5-xkkqc"
Oct 30 00:46:09.108337 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:09.108320    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5de8bebf-cfad-49da-9a98-08aedde4442c-home\") pod \"distinct-2-w4-ca5363-kserve-576576f9b5-xkkqc\" (UID: \"5de8bebf-cfad-49da-9a98-08aedde4442c\") " pod="e2e-models-e2e-worker-w4-ca5363/distinct-2-w4-ca5363-kserve-576576f9b5-xkkqc"
Oct 30 00:46:09.108985 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:09.108961    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5de8bebf-cfad-49da-9a98-08aedde4442c-home\") pod \"distinct-2-w4-ca5363-kserve-576576f9b5-xkkqc\" (UID: \"5de8bebf-cfad-49da-9a98-08aedde4442c\") " pod="e2e-models-e2e-worker-w4-ca5363/distinct-2-w4-ca5363-kserve-576576f9b5-xkkqc"
Oct 30 00:46:09.109404 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:09.109366    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5de8bebf-cfad-49da-9a98-08aedde4442c-model-cache\") pod \"distinct-2-w4-ca5363-kserve-576576f9b5-xkkqc\" (UID: \"5de8bebf-cfad-49da-9a98-08aedde4442c\") " pod="e2e-models-e2e-worker-w4-ca5363/distinct-2-w4-ca5363-kserve-576576f9b5-xkkqc"
Oct 30 00:46:09.112118 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:09.112093    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5de8bebf-cfad-49da-9a98-08aedde4442c-tmp-dir\") pod \"distinct-2-w4-ca5363-kserve-576576f9b5-xkkqc\" (UID: \"5de8bebf-cfad-49da-9a98-08aedde4442c\") " pod="e2e-models-e2e-worker-w4-ca5363/distinct-2-w4-ca5363-kserve-576576f9b5-xkkqc"
Oct 30 00:46:09.124359 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:09.124329    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5de8bebf-cfad-49da-9a98-08aedde4442c-dshm\") pod \"distinct-2-w4-ca5363-kserve-576576f9b5-xkkqc\" (UID: \"5de8bebf-cfad-49da-9a98-08aedde4442c\") " pod="e2e-models-e2e-worker-w4-ca5363/distinct-2-w4-ca5363-kserve-576576f9b5-xkkqc"
Oct 30 00:46:09.136778 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:09.136746    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5de8bebf-cfad-49da-9a98-08aedde4442c-tls-certs\") pod \"distinct-2-w4-ca5363-kserve-576576f9b5-xkkqc\" (UID: \"5de8bebf-cfad-49da-9a98-08aedde4442c\") " pod="e2e-models-e2e-worker-w4-ca5363/distinct-2-w4-ca5363-kserve-576576f9b5-xkkqc"
Oct 30 00:46:09.147286 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:09.147211    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5v78h\" (UniqueName: \"kubernetes.io/projected/5de8bebf-cfad-49da-9a98-08aedde4442c-kube-api-access-5v78h\") pod \"distinct-2-w4-ca5363-kserve-576576f9b5-xkkqc\" (UID: \"5de8bebf-cfad-49da-9a98-08aedde4442c\") " pod="e2e-models-e2e-worker-w4-ca5363/distinct-2-w4-ca5363-kserve-576576f9b5-xkkqc"
Oct 30 00:46:09.203461 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:09.203157    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ca5363/distinct-2-w4-ca5363-kserve-576576f9b5-xkkqc"
Oct 30 00:46:09.409082 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:09.408577    5002 patch_prober.go:29] interesting pod/e2e-worker-w1-29be73-openshift-default-66467f5dcf-gvb85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body=
Oct 30 00:46:09.409082 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:09.408637    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-gvb85" podUID="70f303a4-967c-4a2f-8378-f30e81101a66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused"
Oct 30 00:46:09.588408 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:09.588277    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-ca5363/distinct-2-w4-ca5363-kserve-576576f9b5-xkkqc"]
Oct 30 00:46:09.592215 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:46:09.592179    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5de8bebf_cfad_49da_9a98_08aedde4442c.slice/crio-ad84d9fc58ea0b3962e42b82bd94ed709c56817c604f1f4396b5321d537f0431 WatchSource:0}: Error finding container ad84d9fc58ea0b3962e42b82bd94ed709c56817c604f1f4396b5321d537f0431: Status 404 returned error can't find the container with id ad84d9fc58ea0b3962e42b82bd94ed709c56817c604f1f4396b5321d537f0431
Oct 30 00:46:09.866403 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:09.864735    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ca5363/distinct-2-w4-ca5363-kserve-576576f9b5-xkkqc" event={"ID":"5de8bebf-cfad-49da-9a98-08aedde4442c","Type":"ContainerStarted","Data":"799190f41cf9cd2205e5603084cd81d6af97fb7562ff18ef90213ac781319742"}
Oct 30 00:46:09.866403 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:09.864789    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ca5363/distinct-2-w4-ca5363-kserve-576576f9b5-xkkqc" event={"ID":"5de8bebf-cfad-49da-9a98-08aedde4442c","Type":"ContainerStarted","Data":"ad84d9fc58ea0b3962e42b82bd94ed709c56817c604f1f4396b5321d537f0431"}
Oct 30 00:46:09.916285 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:09.916177    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-ca5363/distinct-2-w4-ca5363-kserve-576576f9b5-xkkqc" podStartSLOduration=1.916163171 podStartE2EDuration="1.916163171s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:46:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:46:09.913519028 +0000 UTC m=+865.529951907" watchObservedRunningTime="2026-10-30 00:46:09.916163171 +0000 UTC m=+865.532596083"
Oct 30 00:46:09.993416 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:09.992846    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-fwrpg"
Oct 30 00:46:09.993416 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:09.993296    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-fwrpg"
Oct 30 00:46:09.994512 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:09.994483    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-fwrpg"
Oct 30 00:46:10.204835 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:10.204804    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ca5363/distinct-2-w4-ca5363-kserve-576576f9b5-xkkqc"
Oct 30 00:46:10.408746 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:10.408709    5002 patch_prober.go:29] interesting pod/e2e-worker-w1-29be73-openshift-default-66467f5dcf-gvb85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body=
Oct 30 00:46:10.408896 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:10.408769    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-gvb85" podUID="70f303a4-967c-4a2f-8378-f30e81101a66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused"
Oct 30 00:46:11.408521 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:11.408488    5002 patch_prober.go:29] interesting pod/e2e-worker-w1-29be73-openshift-default-66467f5dcf-gvb85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body=
Oct 30 00:46:11.408905 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:11.408544    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-gvb85" podUID="70f303a4-967c-4a2f-8378-f30e81101a66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused"
Oct 30 00:46:12.408267 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:12.408229    5002 patch_prober.go:29] interesting pod/e2e-worker-w1-29be73-openshift-default-66467f5dcf-gvb85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body=
Oct 30 00:46:12.408459 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:12.408296    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-gvb85" podUID="70f303a4-967c-4a2f-8378-f30e81101a66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused"
Oct 30 00:46:12.866946 ip-10-0-128-7 kubenswrapper[5002]: E1030 00:46:12.866826    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbjdq" containerName="extractor"
Oct 30 00:46:13.413132 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:13.413015    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-gvb85"
Oct 30 00:46:13.413441 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:13.413289    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-gvb85"
Oct 30 00:46:13.414046 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:13.414028    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-gvb85"
Oct 30 00:46:15.312212 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:15.312176    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7d8z4"]
Oct 30 00:46:19.203984 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:19.203946    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-ca5363/distinct-2-w4-ca5363-kserve-576576f9b5-xkkqc"
Oct 30 00:46:19.216662 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:19.216631    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-ca5363/distinct-2-w4-ca5363-kserve-576576f9b5-xkkqc"
Oct 30 00:46:19.952098 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:19.952066    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ca5363/distinct-2-w4-ca5363-kserve-576576f9b5-xkkqc"
Oct 30 00:46:20.563919 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:20.563880    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-vqjxg"]
Oct 30 00:46:20.569399 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:20.568455    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-vqjxg"
Oct 30 00:46:20.607839 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:20.607801    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-vqjxg"]
Oct 30 00:46:20.625962 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:20.625932    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00e1797d-0629-4b79-8073-eaa093f48f2a-istiod-ca-cert\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-vqjxg\" (UID: \"00e1797d-0629-4b79-8073-eaa093f48f2a\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-vqjxg"
Oct 30 00:46:20.626141 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:20.625984    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00e1797d-0629-4b79-8073-eaa093f48f2a-istio-envoy\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-vqjxg\" (UID: \"00e1797d-0629-4b79-8073-eaa093f48f2a\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-vqjxg"
Oct 30 00:46:20.626141 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:20.626009    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00e1797d-0629-4b79-8073-eaa093f48f2a-istio-podinfo\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-vqjxg\" (UID: \"00e1797d-0629-4b79-8073-eaa093f48f2a\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-vqjxg"
Oct 30 00:46:20.626141 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:20.626071    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00e1797d-0629-4b79-8073-eaa093f48f2a-istio-data\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-vqjxg\" (UID: \"00e1797d-0629-4b79-8073-eaa093f48f2a\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-vqjxg"
Oct 30 00:46:20.626324 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:20.626169    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00e1797d-0629-4b79-8073-eaa093f48f2a-workload-socket\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-vqjxg\" (UID: \"00e1797d-0629-4b79-8073-eaa093f48f2a\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-vqjxg"
Oct 30 00:46:20.626324 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:20.626222    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mqb2\" (UniqueName: \"kubernetes.io/projected/00e1797d-0629-4b79-8073-eaa093f48f2a-kube-api-access-4mqb2\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-vqjxg\" (UID: \"00e1797d-0629-4b79-8073-eaa093f48f2a\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-vqjxg"
Oct 30 00:46:20.626324 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:20.626311    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00e1797d-0629-4b79-8073-eaa093f48f2a-credential-socket\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-vqjxg\" (UID: \"00e1797d-0629-4b79-8073-eaa093f48f2a\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-vqjxg"
Oct 30 00:46:20.626507 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:20.626334    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00e1797d-0629-4b79-8073-eaa093f48f2a-istio-token\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-vqjxg\" (UID: \"00e1797d-0629-4b79-8073-eaa093f48f2a\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-vqjxg"
Oct 30 00:46:20.626507 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:20.626361    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00e1797d-0629-4b79-8073-eaa093f48f2a-workload-certs\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-vqjxg\" (UID: \"00e1797d-0629-4b79-8073-eaa093f48f2a\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-vqjxg"
Oct 30 00:46:20.664441 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:20.664409    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-s5rzw"]
Oct 30 00:46:20.668431 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:20.668410    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-s5rzw"
Oct 30 00:46:20.693257 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:20.693222    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-s5rzw"]
Oct 30 00:46:20.727788 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:20.727758    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc529c8b-d888-412d-b861-f3648aaa473d-workload-socket\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-s5rzw\" (UID: \"cc529c8b-d888-412d-b861-f3648aaa473d\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-s5rzw"
Oct 30 00:46:20.727962 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:20.727827    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00e1797d-0629-4b79-8073-eaa093f48f2a-istiod-ca-cert\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-vqjxg\" (UID: \"00e1797d-0629-4b79-8073-eaa093f48f2a\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-vqjxg"
Oct 30 00:46:20.727962 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:20.727900    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc529c8b-d888-412d-b861-f3648aaa473d-istio-token\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-s5rzw\" (UID: \"cc529c8b-d888-412d-b861-f3648aaa473d\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-s5rzw"
Oct 30 00:46:20.727962 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:20.727950    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00e1797d-0629-4b79-8073-eaa093f48f2a-istio-envoy\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-vqjxg\" (UID: \"00e1797d-0629-4b79-8073-eaa093f48f2a\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-vqjxg"
Oct 30 00:46:20.728116 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:20.727967    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc529c8b-d888-412d-b861-f3648aaa473d-credential-socket\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-s5rzw\" (UID: \"cc529c8b-d888-412d-b861-f3648aaa473d\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-s5rzw"
Oct 30 00:46:20.728116 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:20.728107    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00e1797d-0629-4b79-8073-eaa093f48f2a-istio-podinfo\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-vqjxg\" (UID: \"00e1797d-0629-4b79-8073-eaa093f48f2a\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-vqjxg"
Oct 30 00:46:20.728221 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:20.728129    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dbnt\" (UniqueName: \"kubernetes.io/projected/cc529c8b-d888-412d-b861-f3648aaa473d-kube-api-access-9dbnt\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-s5rzw\" (UID: \"cc529c8b-d888-412d-b861-f3648aaa473d\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-s5rzw"
Oct 30 00:46:20.728221 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:20.728152    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00e1797d-0629-4b79-8073-eaa093f48f2a-istio-data\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-vqjxg\" (UID: \"00e1797d-0629-4b79-8073-eaa093f48f2a\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-vqjxg"
Oct 30 00:46:20.728221 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:20.728178    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc529c8b-d888-412d-b861-f3648aaa473d-istiod-ca-cert\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-s5rzw\" (UID: \"cc529c8b-d888-412d-b861-f3648aaa473d\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-s5rzw"
Oct 30 00:46:20.728221 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:20.728200    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc529c8b-d888-412d-b861-f3648aaa473d-istio-data\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-s5rzw\" (UID: \"cc529c8b-d888-412d-b861-f3648aaa473d\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-s5rzw"
Oct 30 00:46:20.728454 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:20.728226    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc529c8b-d888-412d-b861-f3648aaa473d-istio-envoy\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-s5rzw\" (UID: \"cc529c8b-d888-412d-b861-f3648aaa473d\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-s5rzw"
Oct 30 00:46:20.728454 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:20.728252    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00e1797d-0629-4b79-8073-eaa093f48f2a-workload-socket\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-vqjxg\" (UID: \"00e1797d-0629-4b79-8073-eaa093f48f2a\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-vqjxg"
Oct 30 00:46:20.728454 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:20.728282    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mqb2\" (UniqueName: \"kubernetes.io/projected/00e1797d-0629-4b79-8073-eaa093f48f2a-kube-api-access-4mqb2\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-vqjxg\" (UID: \"00e1797d-0629-4b79-8073-eaa093f48f2a\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-vqjxg"
Oct 30 00:46:20.728454 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:20.728319    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc529c8b-d888-412d-b861-f3648aaa473d-workload-certs\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-s5rzw\" (UID: \"cc529c8b-d888-412d-b861-f3648aaa473d\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-s5rzw"
Oct 30 00:46:20.728454 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:20.728361    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc529c8b-d888-412d-b861-f3648aaa473d-istio-podinfo\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-s5rzw\" (UID: \"cc529c8b-d888-412d-b861-f3648aaa473d\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-s5rzw"
Oct 30 00:46:20.728454 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:20.728410    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00e1797d-0629-4b79-8073-eaa093f48f2a-credential-socket\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-vqjxg\" (UID: \"00e1797d-0629-4b79-8073-eaa093f48f2a\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-vqjxg"
Oct 30 00:46:20.728454 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:20.728428    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00e1797d-0629-4b79-8073-eaa093f48f2a-istio-token\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-vqjxg\" (UID: \"00e1797d-0629-4b79-8073-eaa093f48f2a\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-vqjxg"
Oct 30 00:46:20.728723 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:20.728482    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00e1797d-0629-4b79-8073-eaa093f48f2a-workload-certs\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-vqjxg\" (UID: \"00e1797d-0629-4b79-8073-eaa093f48f2a\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-vqjxg"
Oct 30 00:46:20.728723 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:20.728704    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00e1797d-0629-4b79-8073-eaa093f48f2a-istiod-ca-cert\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-vqjxg\" (UID: \"00e1797d-0629-4b79-8073-eaa093f48f2a\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-vqjxg"
Oct 30 00:46:20.729149 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:20.728815    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00e1797d-0629-4b79-8073-eaa093f48f2a-workload-certs\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-vqjxg\" (UID: \"00e1797d-0629-4b79-8073-eaa093f48f2a\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-vqjxg"
Oct 30 00:46:20.729149 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:20.729085    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00e1797d-0629-4b79-8073-eaa093f48f2a-istio-data\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-vqjxg\" (UID: \"00e1797d-0629-4b79-8073-eaa093f48f2a\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-vqjxg"
Oct 30 00:46:20.729149 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:20.729109    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00e1797d-0629-4b79-8073-eaa093f48f2a-workload-socket\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-vqjxg\" (UID: \"00e1797d-0629-4b79-8073-eaa093f48f2a\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-vqjxg"
Oct 30 00:46:20.729400 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:20.729190    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00e1797d-0629-4b79-8073-eaa093f48f2a-credential-socket\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-vqjxg\" (UID: \"00e1797d-0629-4b79-8073-eaa093f48f2a\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-vqjxg"
Oct 30 00:46:20.730512 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:20.730490    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00e1797d-0629-4b79-8073-eaa093f48f2a-istio-envoy\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-vqjxg\" (UID: \"00e1797d-0629-4b79-8073-eaa093f48f2a\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-vqjxg"
Oct 30 00:46:20.731795 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:20.731776    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00e1797d-0629-4b79-8073-eaa093f48f2a-istio-podinfo\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-vqjxg\" (UID: \"00e1797d-0629-4b79-8073-eaa093f48f2a\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-vqjxg"
Oct 30 00:46:20.754156 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:20.754127    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00e1797d-0629-4b79-8073-eaa093f48f2a-istio-token\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-vqjxg\" (UID: \"00e1797d-0629-4b79-8073-eaa093f48f2a\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-vqjxg"
Oct 30 00:46:20.754248 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:20.754152    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mqb2\" (UniqueName: \"kubernetes.io/projected/00e1797d-0629-4b79-8073-eaa093f48f2a-kube-api-access-4mqb2\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-vqjxg\" (UID: \"00e1797d-0629-4b79-8073-eaa093f48f2a\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-vqjxg"
Oct 30 00:46:20.830041 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:20.829958    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc529c8b-d888-412d-b861-f3648aaa473d-credential-socket\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-s5rzw\" (UID: \"cc529c8b-d888-412d-b861-f3648aaa473d\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-s5rzw"
Oct 30 00:46:20.830041 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:20.830018    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dbnt\" (UniqueName: \"kubernetes.io/projected/cc529c8b-d888-412d-b861-f3648aaa473d-kube-api-access-9dbnt\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-s5rzw\" (UID: \"cc529c8b-d888-412d-b861-f3648aaa473d\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-s5rzw"
Oct 30 00:46:20.830250 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:20.830057    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc529c8b-d888-412d-b861-f3648aaa473d-istiod-ca-cert\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-s5rzw\" (UID: \"cc529c8b-d888-412d-b861-f3648aaa473d\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-s5rzw"
Oct 30 00:46:20.830250 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:20.830092    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc529c8b-d888-412d-b861-f3648aaa473d-istio-data\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-s5rzw\" (UID: \"cc529c8b-d888-412d-b861-f3648aaa473d\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-s5rzw"
Oct 30 00:46:20.830250 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:20.830128    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc529c8b-d888-412d-b861-f3648aaa473d-istio-envoy\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-s5rzw\" (UID: \"cc529c8b-d888-412d-b861-f3648aaa473d\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-s5rzw"
Oct 30 00:46:20.830250 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:20.830196    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc529c8b-d888-412d-b861-f3648aaa473d-workload-certs\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-s5rzw\" (UID: \"cc529c8b-d888-412d-b861-f3648aaa473d\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-s5rzw"
Oct 30 00:46:20.830459 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:20.830249    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc529c8b-d888-412d-b861-f3648aaa473d-istio-podinfo\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-s5rzw\" (UID: \"cc529c8b-d888-412d-b861-f3648aaa473d\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-s5rzw"
Oct 30 00:46:20.830459 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:20.830313    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc529c8b-d888-412d-b861-f3648aaa473d-workload-socket\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-s5rzw\" (UID: \"cc529c8b-d888-412d-b861-f3648aaa473d\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-s5rzw"
Oct 30 00:46:20.830459 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:20.830406    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc529c8b-d888-412d-b861-f3648aaa473d-istio-token\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-s5rzw\" (UID: \"cc529c8b-d888-412d-b861-f3648aaa473d\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-s5rzw"
Oct 30 00:46:20.830606 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:20.830488    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc529c8b-d888-412d-b861-f3648aaa473d-credential-socket\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-s5rzw\" (UID: \"cc529c8b-d888-412d-b861-f3648aaa473d\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-s5rzw"
Oct 30 00:46:20.830955 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:20.830925    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc529c8b-d888-412d-b861-f3648aaa473d-workload-socket\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-s5rzw\" (UID: \"cc529c8b-d888-412d-b861-f3648aaa473d\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-s5rzw"
Oct 30 00:46:20.831241 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:20.831211    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc529c8b-d888-412d-b861-f3648aaa473d-istio-data\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-s5rzw\" (UID: \"cc529c8b-d888-412d-b861-f3648aaa473d\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-s5rzw"
Oct 30 00:46:20.831543 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:20.831521    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc529c8b-d888-412d-b861-f3648aaa473d-workload-certs\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-s5rzw\" (UID: \"cc529c8b-d888-412d-b861-f3648aaa473d\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-s5rzw"
Oct 30 00:46:20.831709 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:20.831684    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc529c8b-d888-412d-b861-f3648aaa473d-istiod-ca-cert\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-s5rzw\" (UID: \"cc529c8b-d888-412d-b861-f3648aaa473d\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-s5rzw"
Oct 30 00:46:20.832922 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:20.832897    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc529c8b-d888-412d-b861-f3648aaa473d-istio-envoy\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-s5rzw\" (UID: \"cc529c8b-d888-412d-b861-f3648aaa473d\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-s5rzw"
Oct 30 00:46:20.833020 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:20.832976    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc529c8b-d888-412d-b861-f3648aaa473d-istio-podinfo\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-s5rzw\" (UID: \"cc529c8b-d888-412d-b861-f3648aaa473d\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-s5rzw"
Oct 30 00:46:20.855171 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:20.855146    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc529c8b-d888-412d-b861-f3648aaa473d-istio-token\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-s5rzw\" (UID: \"cc529c8b-d888-412d-b861-f3648aaa473d\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-s5rzw"
Oct 30 00:46:20.855771 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:20.855746    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dbnt\" (UniqueName: \"kubernetes.io/projected/cc529c8b-d888-412d-b861-f3648aaa473d-kube-api-access-9dbnt\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-s5rzw\" (UID: \"cc529c8b-d888-412d-b861-f3648aaa473d\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-s5rzw"
Oct 30 00:46:20.890645 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:20.890616    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-vqjxg"
Oct 30 00:46:20.984627 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:20.984595    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-s5rzw"
Oct 30 00:46:21.208825 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:46:21.208789    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00e1797d_0629_4b79_8073_eaa093f48f2a.slice/crio-1bf3e181715b960439a93de324f1da7cfcd1197f464b7d0caf84a501b5688011 WatchSource:0}: Error finding container 1bf3e181715b960439a93de324f1da7cfcd1197f464b7d0caf84a501b5688011: Status 404 returned error can't find the container with id 1bf3e181715b960439a93de324f1da7cfcd1197f464b7d0caf84a501b5688011
Oct 30 00:46:21.210814 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:21.210787    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-vqjxg"]
Oct 30 00:46:21.211761 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:21.211713    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 00:46:21.211860 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:21.211800    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 00:46:21.211860 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:21.211847    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 00:46:21.300676 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:21.300642    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-s5rzw"]
Oct 30 00:46:21.302013 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:46:21.301983    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc529c8b_d888_412d_b861_f3648aaa473d.slice/crio-ecd57d08fad9bca1f88fbc9847427a61df4f1aa1c5676a0a2f3ffd963d7ce170 WatchSource:0}: Error finding container ecd57d08fad9bca1f88fbc9847427a61df4f1aa1c5676a0a2f3ffd963d7ce170: Status 404 returned error can't find the container with id ecd57d08fad9bca1f88fbc9847427a61df4f1aa1c5676a0a2f3ffd963d7ce170
Oct 30 00:46:21.304908 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:21.304878    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 00:46:21.304994 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:21.304945    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 00:46:21.304994 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:21.304979    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 00:46:21.981811 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:21.981751    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-s5rzw" event={"ID":"cc529c8b-d888-412d-b861-f3648aaa473d","Type":"ContainerStarted","Data":"f05d8bc4d1168bc3c880bca7107be748bd8bee929866d757b6caa9787dcc0e2d"}
Oct 30 00:46:21.982560 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:21.982530    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-s5rzw" event={"ID":"cc529c8b-d888-412d-b861-f3648aaa473d","Type":"ContainerStarted","Data":"ecd57d08fad9bca1f88fbc9847427a61df4f1aa1c5676a0a2f3ffd963d7ce170"}
Oct 30 00:46:21.985376 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:21.985351    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-s5rzw"
Oct 30 00:46:21.996867 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:21.996827    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-vqjxg" event={"ID":"00e1797d-0629-4b79-8073-eaa093f48f2a","Type":"ContainerStarted","Data":"6a9e84f080cf9b9a34947b12801f2a10a8ada55922ca91adf63d0e553378486a"}
Oct 30 00:46:21.998816 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:21.998783    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-vqjxg" event={"ID":"00e1797d-0629-4b79-8073-eaa093f48f2a","Type":"ContainerStarted","Data":"1bf3e181715b960439a93de324f1da7cfcd1197f464b7d0caf84a501b5688011"}
Oct 30 00:46:22.034661 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:22.034613    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-s5rzw" podStartSLOduration=2.034598772 podStartE2EDuration="2.034598772s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:46:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:46:22.032566583 +0000 UTC m=+877.648999462" watchObservedRunningTime="2026-10-30 00:46:22.034598772 +0000 UTC m=+877.651031650"
Oct 30 00:46:22.898405 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:22.897440    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-vqjxg"
Oct 30 00:46:22.898405 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:22.897508    5002 patch_prober.go:29] interesting pod/e2e-worker-w6-021d6d-openshift-default-65c58b6947-vqjxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body=
Oct 30 00:46:22.898405 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:22.897622    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-vqjxg" podUID="00e1797d-0629-4b79-8073-eaa093f48f2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused"
Oct 30 00:46:22.990219 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:22.990181    5002 patch_prober.go:29] interesting pod/e2e-worker-w2-1a11cf-openshift-default-748ddd859-s5rzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body=
Oct 30 00:46:22.990855 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:22.990821    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-s5rzw" podUID="cc529c8b-d888-412d-b861-f3648aaa473d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused"
Oct 30 00:46:23.896402 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:23.895664    5002 patch_prober.go:29] interesting pod/e2e-worker-w6-021d6d-openshift-default-65c58b6947-vqjxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body=
Oct 30 00:46:23.896402 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:23.895729    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-vqjxg" podUID="00e1797d-0629-4b79-8073-eaa093f48f2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused"
Oct 30 00:46:23.987400 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:23.986788    5002 patch_prober.go:29] interesting pod/e2e-worker-w2-1a11cf-openshift-default-748ddd859-s5rzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body=
Oct 30 00:46:23.987400 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:23.986852    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-s5rzw" podUID="cc529c8b-d888-412d-b861-f3648aaa473d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused"
Oct 30 00:46:24.895406 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:24.894754    5002 patch_prober.go:29] interesting pod/e2e-worker-w6-021d6d-openshift-default-65c58b6947-vqjxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body=
Oct 30 00:46:24.895406 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:24.894814    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-vqjxg" podUID="00e1797d-0629-4b79-8073-eaa093f48f2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused"
Oct 30 00:46:24.986404 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:24.985845    5002 patch_prober.go:29] interesting pod/e2e-worker-w2-1a11cf-openshift-default-748ddd859-s5rzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body=
Oct 30 00:46:24.986404 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:24.985936    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-s5rzw" podUID="cc529c8b-d888-412d-b861-f3648aaa473d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused"
Oct 30 00:46:25.892402 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:25.891765    5002 patch_prober.go:29] interesting pod/e2e-worker-w6-021d6d-openshift-default-65c58b6947-vqjxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body=
Oct 30 00:46:25.892402 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:25.891830    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-vqjxg" podUID="00e1797d-0629-4b79-8073-eaa093f48f2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused"
Oct 30 00:46:25.988421 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:25.988291    5002 patch_prober.go:29] interesting pod/e2e-worker-w2-1a11cf-openshift-default-748ddd859-s5rzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body=
Oct 30 00:46:25.988421 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:25.988366    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-s5rzw" podUID="cc529c8b-d888-412d-b861-f3648aaa473d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused"
Oct 30 00:46:26.891835 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:26.891793    5002 patch_prober.go:29] interesting pod/e2e-worker-w6-021d6d-openshift-default-65c58b6947-vqjxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body=
Oct 30 00:46:26.892157 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:26.892127    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-vqjxg" podUID="00e1797d-0629-4b79-8073-eaa093f48f2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused"
Oct 30 00:46:26.986323 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:26.986274    5002 patch_prober.go:29] interesting pod/e2e-worker-w2-1a11cf-openshift-default-748ddd859-s5rzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body=
Oct 30 00:46:26.986664 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:26.986618    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-s5rzw" podUID="cc529c8b-d888-412d-b861-f3648aaa473d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused"
Oct 30 00:46:27.985789 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:27.985745    5002 patch_prober.go:29] interesting pod/e2e-worker-w2-1a11cf-openshift-default-748ddd859-s5rzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body=
Oct 30 00:46:27.986195 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:27.985824    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-s5rzw" podUID="cc529c8b-d888-412d-b861-f3648aaa473d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused"
Oct 30 00:46:28.121400 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:28.121353    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-vqjxg"
Oct 30 00:46:28.122566 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:28.122123    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-vqjxg"
Oct 30 00:46:28.124130 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:28.124110    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-vqjxg"
Oct 30 00:46:28.199168 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:28.199102    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-vqjxg" podStartSLOduration=8.199085637 podStartE2EDuration="8.199085637s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:46:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:46:22.085632517 +0000 UTC m=+877.702065397" watchObservedRunningTime="2026-10-30 00:46:28.199085637 +0000 UTC m=+883.815518516"
Oct 30 00:46:28.992215 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:28.991611    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-s5rzw"
Oct 30 00:46:28.992721 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:28.992329    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-s5rzw"
Oct 30 00:46:28.993114 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:28.993091    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-s5rzw"
Oct 30 00:46:35.368805 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:35.368770    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-ztgc9"]
Oct 30 00:46:35.404041 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:35.404006    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-ztgc9"
Oct 30 00:46:35.487701 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:35.487666    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-ztgc9"]
Oct 30 00:46:35.506054 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:35.506006    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3146dece-bcb2-4625-b507-fec96b90ba59-istio-envoy\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-ztgc9\" (UID: \"3146dece-bcb2-4625-b507-fec96b90ba59\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-ztgc9"
Oct 30 00:46:35.506416 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:35.506356    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6swfb\" (UniqueName: \"kubernetes.io/projected/3146dece-bcb2-4625-b507-fec96b90ba59-kube-api-access-6swfb\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-ztgc9\" (UID: \"3146dece-bcb2-4625-b507-fec96b90ba59\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-ztgc9"
Oct 30 00:46:35.506601 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:35.506584    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3146dece-bcb2-4625-b507-fec96b90ba59-istiod-ca-cert\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-ztgc9\" (UID: \"3146dece-bcb2-4625-b507-fec96b90ba59\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-ztgc9"
Oct 30 00:46:35.506727 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:35.506714    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3146dece-bcb2-4625-b507-fec96b90ba59-istio-podinfo\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-ztgc9\" (UID: \"3146dece-bcb2-4625-b507-fec96b90ba59\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-ztgc9"
Oct 30 00:46:35.506871 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:35.506840    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3146dece-bcb2-4625-b507-fec96b90ba59-istio-token\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-ztgc9\" (UID: \"3146dece-bcb2-4625-b507-fec96b90ba59\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-ztgc9"
Oct 30 00:46:35.506928 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:35.506909    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3146dece-bcb2-4625-b507-fec96b90ba59-istio-data\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-ztgc9\" (UID: \"3146dece-bcb2-4625-b507-fec96b90ba59\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-ztgc9"
Oct 30 00:46:35.506985 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:35.506943    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3146dece-bcb2-4625-b507-fec96b90ba59-workload-certs\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-ztgc9\" (UID: \"3146dece-bcb2-4625-b507-fec96b90ba59\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-ztgc9"
Oct 30 00:46:35.507079 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:35.507061    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3146dece-bcb2-4625-b507-fec96b90ba59-workload-socket\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-ztgc9\" (UID: \"3146dece-bcb2-4625-b507-fec96b90ba59\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-ztgc9"
Oct 30 00:46:35.507134 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:35.507109    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3146dece-bcb2-4625-b507-fec96b90ba59-credential-socket\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-ztgc9\" (UID: \"3146dece-bcb2-4625-b507-fec96b90ba59\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-ztgc9"
Oct 30 00:46:35.608069 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:35.608028    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3146dece-bcb2-4625-b507-fec96b90ba59-workload-socket\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-ztgc9\" (UID: \"3146dece-bcb2-4625-b507-fec96b90ba59\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-ztgc9"
Oct 30 00:46:35.608261 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:35.608097    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3146dece-bcb2-4625-b507-fec96b90ba59-credential-socket\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-ztgc9\" (UID: \"3146dece-bcb2-4625-b507-fec96b90ba59\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-ztgc9"
Oct 30 00:46:35.608261 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:35.608162    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3146dece-bcb2-4625-b507-fec96b90ba59-istio-envoy\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-ztgc9\" (UID: \"3146dece-bcb2-4625-b507-fec96b90ba59\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-ztgc9"
Oct 30 00:46:35.608261 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:35.608235    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6swfb\" (UniqueName: \"kubernetes.io/projected/3146dece-bcb2-4625-b507-fec96b90ba59-kube-api-access-6swfb\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-ztgc9\" (UID: \"3146dece-bcb2-4625-b507-fec96b90ba59\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-ztgc9"
Oct 30 00:46:35.608440 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:35.608290    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3146dece-bcb2-4625-b507-fec96b90ba59-istiod-ca-cert\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-ztgc9\" (UID: \"3146dece-bcb2-4625-b507-fec96b90ba59\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-ztgc9"
Oct 30 00:46:35.608440 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:35.608335    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3146dece-bcb2-4625-b507-fec96b90ba59-istio-podinfo\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-ztgc9\" (UID: \"3146dece-bcb2-4625-b507-fec96b90ba59\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-ztgc9"
Oct 30 00:46:35.608440 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:35.608367    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3146dece-bcb2-4625-b507-fec96b90ba59-istio-token\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-ztgc9\" (UID: \"3146dece-bcb2-4625-b507-fec96b90ba59\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-ztgc9"
Oct 30 00:46:35.608440 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:35.608431    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3146dece-bcb2-4625-b507-fec96b90ba59-istio-data\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-ztgc9\" (UID: \"3146dece-bcb2-4625-b507-fec96b90ba59\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-ztgc9"
Oct 30 00:46:35.608644 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:35.608467    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3146dece-bcb2-4625-b507-fec96b90ba59-workload-certs\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-ztgc9\" (UID: \"3146dece-bcb2-4625-b507-fec96b90ba59\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-ztgc9"
Oct 30 00:46:35.608644 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:35.608570    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3146dece-bcb2-4625-b507-fec96b90ba59-workload-socket\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-ztgc9\" (UID: \"3146dece-bcb2-4625-b507-fec96b90ba59\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-ztgc9"
Oct 30 00:46:35.608959 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:35.608933    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3146dece-bcb2-4625-b507-fec96b90ba59-credential-socket\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-ztgc9\" (UID: \"3146dece-bcb2-4625-b507-fec96b90ba59\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-ztgc9"
Oct 30 00:46:35.609156 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:35.609139    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3146dece-bcb2-4625-b507-fec96b90ba59-workload-certs\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-ztgc9\" (UID: \"3146dece-bcb2-4625-b507-fec96b90ba59\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-ztgc9"
Oct 30 00:46:35.609576 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:35.609543    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3146dece-bcb2-4625-b507-fec96b90ba59-istiod-ca-cert\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-ztgc9\" (UID: \"3146dece-bcb2-4625-b507-fec96b90ba59\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-ztgc9"
Oct 30 00:46:35.609711 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:35.609550    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3146dece-bcb2-4625-b507-fec96b90ba59-istio-data\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-ztgc9\" (UID: \"3146dece-bcb2-4625-b507-fec96b90ba59\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-ztgc9"
Oct 30 00:46:35.611564 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:35.611529    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3146dece-bcb2-4625-b507-fec96b90ba59-istio-envoy\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-ztgc9\" (UID: \"3146dece-bcb2-4625-b507-fec96b90ba59\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-ztgc9"
Oct 30 00:46:35.612083 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:35.612062    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3146dece-bcb2-4625-b507-fec96b90ba59-istio-podinfo\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-ztgc9\" (UID: \"3146dece-bcb2-4625-b507-fec96b90ba59\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-ztgc9"
Oct 30 00:46:35.634577 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:35.634506    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6swfb\" (UniqueName: \"kubernetes.io/projected/3146dece-bcb2-4625-b507-fec96b90ba59-kube-api-access-6swfb\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-ztgc9\" (UID: \"3146dece-bcb2-4625-b507-fec96b90ba59\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-ztgc9"
Oct 30 00:46:35.635699 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:35.635671    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3146dece-bcb2-4625-b507-fec96b90ba59-istio-token\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-ztgc9\" (UID: \"3146dece-bcb2-4625-b507-fec96b90ba59\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-ztgc9"
Oct 30 00:46:35.740209 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:35.740146    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-ztgc9"
Oct 30 00:46:36.110643 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:46:36.110609    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3146dece_bcb2_4625_b507_fec96b90ba59.slice/crio-1bf2b9029f3a5f7606afde95962d16a337e338584f9a3ae522003d4dcaf02031 WatchSource:0}: Error finding container 1bf2b9029f3a5f7606afde95962d16a337e338584f9a3ae522003d4dcaf02031: Status 404 returned error can't find the container with id 1bf2b9029f3a5f7606afde95962d16a337e338584f9a3ae522003d4dcaf02031
Oct 30 00:46:36.114409 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:36.114364    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-ztgc9"]
Oct 30 00:46:36.117595 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:36.117552    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 00:46:36.117715 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:36.117634    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 00:46:36.117715 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:36.117678    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 00:46:37.127987 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:37.127948    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-ztgc9" event={"ID":"3146dece-bcb2-4625-b507-fec96b90ba59","Type":"ContainerStarted","Data":"283945a992c2b052408456a0f9c268ce1e243fbcaff3b487e36158037732072f"}
Oct 30 00:46:37.128501 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:37.128012    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-ztgc9" event={"ID":"3146dece-bcb2-4625-b507-fec96b90ba59","Type":"ContainerStarted","Data":"1bf2b9029f3a5f7606afde95962d16a337e338584f9a3ae522003d4dcaf02031"}
Oct 30 00:46:37.184532 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:37.184346    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-ztgc9" podStartSLOduration=2.184324823 podStartE2EDuration="2.184324823s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:46:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:46:37.182878809 +0000 UTC m=+892.799311688" watchObservedRunningTime="2026-10-30 00:46:37.184324823 +0000 UTC m=+892.800757703"
Oct 30 00:46:37.740455 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:37.740426    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-ztgc9"
Oct 30 00:46:37.742156 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:37.742128    5002 patch_prober.go:29] interesting pod/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-ztgc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body=
Oct 30 00:46:37.742279 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:37.742176    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-ztgc9" podUID="3146dece-bcb2-4625-b507-fec96b90ba59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused"
Oct 30 00:46:38.741389 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:38.741341    5002 patch_prober.go:29] interesting pod/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-ztgc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body=
Oct 30 00:46:38.741882 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:38.741410    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-ztgc9" podUID="3146dece-bcb2-4625-b507-fec96b90ba59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused"
Oct 30 00:46:38.829005 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:38.828968    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-1a11cf/distinct-2-w2-1a11cf-kserve-758bb6dd57-j5f2k"]
Oct 30 00:46:38.833707 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:38.833681    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-1a11cf/distinct-2-w2-1a11cf-kserve-758bb6dd57-j5f2k"
Oct 30 00:46:38.863910 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:38.863878    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-1a11cf\"/\"distinct-2-w2-1a11cf-kserve-self-signed-certs\""
Oct 30 00:46:38.887536 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:38.887495    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-1a11cf/distinct-2-w2-1a11cf-kserve-758bb6dd57-j5f2k"]
Oct 30 00:46:38.993552 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:38.992184    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2d66dea-9f0b-43f3-a774-bb065e88f378-tls-certs\") pod \"distinct-2-w2-1a11cf-kserve-758bb6dd57-j5f2k\" (UID: \"f2d66dea-9f0b-43f3-a774-bb065e88f378\") " pod="e2e-models-e2e-worker-w2-1a11cf/distinct-2-w2-1a11cf-kserve-758bb6dd57-j5f2k"
Oct 30 00:46:38.993552 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:38.992232    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dklb\" (UniqueName: \"kubernetes.io/projected/f2d66dea-9f0b-43f3-a774-bb065e88f378-kube-api-access-7dklb\") pod \"distinct-2-w2-1a11cf-kserve-758bb6dd57-j5f2k\" (UID: \"f2d66dea-9f0b-43f3-a774-bb065e88f378\") " pod="e2e-models-e2e-worker-w2-1a11cf/distinct-2-w2-1a11cf-kserve-758bb6dd57-j5f2k"
Oct 30 00:46:38.993552 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:38.992267    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2d66dea-9f0b-43f3-a774-bb065e88f378-tmp-dir\") pod \"distinct-2-w2-1a11cf-kserve-758bb6dd57-j5f2k\" (UID: \"f2d66dea-9f0b-43f3-a774-bb065e88f378\") " pod="e2e-models-e2e-worker-w2-1a11cf/distinct-2-w2-1a11cf-kserve-758bb6dd57-j5f2k"
Oct 30 00:46:38.993552 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:38.992395    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2d66dea-9f0b-43f3-a774-bb065e88f378-dshm\") pod \"distinct-2-w2-1a11cf-kserve-758bb6dd57-j5f2k\" (UID: \"f2d66dea-9f0b-43f3-a774-bb065e88f378\") " pod="e2e-models-e2e-worker-w2-1a11cf/distinct-2-w2-1a11cf-kserve-758bb6dd57-j5f2k"
Oct 30 00:46:38.993552 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:38.992424    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2d66dea-9f0b-43f3-a774-bb065e88f378-home\") pod \"distinct-2-w2-1a11cf-kserve-758bb6dd57-j5f2k\" (UID: \"f2d66dea-9f0b-43f3-a774-bb065e88f378\") " pod="e2e-models-e2e-worker-w2-1a11cf/distinct-2-w2-1a11cf-kserve-758bb6dd57-j5f2k"
Oct 30 00:46:38.993552 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:38.992503    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2d66dea-9f0b-43f3-a774-bb065e88f378-model-cache\") pod \"distinct-2-w2-1a11cf-kserve-758bb6dd57-j5f2k\" (UID: \"f2d66dea-9f0b-43f3-a774-bb065e88f378\") " pod="e2e-models-e2e-worker-w2-1a11cf/distinct-2-w2-1a11cf-kserve-758bb6dd57-j5f2k"
Oct 30 00:46:39.098411 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:39.095259    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2d66dea-9f0b-43f3-a774-bb065e88f378-model-cache\") pod \"distinct-2-w2-1a11cf-kserve-758bb6dd57-j5f2k\" (UID: \"f2d66dea-9f0b-43f3-a774-bb065e88f378\") " pod="e2e-models-e2e-worker-w2-1a11cf/distinct-2-w2-1a11cf-kserve-758bb6dd57-j5f2k"
Oct 30 00:46:39.098411 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:39.095463    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2d66dea-9f0b-43f3-a774-bb065e88f378-tls-certs\") pod \"distinct-2-w2-1a11cf-kserve-758bb6dd57-j5f2k\" (UID: \"f2d66dea-9f0b-43f3-a774-bb065e88f378\") " pod="e2e-models-e2e-worker-w2-1a11cf/distinct-2-w2-1a11cf-kserve-758bb6dd57-j5f2k"
Oct 30 00:46:39.098411 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:39.095509    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dklb\" (UniqueName: \"kubernetes.io/projected/f2d66dea-9f0b-43f3-a774-bb065e88f378-kube-api-access-7dklb\") pod \"distinct-2-w2-1a11cf-kserve-758bb6dd57-j5f2k\" (UID: \"f2d66dea-9f0b-43f3-a774-bb065e88f378\") " pod="e2e-models-e2e-worker-w2-1a11cf/distinct-2-w2-1a11cf-kserve-758bb6dd57-j5f2k"
Oct 30 00:46:39.098411 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:39.095539    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2d66dea-9f0b-43f3-a774-bb065e88f378-tmp-dir\") pod \"distinct-2-w2-1a11cf-kserve-758bb6dd57-j5f2k\" (UID: \"f2d66dea-9f0b-43f3-a774-bb065e88f378\") " pod="e2e-models-e2e-worker-w2-1a11cf/distinct-2-w2-1a11cf-kserve-758bb6dd57-j5f2k"
Oct 30 00:46:39.098411 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:39.095659    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2d66dea-9f0b-43f3-a774-bb065e88f378-dshm\") pod \"distinct-2-w2-1a11cf-kserve-758bb6dd57-j5f2k\" (UID: \"f2d66dea-9f0b-43f3-a774-bb065e88f378\") " pod="e2e-models-e2e-worker-w2-1a11cf/distinct-2-w2-1a11cf-kserve-758bb6dd57-j5f2k"
Oct 30 00:46:39.098411 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:39.095684    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2d66dea-9f0b-43f3-a774-bb065e88f378-home\") pod \"distinct-2-w2-1a11cf-kserve-758bb6dd57-j5f2k\" (UID: \"f2d66dea-9f0b-43f3-a774-bb065e88f378\") " pod="e2e-models-e2e-worker-w2-1a11cf/distinct-2-w2-1a11cf-kserve-758bb6dd57-j5f2k"
Oct 30 00:46:39.098411 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:39.096240    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2d66dea-9f0b-43f3-a774-bb065e88f378-home\") pod \"distinct-2-w2-1a11cf-kserve-758bb6dd57-j5f2k\" (UID: \"f2d66dea-9f0b-43f3-a774-bb065e88f378\") " pod="e2e-models-e2e-worker-w2-1a11cf/distinct-2-w2-1a11cf-kserve-758bb6dd57-j5f2k"
Oct 30 00:46:39.098411 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:39.096527    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2d66dea-9f0b-43f3-a774-bb065e88f378-model-cache\") pod \"distinct-2-w2-1a11cf-kserve-758bb6dd57-j5f2k\" (UID: \"f2d66dea-9f0b-43f3-a774-bb065e88f378\") " pod="e2e-models-e2e-worker-w2-1a11cf/distinct-2-w2-1a11cf-kserve-758bb6dd57-j5f2k"
Oct 30 00:46:39.105476 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:39.104813    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2d66dea-9f0b-43f3-a774-bb065e88f378-tmp-dir\") pod \"distinct-2-w2-1a11cf-kserve-758bb6dd57-j5f2k\" (UID: \"f2d66dea-9f0b-43f3-a774-bb065e88f378\") " pod="e2e-models-e2e-worker-w2-1a11cf/distinct-2-w2-1a11cf-kserve-758bb6dd57-j5f2k"
Oct 30 00:46:39.109594 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:39.109531    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2d66dea-9f0b-43f3-a774-bb065e88f378-dshm\") pod \"distinct-2-w2-1a11cf-kserve-758bb6dd57-j5f2k\" (UID: \"f2d66dea-9f0b-43f3-a774-bb065e88f378\") " pod="e2e-models-e2e-worker-w2-1a11cf/distinct-2-w2-1a11cf-kserve-758bb6dd57-j5f2k"
Oct 30 00:46:39.123205 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:39.123049    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2d66dea-9f0b-43f3-a774-bb065e88f378-tls-certs\") pod \"distinct-2-w2-1a11cf-kserve-758bb6dd57-j5f2k\" (UID: \"f2d66dea-9f0b-43f3-a774-bb065e88f378\") " pod="e2e-models-e2e-worker-w2-1a11cf/distinct-2-w2-1a11cf-kserve-758bb6dd57-j5f2k"
Oct 30 00:46:39.140106 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:39.137134    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dklb\" (UniqueName: \"kubernetes.io/projected/f2d66dea-9f0b-43f3-a774-bb065e88f378-kube-api-access-7dklb\") pod \"distinct-2-w2-1a11cf-kserve-758bb6dd57-j5f2k\" (UID: \"f2d66dea-9f0b-43f3-a774-bb065e88f378\") " pod="e2e-models-e2e-worker-w2-1a11cf/distinct-2-w2-1a11cf-kserve-758bb6dd57-j5f2k"
Oct 30 00:46:39.204909 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:39.204595    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-1a11cf/distinct-2-w2-1a11cf-kserve-758bb6dd57-j5f2k"
Oct 30 00:46:39.572897 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:39.572804    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-1a11cf/distinct-2-w2-1a11cf-kserve-758bb6dd57-j5f2k"]
Oct 30 00:46:39.586831 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:46:39.586794    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf2d66dea_9f0b_43f3_a774_bb065e88f378.slice/crio-7cbb4f5bff2ac65c0d1c45834e666fa2ebb5b8499382af1454c7223c42ff432a WatchSource:0}: Error finding container 7cbb4f5bff2ac65c0d1c45834e666fa2ebb5b8499382af1454c7223c42ff432a: Status 404 returned error can't find the container with id 7cbb4f5bff2ac65c0d1c45834e666fa2ebb5b8499382af1454c7223c42ff432a
Oct 30 00:46:39.741510 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:39.741469    5002 patch_prober.go:29] interesting pod/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-ztgc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body=
Oct 30 00:46:39.741951 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:39.741565    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-ztgc9" podUID="3146dece-bcb2-4625-b507-fec96b90ba59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused"
Oct 30 00:46:40.186365 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:40.186323    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-1a11cf/distinct-2-w2-1a11cf-kserve-758bb6dd57-j5f2k" event={"ID":"f2d66dea-9f0b-43f3-a774-bb065e88f378","Type":"ContainerStarted","Data":"b601a0f189cbbee81fcc9087c896e022ebe09ab1a82eee2857b16e53c827c75c"}
Oct 30 00:46:40.186365 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:40.186374    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-1a11cf/distinct-2-w2-1a11cf-kserve-758bb6dd57-j5f2k" event={"ID":"f2d66dea-9f0b-43f3-a774-bb065e88f378","Type":"ContainerStarted","Data":"7cbb4f5bff2ac65c0d1c45834e666fa2ebb5b8499382af1454c7223c42ff432a"}
Oct 30 00:46:40.205631 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:40.205602    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-1a11cf/distinct-2-w2-1a11cf-kserve-758bb6dd57-j5f2k"
Oct 30 00:46:40.231292 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:40.231239    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-1a11cf/distinct-2-w2-1a11cf-kserve-758bb6dd57-j5f2k" podStartSLOduration=2.231224347 podStartE2EDuration="2.231224347s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:46:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:46:40.230539442 +0000 UTC m=+895.846972340" watchObservedRunningTime="2026-10-30 00:46:40.231224347 +0000 UTC m=+895.847657226"
Oct 30 00:46:40.741454 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:40.741422    5002 patch_prober.go:29] interesting pod/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-ztgc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body=
Oct 30 00:46:40.741634 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:40.741473    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-ztgc9" podUID="3146dece-bcb2-4625-b507-fec96b90ba59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused"
Oct 30 00:46:41.740883 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:41.740850    5002 patch_prober.go:29] interesting pod/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-ztgc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body=
Oct 30 00:46:41.741070 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:41.740906    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-ztgc9" podUID="3146dece-bcb2-4625-b507-fec96b90ba59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused"
Oct 30 00:46:42.743645 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:42.743118    5002 patch_prober.go:29] interesting pod/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-ztgc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body=
Oct 30 00:46:42.743645 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:42.743218    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-ztgc9" podUID="3146dece-bcb2-4625-b507-fec96b90ba59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused"
Oct 30 00:46:42.870794 ip-10-0-128-7 kubenswrapper[5002]: E1030 00:46:42.870751    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbjdq" containerName="extractor"
Oct 30 00:46:43.809253 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:43.809217    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-ztgc9"
Oct 30 00:46:43.810069 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:43.810042    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-ztgc9"
Oct 30 00:46:43.810821 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:43.810803    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-ztgc9"
Oct 30 00:46:44.582065 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:44.582033    5002 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-g7c2w_5dc56e7e-f3c1-43eb-a21b-8be6bb4706cf/console-operator/1.log"
Oct 30 00:46:44.587331 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:44.587305    5002 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-g7c2w_5dc56e7e-f3c1-43eb-a21b-8be6bb4706cf/console-operator/1.log"
Oct 30 00:46:49.205570 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:49.205537    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-1a11cf/distinct-2-w2-1a11cf-kserve-758bb6dd57-j5f2k"
Oct 30 00:46:49.218241 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:49.218210    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-1a11cf/distinct-2-w2-1a11cf-kserve-758bb6dd57-j5f2k"
Oct 30 00:46:49.249917 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:49.249887    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-1a11cf/distinct-2-w2-1a11cf-kserve-758bb6dd57-j5f2k"
Oct 30 00:46:50.842463 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:50.842425    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-fjlsm"]
Oct 30 00:46:50.847906 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:50.847884    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-fjlsm"
Oct 30 00:46:50.884498 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:50.884462    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-fjlsm"]
Oct 30 00:46:50.885236 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:50.885196    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-52650f-openshift-default-dockercfg-5tktc\""
Oct 30 00:46:50.942455 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:50.942376    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c20eb99-f057-44bd-94e2-1972b67aa7f0-workload-certs\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-fjlsm\" (UID: \"9c20eb99-f057-44bd-94e2-1972b67aa7f0\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-fjlsm"
Oct 30 00:46:50.942633 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:50.942520    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c20eb99-f057-44bd-94e2-1972b67aa7f0-istio-data\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-fjlsm\" (UID: \"9c20eb99-f057-44bd-94e2-1972b67aa7f0\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-fjlsm"
Oct 30 00:46:50.942633 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:50.942569    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c20eb99-f057-44bd-94e2-1972b67aa7f0-istio-podinfo\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-fjlsm\" (UID: \"9c20eb99-f057-44bd-94e2-1972b67aa7f0\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-fjlsm"
Oct 30 00:46:50.942728 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:50.942715    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c20eb99-f057-44bd-94e2-1972b67aa7f0-istio-token\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-fjlsm\" (UID: \"9c20eb99-f057-44bd-94e2-1972b67aa7f0\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-fjlsm"
Oct 30 00:46:50.942798 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:50.942784    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxkq7\" (UniqueName: \"kubernetes.io/projected/9c20eb99-f057-44bd-94e2-1972b67aa7f0-kube-api-access-kxkq7\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-fjlsm\" (UID: \"9c20eb99-f057-44bd-94e2-1972b67aa7f0\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-fjlsm"
Oct 30 00:46:50.942853 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:50.942838    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c20eb99-f057-44bd-94e2-1972b67aa7f0-credential-socket\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-fjlsm\" (UID: \"9c20eb99-f057-44bd-94e2-1972b67aa7f0\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-fjlsm"
Oct 30 00:46:50.942906 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:50.942869    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c20eb99-f057-44bd-94e2-1972b67aa7f0-workload-socket\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-fjlsm\" (UID: \"9c20eb99-f057-44bd-94e2-1972b67aa7f0\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-fjlsm"
Oct 30 00:46:50.943016 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:50.942993    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c20eb99-f057-44bd-94e2-1972b67aa7f0-istio-envoy\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-fjlsm\" (UID: \"9c20eb99-f057-44bd-94e2-1972b67aa7f0\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-fjlsm"
Oct 30 00:46:50.943086 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:50.943061    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c20eb99-f057-44bd-94e2-1972b67aa7f0-istiod-ca-cert\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-fjlsm\" (UID: \"9c20eb99-f057-44bd-94e2-1972b67aa7f0\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-fjlsm"
Oct 30 00:46:51.044308 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:51.044268    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c20eb99-f057-44bd-94e2-1972b67aa7f0-credential-socket\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-fjlsm\" (UID: \"9c20eb99-f057-44bd-94e2-1972b67aa7f0\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-fjlsm"
Oct 30 00:46:51.044527 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:51.044321    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c20eb99-f057-44bd-94e2-1972b67aa7f0-workload-socket\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-fjlsm\" (UID: \"9c20eb99-f057-44bd-94e2-1972b67aa7f0\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-fjlsm"
Oct 30 00:46:51.044527 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:51.044341    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c20eb99-f057-44bd-94e2-1972b67aa7f0-istio-envoy\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-fjlsm\" (UID: \"9c20eb99-f057-44bd-94e2-1972b67aa7f0\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-fjlsm"
Oct 30 00:46:51.044527 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:51.044375    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c20eb99-f057-44bd-94e2-1972b67aa7f0-istiod-ca-cert\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-fjlsm\" (UID: \"9c20eb99-f057-44bd-94e2-1972b67aa7f0\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-fjlsm"
Oct 30 00:46:51.044527 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:51.044443    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c20eb99-f057-44bd-94e2-1972b67aa7f0-workload-certs\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-fjlsm\" (UID: \"9c20eb99-f057-44bd-94e2-1972b67aa7f0\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-fjlsm"
Oct 30 00:46:51.044527 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:51.044466    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c20eb99-f057-44bd-94e2-1972b67aa7f0-istio-data\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-fjlsm\" (UID: \"9c20eb99-f057-44bd-94e2-1972b67aa7f0\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-fjlsm"
Oct 30 00:46:51.044527 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:51.044490    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c20eb99-f057-44bd-94e2-1972b67aa7f0-istio-podinfo\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-fjlsm\" (UID: \"9c20eb99-f057-44bd-94e2-1972b67aa7f0\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-fjlsm"
Oct 30 00:46:51.044851 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:51.044565    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c20eb99-f057-44bd-94e2-1972b67aa7f0-istio-token\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-fjlsm\" (UID: \"9c20eb99-f057-44bd-94e2-1972b67aa7f0\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-fjlsm"
Oct 30 00:46:51.044851 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:51.044619    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxkq7\" (UniqueName: \"kubernetes.io/projected/9c20eb99-f057-44bd-94e2-1972b67aa7f0-kube-api-access-kxkq7\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-fjlsm\" (UID: \"9c20eb99-f057-44bd-94e2-1972b67aa7f0\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-fjlsm"
Oct 30 00:46:51.044851 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:51.044751    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c20eb99-f057-44bd-94e2-1972b67aa7f0-credential-socket\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-fjlsm\" (UID: \"9c20eb99-f057-44bd-94e2-1972b67aa7f0\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-fjlsm"
Oct 30 00:46:51.044851 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:51.044797    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c20eb99-f057-44bd-94e2-1972b67aa7f0-workload-socket\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-fjlsm\" (UID: \"9c20eb99-f057-44bd-94e2-1972b67aa7f0\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-fjlsm"
Oct 30 00:46:51.045060 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:51.044936    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c20eb99-f057-44bd-94e2-1972b67aa7f0-workload-certs\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-fjlsm\" (UID: \"9c20eb99-f057-44bd-94e2-1972b67aa7f0\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-fjlsm"
Oct 30 00:46:51.045162 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:51.045140    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c20eb99-f057-44bd-94e2-1972b67aa7f0-istio-data\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-fjlsm\" (UID: \"9c20eb99-f057-44bd-94e2-1972b67aa7f0\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-fjlsm"
Oct 30 00:46:51.045260 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:51.045234    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c20eb99-f057-44bd-94e2-1972b67aa7f0-istiod-ca-cert\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-fjlsm\" (UID: \"9c20eb99-f057-44bd-94e2-1972b67aa7f0\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-fjlsm"
Oct 30 00:46:51.046718 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:51.046691    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c20eb99-f057-44bd-94e2-1972b67aa7f0-istio-envoy\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-fjlsm\" (UID: \"9c20eb99-f057-44bd-94e2-1972b67aa7f0\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-fjlsm"
Oct 30 00:46:51.047076 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:51.047057    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c20eb99-f057-44bd-94e2-1972b67aa7f0-istio-podinfo\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-fjlsm\" (UID: \"9c20eb99-f057-44bd-94e2-1972b67aa7f0\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-fjlsm"
Oct 30 00:46:51.069873 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:51.069834    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c20eb99-f057-44bd-94e2-1972b67aa7f0-istio-token\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-fjlsm\" (UID: \"9c20eb99-f057-44bd-94e2-1972b67aa7f0\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-fjlsm"
Oct 30 00:46:51.070012 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:51.069957    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxkq7\" (UniqueName: \"kubernetes.io/projected/9c20eb99-f057-44bd-94e2-1972b67aa7f0-kube-api-access-kxkq7\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-fjlsm\" (UID: \"9c20eb99-f057-44bd-94e2-1972b67aa7f0\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-fjlsm"
Oct 30 00:46:51.165019 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:51.164934    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-fjlsm"
Oct 30 00:46:51.488482 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:46:51.488421    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c20eb99_f057_44bd_94e2_1972b67aa7f0.slice/crio-74b5e630cd91f614eafd425f493ba3b70e9d665b92376541263d0dc1f7f8b470 WatchSource:0}: Error finding container 74b5e630cd91f614eafd425f493ba3b70e9d665b92376541263d0dc1f7f8b470: Status 404 returned error can't find the container with id 74b5e630cd91f614eafd425f493ba3b70e9d665b92376541263d0dc1f7f8b470
Oct 30 00:46:51.488838 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:51.488545    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-fjlsm"]
Oct 30 00:46:51.491128 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:51.491090    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 00:46:51.491237 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:51.491169    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 00:46:51.491237 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:51.491215    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 00:46:52.275403 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:52.272795    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-fjlsm" event={"ID":"9c20eb99-f057-44bd-94e2-1972b67aa7f0","Type":"ContainerStarted","Data":"77e997a5afac45add3fe0b7b8a7e5224e426820f35fd950ee3e772038f1b2c5e"}
Oct 30 00:46:52.275403 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:52.272868    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-fjlsm" event={"ID":"9c20eb99-f057-44bd-94e2-1972b67aa7f0","Type":"ContainerStarted","Data":"74b5e630cd91f614eafd425f493ba3b70e9d665b92376541263d0dc1f7f8b470"}
Oct 30 00:46:52.337730 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:52.337674    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-fjlsm" podStartSLOduration=2.337658831 podStartE2EDuration="2.337658831s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:46:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:46:52.335626033 +0000 UTC m=+907.952058914" watchObservedRunningTime="2026-10-30 00:46:52.337658831 +0000 UTC m=+907.954091706"
Oct 30 00:46:53.169404 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:53.168737    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-fjlsm"
Oct 30 00:46:53.174398 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:53.172486    5002 patch_prober.go:29] interesting pod/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-fjlsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body=
Oct 30 00:46:53.174398 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:53.172550    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-fjlsm" podUID="9c20eb99-f057-44bd-94e2-1972b67aa7f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Oct 30 00:46:53.536540 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:53.536506    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7d8z4"]
Oct 30 00:46:54.175792 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:54.175548    5002 patch_prober.go:29] interesting pod/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-fjlsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body=
Oct 30 00:46:54.175792 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:54.175621    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-fjlsm" podUID="9c20eb99-f057-44bd-94e2-1972b67aa7f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Oct 30 00:46:55.165782 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:55.165742    5002 patch_prober.go:29] interesting pod/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-fjlsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body=
Oct 30 00:46:55.166351 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:55.166320    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-fjlsm" podUID="9c20eb99-f057-44bd-94e2-1972b67aa7f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Oct 30 00:46:56.171402 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:56.170646    5002 patch_prober.go:29] interesting pod/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-fjlsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body=
Oct 30 00:46:56.171402 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:56.170709    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-fjlsm" podUID="9c20eb99-f057-44bd-94e2-1972b67aa7f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Oct 30 00:46:56.667436 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:56.665889    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cefb27/simulator-w3-cefb27-kserve-6b669c9d9f-zx2vm" podUID="7121e81e-f903-42c0-a7a4-bc18d49194e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:46:56.733526 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:56.733472    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6f4fba/simulator-w5-6f4fba-kserve-5bf8dfddd5-wcwzh" podUID="3a2c2f8f-649c-4ab6-af98-5d87df0e511f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:46:56.768728 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:56.768681    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-52650f/distinct-w0-52650f-kserve-d85899b5-vlhps" podUID="ea1850d9-abba-4977-a540-f6e0eaefb17d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:46:56.824549 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:56.824499    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-52650f/simulator-w0-52650f-kserve-58b64766c-94qxx" podUID="659c1425-5b67-4d01-bfc2-f1ba56b9a59b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": context deadline exceeded"
Oct 30 00:46:56.825845 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:56.825810    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-29be73/simulator-w1-29be73-kserve-9958d76db-fks2t" podUID="fbb3c806-8967-4d39-84c6-9dc022fc5cae" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:46:56.854913 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:56.854869    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-021d6d/simulator-w6-021d6d-kserve-6cc97fdf86-dqtpw" podUID="50586f76-4f3b-4e7a-83ac-6c573faee415" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:46:56.941771 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:56.940766    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca5363/distinct-2-w4-ca5363-kserve-576576f9b5-xkkqc" podUID="5de8bebf-cfad-49da-9a98-08aedde4442c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:46:57.018591 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:57.018524    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-29be73/premium-w1-29be73-kserve-7b777cfc8c-7kgp5" podUID="e6fabbdd-86ca-4cbf-85be-ef250575a355" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:46:57.032565 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:57.032502    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-52650f/premium-w0-52650f-kserve-c5bf66f9d-qq5dm" podUID="3455d983-b11b-4cee-bfce-d1f399cf9679" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:46:57.175543 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:57.175506    5002 patch_prober.go:29] interesting pod/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-fjlsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body=
Oct 30 00:46:57.176162 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:57.176110    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-fjlsm" podUID="9c20eb99-f057-44bd-94e2-1972b67aa7f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Oct 30 00:46:57.248570 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:57.248462    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1a11cf/distinct-2-w2-1a11cf-kserve-758bb6dd57-j5f2k" podUID="f2d66dea-9f0b-43f3-a774-bb065e88f378" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:46:57.738520 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:57.736472    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6f4fba/simulator-w5-6f4fba-kserve-5bf8dfddd5-wcwzh" podUID="3a2c2f8f-649c-4ab6-af98-5d87df0e511f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/ready\": context deadline exceeded"
Oct 30 00:46:57.738520 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:57.737101    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-6f4fba/simulator-w5-6f4fba-kserve-5bf8dfddd5-wcwzh"
Oct 30 00:46:57.776521 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:57.776470    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-52650f/distinct-w0-52650f-kserve-d85899b5-vlhps" podUID="ea1850d9-abba-4977-a540-f6e0eaefb17d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:46:57.777152 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:57.777126    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-52650f/distinct-w0-52650f-kserve-d85899b5-vlhps"
Oct 30 00:46:57.950957 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:57.950888    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-29be73/simulator-w1-29be73-kserve-9958d76db-fks2t" podUID="fbb3c806-8967-4d39-84c6-9dc022fc5cae" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:46:57.991466 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:57.968485    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-29be73/simulator-w1-29be73-kserve-9958d76db-fks2t"
Oct 30 00:46:58.028070 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:58.019193    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-29be73/premium-w1-29be73-kserve-7b777cfc8c-7kgp5" podUID="e6fabbdd-86ca-4cbf-85be-ef250575a355" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:46:58.047721 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:58.047671    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-52650f/premium-w0-52650f-kserve-c5bf66f9d-qq5dm" podUID="3455d983-b11b-4cee-bfce-d1f399cf9679" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:46:58.048784 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:58.048759    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-29be73/premium-w1-29be73-kserve-7b777cfc8c-7kgp5"
Oct 30 00:46:58.070540 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:58.070508    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-52650f/premium-w0-52650f-kserve-c5bf66f9d-qq5dm"
Oct 30 00:46:58.193406 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:58.190630    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-6f4fba/simulator-w5-6f4fba-kserve-5bf8dfddd5-wcwzh"
Oct 30 00:46:58.193406 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:58.191168    5002 patch_prober.go:29] interesting pod/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-fjlsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body=
Oct 30 00:46:58.193406 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:58.191214    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-fjlsm" podUID="9c20eb99-f057-44bd-94e2-1972b67aa7f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Oct 30 00:46:58.239403 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:58.231103    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-52650f/distinct-w0-52650f-kserve-d85899b5-vlhps"
Oct 30 00:46:58.239403 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:58.233413    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-29be73/simulator-w1-29be73-kserve-9958d76db-fks2t"
Oct 30 00:46:58.239403 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:58.236677    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-52650f/premium-w0-52650f-kserve-c5bf66f9d-qq5dm"
Oct 30 00:46:58.300544 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:58.300467    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-29be73/premium-w1-29be73-kserve-7b777cfc8c-7kgp5"
Oct 30 00:46:59.165781 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:59.165737    5002 patch_prober.go:29] interesting pod/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-fjlsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body=
Oct 30 00:46:59.165988 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:46:59.165802    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-fjlsm" podUID="9c20eb99-f057-44bd-94e2-1972b67aa7f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Oct 30 00:47:00.171543 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:00.171503    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-fjlsm"
Oct 30 00:47:00.171993 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:00.171915    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-fjlsm"
Oct 30 00:47:00.172727 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:00.172699    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-fjlsm"
Oct 30 00:47:12.862606 ip-10-0-128-7 kubenswrapper[5002]: E1030 00:47:12.862559    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbjdq" containerName="extractor"
Oct 30 00:47:17.468877 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:17.468839    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eabf2670-bg57b"]
Oct 30 00:47:17.473912 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:17.473892    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eabf2670-bg57b"
Oct 30 00:47:17.492630 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:17.492598    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eabf2670-bg57b"]
Oct 30 00:47:17.502455 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:17.502420    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 00:47:17.502628 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:17.502428    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 00:47:17.502850 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:17.502822    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 00:47:17.504895 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:17.504873    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5phsh\""
Oct 30 00:47:17.558908 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:17.558875    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf85f00d-1e7a-458c-8cc3-bc8eebc361c0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eabf2670-bg57b\" (UID: \"bf85f00d-1e7a-458c-8cc3-bc8eebc361c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eabf2670-bg57b"
Oct 30 00:47:17.559077 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:17.558940    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf85f00d-1e7a-458c-8cc3-bc8eebc361c0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eabf2670-bg57b\" (UID: \"bf85f00d-1e7a-458c-8cc3-bc8eebc361c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eabf2670-bg57b"
Oct 30 00:47:17.559077 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:17.559046    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htzd2\" (UniqueName: \"kubernetes.io/projected/bf85f00d-1e7a-458c-8cc3-bc8eebc361c0-kube-api-access-htzd2\") pod \"maas-api-revoke-sub-eabf2670-bg57b\" (UID: \"bf85f00d-1e7a-458c-8cc3-bc8eebc361c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eabf2670-bg57b"
Oct 30 00:47:17.660074 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:17.660040    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htzd2\" (UniqueName: \"kubernetes.io/projected/bf85f00d-1e7a-458c-8cc3-bc8eebc361c0-kube-api-access-htzd2\") pod \"maas-api-revoke-sub-eabf2670-bg57b\" (UID: \"bf85f00d-1e7a-458c-8cc3-bc8eebc361c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eabf2670-bg57b"
Oct 30 00:47:17.660244 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:17.660185    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf85f00d-1e7a-458c-8cc3-bc8eebc361c0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eabf2670-bg57b\" (UID: \"bf85f00d-1e7a-458c-8cc3-bc8eebc361c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eabf2670-bg57b"
Oct 30 00:47:17.660244 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:17.660231    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf85f00d-1e7a-458c-8cc3-bc8eebc361c0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eabf2670-bg57b\" (UID: \"bf85f00d-1e7a-458c-8cc3-bc8eebc361c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eabf2670-bg57b"
Oct 30 00:47:17.660937 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:17.660905    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf85f00d-1e7a-458c-8cc3-bc8eebc361c0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eabf2670-bg57b\" (UID: \"bf85f00d-1e7a-458c-8cc3-bc8eebc361c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eabf2670-bg57b"
Oct 30 00:47:17.662609 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:17.662584    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf85f00d-1e7a-458c-8cc3-bc8eebc361c0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eabf2670-bg57b\" (UID: \"bf85f00d-1e7a-458c-8cc3-bc8eebc361c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eabf2670-bg57b"
Oct 30 00:47:17.687321 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:17.687291    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-htzd2\" (UniqueName: \"kubernetes.io/projected/bf85f00d-1e7a-458c-8cc3-bc8eebc361c0-kube-api-access-htzd2\") pod \"maas-api-revoke-sub-eabf2670-bg57b\" (UID: \"bf85f00d-1e7a-458c-8cc3-bc8eebc361c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eabf2670-bg57b"
Oct 30 00:47:17.788092 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:17.787992    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eabf2670-bg57b"
Oct 30 00:47:18.096812 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:18.096783    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eabf2670-bg57b"]
Oct 30 00:47:18.097545 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:47:18.097508    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf85f00d_1e7a_458c_8cc3_bc8eebc361c0.slice/crio-2d5dcf22e8ca240f4a22948fb6970120deee5cf2a89f92f3381941a178954c1c WatchSource:0}: Error finding container 2d5dcf22e8ca240f4a22948fb6970120deee5cf2a89f92f3381941a178954c1c: Status 404 returned error can't find the container with id 2d5dcf22e8ca240f4a22948fb6970120deee5cf2a89f92f3381941a178954c1c
Oct 30 00:47:18.702870 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:18.702828    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eabf2670-bg57b" event={"ID":"bf85f00d-1e7a-458c-8cc3-bc8eebc361c0","Type":"ContainerStarted","Data":"29ca663c81e62a7da53693c4b8536acba9b2157c98f854d3a288b6df166484ad"}
Oct 30 00:47:18.703352 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:18.702878    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eabf2670-bg57b" event={"ID":"bf85f00d-1e7a-458c-8cc3-bc8eebc361c0","Type":"ContainerStarted","Data":"2d5dcf22e8ca240f4a22948fb6970120deee5cf2a89f92f3381941a178954c1c"}
Oct 30 00:47:18.748790 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:18.748731    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eabf2670-bg57b" podStartSLOduration=1.748716366 podStartE2EDuration="1.748716366s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:47:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:47:18.745658574 +0000 UTC m=+934.362091453" watchObservedRunningTime="2026-10-30 00:47:18.748716366 +0000 UTC m=+934.365149245"
Oct 30 00:47:19.711414 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:19.711348    5002 generic.go:356] "Generic (PLEG): container finished" podID="bf85f00d-1e7a-458c-8cc3-bc8eebc361c0" containerID="29ca663c81e62a7da53693c4b8536acba9b2157c98f854d3a288b6df166484ad" exitCode=0
Oct 30 00:47:19.711822 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:19.711416    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eabf2670-bg57b" event={"ID":"bf85f00d-1e7a-458c-8cc3-bc8eebc361c0","Type":"ContainerDied","Data":"29ca663c81e62a7da53693c4b8536acba9b2157c98f854d3a288b6df166484ad"}
Oct 30 00:47:20.481706 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:20.481665    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-xfr4r"]
Oct 30 00:47:20.485363 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:20.485339    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-xfr4r"
Oct 30 00:47:20.513134 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:20.513100    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-xfr4r"]
Oct 30 00:47:20.598678 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:20.598642    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/735d76ff-0add-4acc-b5e9-4ebeb27c0e9a-istio-token\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-xfr4r\" (UID: \"735d76ff-0add-4acc-b5e9-4ebeb27c0e9a\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-xfr4r"
Oct 30 00:47:20.598854 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:20.598704    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/735d76ff-0add-4acc-b5e9-4ebeb27c0e9a-workload-certs\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-xfr4r\" (UID: \"735d76ff-0add-4acc-b5e9-4ebeb27c0e9a\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-xfr4r"
Oct 30 00:47:20.598854 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:20.598758    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzp59\" (UniqueName: \"kubernetes.io/projected/735d76ff-0add-4acc-b5e9-4ebeb27c0e9a-kube-api-access-jzp59\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-xfr4r\" (UID: \"735d76ff-0add-4acc-b5e9-4ebeb27c0e9a\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-xfr4r"
Oct 30 00:47:20.599013 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:20.598977    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/735d76ff-0add-4acc-b5e9-4ebeb27c0e9a-istio-envoy\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-xfr4r\" (UID: \"735d76ff-0add-4acc-b5e9-4ebeb27c0e9a\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-xfr4r"
Oct 30 00:47:20.599149 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:20.599073    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/735d76ff-0add-4acc-b5e9-4ebeb27c0e9a-istiod-ca-cert\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-xfr4r\" (UID: \"735d76ff-0add-4acc-b5e9-4ebeb27c0e9a\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-xfr4r"
Oct 30 00:47:20.599149 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:20.599138    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/735d76ff-0add-4acc-b5e9-4ebeb27c0e9a-credential-socket\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-xfr4r\" (UID: \"735d76ff-0add-4acc-b5e9-4ebeb27c0e9a\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-xfr4r"
Oct 30 00:47:20.599262 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:20.599163    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/735d76ff-0add-4acc-b5e9-4ebeb27c0e9a-istio-podinfo\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-xfr4r\" (UID: \"735d76ff-0add-4acc-b5e9-4ebeb27c0e9a\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-xfr4r"
Oct 30 00:47:20.599347 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:20.599329    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/735d76ff-0add-4acc-b5e9-4ebeb27c0e9a-istio-data\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-xfr4r\" (UID: \"735d76ff-0add-4acc-b5e9-4ebeb27c0e9a\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-xfr4r"
Oct 30 00:47:20.599448 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:20.599374    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/735d76ff-0add-4acc-b5e9-4ebeb27c0e9a-workload-socket\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-xfr4r\" (UID: \"735d76ff-0add-4acc-b5e9-4ebeb27c0e9a\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-xfr4r"
Oct 30 00:47:20.700973 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:20.700931    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/735d76ff-0add-4acc-b5e9-4ebeb27c0e9a-istio-envoy\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-xfr4r\" (UID: \"735d76ff-0add-4acc-b5e9-4ebeb27c0e9a\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-xfr4r"
Oct 30 00:47:20.701150 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:20.700999    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/735d76ff-0add-4acc-b5e9-4ebeb27c0e9a-istiod-ca-cert\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-xfr4r\" (UID: \"735d76ff-0add-4acc-b5e9-4ebeb27c0e9a\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-xfr4r"
Oct 30 00:47:20.701150 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:20.701028    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/735d76ff-0add-4acc-b5e9-4ebeb27c0e9a-credential-socket\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-xfr4r\" (UID: \"735d76ff-0add-4acc-b5e9-4ebeb27c0e9a\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-xfr4r"
Oct 30 00:47:20.701150 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:20.701056    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/735d76ff-0add-4acc-b5e9-4ebeb27c0e9a-istio-podinfo\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-xfr4r\" (UID: \"735d76ff-0add-4acc-b5e9-4ebeb27c0e9a\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-xfr4r"
Oct 30 00:47:20.701150 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:20.701102    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/735d76ff-0add-4acc-b5e9-4ebeb27c0e9a-istio-data\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-xfr4r\" (UID: \"735d76ff-0add-4acc-b5e9-4ebeb27c0e9a\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-xfr4r"
Oct 30 00:47:20.701150 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:20.701134    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/735d76ff-0add-4acc-b5e9-4ebeb27c0e9a-workload-socket\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-xfr4r\" (UID: \"735d76ff-0add-4acc-b5e9-4ebeb27c0e9a\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-xfr4r"
Oct 30 00:47:20.701453 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:20.701211    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/735d76ff-0add-4acc-b5e9-4ebeb27c0e9a-istio-token\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-xfr4r\" (UID: \"735d76ff-0add-4acc-b5e9-4ebeb27c0e9a\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-xfr4r"
Oct 30 00:47:20.701453 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:20.701254    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/735d76ff-0add-4acc-b5e9-4ebeb27c0e9a-workload-certs\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-xfr4r\" (UID: \"735d76ff-0add-4acc-b5e9-4ebeb27c0e9a\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-xfr4r"
Oct 30 00:47:20.701453 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:20.701308    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzp59\" (UniqueName: \"kubernetes.io/projected/735d76ff-0add-4acc-b5e9-4ebeb27c0e9a-kube-api-access-jzp59\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-xfr4r\" (UID: \"735d76ff-0add-4acc-b5e9-4ebeb27c0e9a\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-xfr4r"
Oct 30 00:47:20.701601 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:20.701568    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/735d76ff-0add-4acc-b5e9-4ebeb27c0e9a-istio-data\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-xfr4r\" (UID: \"735d76ff-0add-4acc-b5e9-4ebeb27c0e9a\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-xfr4r"
Oct 30 00:47:20.701788 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:20.701770    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/735d76ff-0add-4acc-b5e9-4ebeb27c0e9a-workload-certs\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-xfr4r\" (UID: \"735d76ff-0add-4acc-b5e9-4ebeb27c0e9a\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-xfr4r"
Oct 30 00:47:20.701998 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:20.701967    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/735d76ff-0add-4acc-b5e9-4ebeb27c0e9a-istiod-ca-cert\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-xfr4r\" (UID: \"735d76ff-0add-4acc-b5e9-4ebeb27c0e9a\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-xfr4r"
Oct 30 00:47:20.702088 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:20.701974    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/735d76ff-0add-4acc-b5e9-4ebeb27c0e9a-credential-socket\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-xfr4r\" (UID: \"735d76ff-0add-4acc-b5e9-4ebeb27c0e9a\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-xfr4r"
Oct 30 00:47:20.702088 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:20.702074    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/735d76ff-0add-4acc-b5e9-4ebeb27c0e9a-workload-socket\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-xfr4r\" (UID: \"735d76ff-0add-4acc-b5e9-4ebeb27c0e9a\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-xfr4r"
Oct 30 00:47:20.703674 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:20.703643    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/735d76ff-0add-4acc-b5e9-4ebeb27c0e9a-istio-envoy\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-xfr4r\" (UID: \"735d76ff-0add-4acc-b5e9-4ebeb27c0e9a\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-xfr4r"
Oct 30 00:47:20.703758 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:20.703703    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/735d76ff-0add-4acc-b5e9-4ebeb27c0e9a-istio-podinfo\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-xfr4r\" (UID: \"735d76ff-0add-4acc-b5e9-4ebeb27c0e9a\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-xfr4r"
Oct 30 00:47:20.710056 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:20.710030    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-x8wgq"]
Oct 30 00:47:20.714655 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:20.714632    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-x8wgq"
Oct 30 00:47:20.733136 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:20.733109    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/735d76ff-0add-4acc-b5e9-4ebeb27c0e9a-istio-token\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-xfr4r\" (UID: \"735d76ff-0add-4acc-b5e9-4ebeb27c0e9a\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-xfr4r"
Oct 30 00:47:20.733242 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:20.733227    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzp59\" (UniqueName: \"kubernetes.io/projected/735d76ff-0add-4acc-b5e9-4ebeb27c0e9a-kube-api-access-jzp59\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-xfr4r\" (UID: \"735d76ff-0add-4acc-b5e9-4ebeb27c0e9a\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-xfr4r"
Oct 30 00:47:20.740356 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:20.740320    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-x8wgq"]
Oct 30 00:47:20.801838 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:20.801800    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-xfr4r"
Oct 30 00:47:20.802714 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:20.802687    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c534def0-b8f3-4be1-8f47-72bffc1234e1-workload-socket\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-x8wgq\" (UID: \"c534def0-b8f3-4be1-8f47-72bffc1234e1\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-x8wgq"
Oct 30 00:47:20.802881 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:20.802858    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c534def0-b8f3-4be1-8f47-72bffc1234e1-istio-token\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-x8wgq\" (UID: \"c534def0-b8f3-4be1-8f47-72bffc1234e1\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-x8wgq"
Oct 30 00:47:20.802947 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:20.802930    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c534def0-b8f3-4be1-8f47-72bffc1234e1-workload-certs\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-x8wgq\" (UID: \"c534def0-b8f3-4be1-8f47-72bffc1234e1\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-x8wgq"
Oct 30 00:47:20.803059 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:20.803041    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c534def0-b8f3-4be1-8f47-72bffc1234e1-istio-podinfo\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-x8wgq\" (UID: \"c534def0-b8f3-4be1-8f47-72bffc1234e1\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-x8wgq"
Oct 30 00:47:20.803119 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:20.803077    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c534def0-b8f3-4be1-8f47-72bffc1234e1-credential-socket\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-x8wgq\" (UID: \"c534def0-b8f3-4be1-8f47-72bffc1234e1\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-x8wgq"
Oct 30 00:47:20.803176 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:20.803119    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c534def0-b8f3-4be1-8f47-72bffc1234e1-istio-data\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-x8wgq\" (UID: \"c534def0-b8f3-4be1-8f47-72bffc1234e1\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-x8wgq"
Oct 30 00:47:20.803176 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:20.803167    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgxbc\" (UniqueName: \"kubernetes.io/projected/c534def0-b8f3-4be1-8f47-72bffc1234e1-kube-api-access-fgxbc\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-x8wgq\" (UID: \"c534def0-b8f3-4be1-8f47-72bffc1234e1\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-x8wgq"
Oct 30 00:47:20.803279 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:20.803201    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c534def0-b8f3-4be1-8f47-72bffc1234e1-istio-envoy\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-x8wgq\" (UID: \"c534def0-b8f3-4be1-8f47-72bffc1234e1\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-x8wgq"
Oct 30 00:47:20.803279 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:20.803237    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c534def0-b8f3-4be1-8f47-72bffc1234e1-istiod-ca-cert\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-x8wgq\" (UID: \"c534def0-b8f3-4be1-8f47-72bffc1234e1\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-x8wgq"
Oct 30 00:47:20.843585 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:20.843555    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-7rsqp"]
Oct 30 00:47:20.867019 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:20.866993    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-7rsqp"
Oct 30 00:47:20.873876 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:20.873762    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-7rsqp"]
Oct 30 00:47:20.908266 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:20.908230    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c534def0-b8f3-4be1-8f47-72bffc1234e1-istio-token\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-x8wgq\" (UID: \"c534def0-b8f3-4be1-8f47-72bffc1234e1\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-x8wgq"
Oct 30 00:47:20.910540 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:20.910216    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c534def0-b8f3-4be1-8f47-72bffc1234e1-workload-certs\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-x8wgq\" (UID: \"c534def0-b8f3-4be1-8f47-72bffc1234e1\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-x8wgq"
Oct 30 00:47:20.910540 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:20.910405    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c534def0-b8f3-4be1-8f47-72bffc1234e1-istio-podinfo\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-x8wgq\" (UID: \"c534def0-b8f3-4be1-8f47-72bffc1234e1\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-x8wgq"
Oct 30 00:47:20.910889 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:20.910868    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c534def0-b8f3-4be1-8f47-72bffc1234e1-credential-socket\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-x8wgq\" (UID: \"c534def0-b8f3-4be1-8f47-72bffc1234e1\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-x8wgq"
Oct 30 00:47:20.910986 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:20.910886    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c534def0-b8f3-4be1-8f47-72bffc1234e1-workload-certs\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-x8wgq\" (UID: \"c534def0-b8f3-4be1-8f47-72bffc1234e1\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-x8wgq"
Oct 30 00:47:20.910986 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:20.910867    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c534def0-b8f3-4be1-8f47-72bffc1234e1-credential-socket\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-x8wgq\" (UID: \"c534def0-b8f3-4be1-8f47-72bffc1234e1\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-x8wgq"
Oct 30 00:47:20.911105 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:20.911000    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c534def0-b8f3-4be1-8f47-72bffc1234e1-istio-data\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-x8wgq\" (UID: \"c534def0-b8f3-4be1-8f47-72bffc1234e1\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-x8wgq"
Oct 30 00:47:20.911185 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:20.911164    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgxbc\" (UniqueName: \"kubernetes.io/projected/c534def0-b8f3-4be1-8f47-72bffc1234e1-kube-api-access-fgxbc\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-x8wgq\" (UID: \"c534def0-b8f3-4be1-8f47-72bffc1234e1\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-x8wgq"
Oct 30 00:47:20.911266 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:20.911225    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c534def0-b8f3-4be1-8f47-72bffc1234e1-istio-envoy\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-x8wgq\" (UID: \"c534def0-b8f3-4be1-8f47-72bffc1234e1\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-x8wgq"
Oct 30 00:47:20.911326 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:20.911269    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c534def0-b8f3-4be1-8f47-72bffc1234e1-istiod-ca-cert\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-x8wgq\" (UID: \"c534def0-b8f3-4be1-8f47-72bffc1234e1\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-x8wgq"
Oct 30 00:47:20.911326 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:20.911297    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c534def0-b8f3-4be1-8f47-72bffc1234e1-istio-data\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-x8wgq\" (UID: \"c534def0-b8f3-4be1-8f47-72bffc1234e1\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-x8wgq"
Oct 30 00:47:20.911553 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:20.911337    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c534def0-b8f3-4be1-8f47-72bffc1234e1-workload-socket\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-x8wgq\" (UID: \"c534def0-b8f3-4be1-8f47-72bffc1234e1\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-x8wgq"
Oct 30 00:47:20.911836 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:20.911818    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c534def0-b8f3-4be1-8f47-72bffc1234e1-workload-socket\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-x8wgq\" (UID: \"c534def0-b8f3-4be1-8f47-72bffc1234e1\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-x8wgq"
Oct 30 00:47:20.913405 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:20.913370    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eabf2670-bg57b"
Oct 30 00:47:20.913675 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:20.913647    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c534def0-b8f3-4be1-8f47-72bffc1234e1-istiod-ca-cert\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-x8wgq\" (UID: \"c534def0-b8f3-4be1-8f47-72bffc1234e1\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-x8wgq"
Oct 30 00:47:20.914785 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:20.914757    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c534def0-b8f3-4be1-8f47-72bffc1234e1-istio-podinfo\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-x8wgq\" (UID: \"c534def0-b8f3-4be1-8f47-72bffc1234e1\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-x8wgq"
Oct 30 00:47:20.916754 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:20.916731    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c534def0-b8f3-4be1-8f47-72bffc1234e1-istio-envoy\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-x8wgq\" (UID: \"c534def0-b8f3-4be1-8f47-72bffc1234e1\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-x8wgq"
Oct 30 00:47:20.935264 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:20.935240    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c534def0-b8f3-4be1-8f47-72bffc1234e1-istio-token\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-x8wgq\" (UID: \"c534def0-b8f3-4be1-8f47-72bffc1234e1\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-x8wgq"
Oct 30 00:47:20.937578 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:20.937554    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgxbc\" (UniqueName: \"kubernetes.io/projected/c534def0-b8f3-4be1-8f47-72bffc1234e1-kube-api-access-fgxbc\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-x8wgq\" (UID: \"c534def0-b8f3-4be1-8f47-72bffc1234e1\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-x8wgq"
Oct 30 00:47:21.012305 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:21.012218    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bf85f00d-1e7a-458c-8cc3-bc8eebc361c0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf85f00d-1e7a-458c-8cc3-bc8eebc361c0-maas-api-xks-ca\") pod \"bf85f00d-1e7a-458c-8cc3-bc8eebc361c0\" (UID: \"bf85f00d-1e7a-458c-8cc3-bc8eebc361c0\") "
Oct 30 00:47:21.012485 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:21.012416    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf85f00d-1e7a-458c-8cc3-bc8eebc361c0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf85f00d-1e7a-458c-8cc3-bc8eebc361c0-maas-api-openshift-ca\") pod \"bf85f00d-1e7a-458c-8cc3-bc8eebc361c0\" (UID: \"bf85f00d-1e7a-458c-8cc3-bc8eebc361c0\") "
Oct 30 00:47:21.012565 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:21.012518    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf85f00d-1e7a-458c-8cc3-bc8eebc361c0-kube-api-access-htzd2\" (UniqueName: \"kubernetes.io/projected/bf85f00d-1e7a-458c-8cc3-bc8eebc361c0-kube-api-access-htzd2\") pod \"bf85f00d-1e7a-458c-8cc3-bc8eebc361c0\" (UID: \"bf85f00d-1e7a-458c-8cc3-bc8eebc361c0\") "
Oct 30 00:47:21.012744 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:21.012726    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79f13664-8af6-4e86-bf3f-8f2dcd645788-workload-certs\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-7rsqp\" (UID: \"79f13664-8af6-4e86-bf3f-8f2dcd645788\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-7rsqp"
Oct 30 00:47:21.012846 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:21.012764    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79f13664-8af6-4e86-bf3f-8f2dcd645788-istio-data\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-7rsqp\" (UID: \"79f13664-8af6-4e86-bf3f-8f2dcd645788\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-7rsqp"
Oct 30 00:47:21.012846 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:21.012815    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf85f00d-1e7a-458c-8cc3-bc8eebc361c0-maas-api-openshift-ca" pod "bf85f00d-1e7a-458c-8cc3-bc8eebc361c0" (UID: "bf85f00d-1e7a-458c-8cc3-bc8eebc361c0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:47:21.012963 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:21.012923    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79f13664-8af6-4e86-bf3f-8f2dcd645788-istio-envoy\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-7rsqp\" (UID: \"79f13664-8af6-4e86-bf3f-8f2dcd645788\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-7rsqp"
Oct 30 00:47:21.013017 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:21.012968    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79f13664-8af6-4e86-bf3f-8f2dcd645788-workload-socket\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-7rsqp\" (UID: \"79f13664-8af6-4e86-bf3f-8f2dcd645788\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-7rsqp"
Oct 30 00:47:21.013071 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:21.013053    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79f13664-8af6-4e86-bf3f-8f2dcd645788-credential-socket\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-7rsqp\" (UID: \"79f13664-8af6-4e86-bf3f-8f2dcd645788\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-7rsqp"
Oct 30 00:47:21.013145 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:21.013111    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79f13664-8af6-4e86-bf3f-8f2dcd645788-istio-token\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-7rsqp\" (UID: \"79f13664-8af6-4e86-bf3f-8f2dcd645788\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-7rsqp"
Oct 30 00:47:21.013193 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:21.013178    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdqnc\" (UniqueName: \"kubernetes.io/projected/79f13664-8af6-4e86-bf3f-8f2dcd645788-kube-api-access-cdqnc\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-7rsqp\" (UID: \"79f13664-8af6-4e86-bf3f-8f2dcd645788\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-7rsqp"
Oct 30 00:47:21.013293 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:21.013272    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79f13664-8af6-4e86-bf3f-8f2dcd645788-istiod-ca-cert\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-7rsqp\" (UID: \"79f13664-8af6-4e86-bf3f-8f2dcd645788\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-7rsqp"
Oct 30 00:47:21.013336 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:21.013323    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79f13664-8af6-4e86-bf3f-8f2dcd645788-istio-podinfo\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-7rsqp\" (UID: \"79f13664-8af6-4e86-bf3f-8f2dcd645788\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-7rsqp"
Oct 30 00:47:21.013579 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:21.013562    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf85f00d-1e7a-458c-8cc3-bc8eebc361c0-maas-api-openshift-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:47:21.014647 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:21.014627    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf85f00d-1e7a-458c-8cc3-bc8eebc361c0-kube-api-access-htzd2" pod "bf85f00d-1e7a-458c-8cc3-bc8eebc361c0" (UID: "bf85f00d-1e7a-458c-8cc3-bc8eebc361c0"). InnerVolumeSpecName "kube-api-access-htzd2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:47:21.014885 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:21.014867    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf85f00d-1e7a-458c-8cc3-bc8eebc361c0-maas-api-xks-ca" pod "bf85f00d-1e7a-458c-8cc3-bc8eebc361c0" (UID: "bf85f00d-1e7a-458c-8cc3-bc8eebc361c0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:47:21.078273 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:21.078234    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-x8wgq"
Oct 30 00:47:21.114634 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:21.114601    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79f13664-8af6-4e86-bf3f-8f2dcd645788-istio-envoy\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-7rsqp\" (UID: \"79f13664-8af6-4e86-bf3f-8f2dcd645788\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-7rsqp"
Oct 30 00:47:21.114797 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:21.114663    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79f13664-8af6-4e86-bf3f-8f2dcd645788-workload-socket\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-7rsqp\" (UID: \"79f13664-8af6-4e86-bf3f-8f2dcd645788\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-7rsqp"
Oct 30 00:47:21.114797 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:21.114687    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79f13664-8af6-4e86-bf3f-8f2dcd645788-credential-socket\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-7rsqp\" (UID: \"79f13664-8af6-4e86-bf3f-8f2dcd645788\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-7rsqp"
Oct 30 00:47:21.114797 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:21.114712    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79f13664-8af6-4e86-bf3f-8f2dcd645788-istio-token\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-7rsqp\" (UID: \"79f13664-8af6-4e86-bf3f-8f2dcd645788\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-7rsqp"
Oct 30 00:47:21.114976 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:21.114843    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdqnc\" (UniqueName: \"kubernetes.io/projected/79f13664-8af6-4e86-bf3f-8f2dcd645788-kube-api-access-cdqnc\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-7rsqp\" (UID: \"79f13664-8af6-4e86-bf3f-8f2dcd645788\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-7rsqp"
Oct 30 00:47:21.114976 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:21.114935    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79f13664-8af6-4e86-bf3f-8f2dcd645788-istiod-ca-cert\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-7rsqp\" (UID: \"79f13664-8af6-4e86-bf3f-8f2dcd645788\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-7rsqp"
Oct 30 00:47:21.115086 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:21.114978    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79f13664-8af6-4e86-bf3f-8f2dcd645788-istio-podinfo\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-7rsqp\" (UID: \"79f13664-8af6-4e86-bf3f-8f2dcd645788\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-7rsqp"
Oct 30 00:47:21.115142 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:21.115108    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79f13664-8af6-4e86-bf3f-8f2dcd645788-credential-socket\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-7rsqp\" (UID: \"79f13664-8af6-4e86-bf3f-8f2dcd645788\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-7rsqp"
Oct 30 00:47:21.115196 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:21.115140    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79f13664-8af6-4e86-bf3f-8f2dcd645788-workload-socket\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-7rsqp\" (UID: \"79f13664-8af6-4e86-bf3f-8f2dcd645788\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-7rsqp"
Oct 30 00:47:21.115250 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:21.115214    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79f13664-8af6-4e86-bf3f-8f2dcd645788-workload-certs\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-7rsqp\" (UID: \"79f13664-8af6-4e86-bf3f-8f2dcd645788\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-7rsqp"
Oct 30 00:47:21.115300 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:21.115248    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79f13664-8af6-4e86-bf3f-8f2dcd645788-istio-data\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-7rsqp\" (UID: \"79f13664-8af6-4e86-bf3f-8f2dcd645788\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-7rsqp"
Oct 30 00:47:21.115425 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:21.115403    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-htzd2\" (UniqueName: \"kubernetes.io/projected/bf85f00d-1e7a-458c-8cc3-bc8eebc361c0-kube-api-access-htzd2\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:47:21.115511 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:21.115431    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf85f00d-1e7a-458c-8cc3-bc8eebc361c0-maas-api-xks-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:47:21.115577 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:21.115557    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79f13664-8af6-4e86-bf3f-8f2dcd645788-workload-certs\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-7rsqp\" (UID: \"79f13664-8af6-4e86-bf3f-8f2dcd645788\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-7rsqp"
Oct 30 00:47:21.115677 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:21.115654    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79f13664-8af6-4e86-bf3f-8f2dcd645788-istio-data\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-7rsqp\" (UID: \"79f13664-8af6-4e86-bf3f-8f2dcd645788\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-7rsqp"
Oct 30 00:47:21.115753 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:21.115729    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79f13664-8af6-4e86-bf3f-8f2dcd645788-istiod-ca-cert\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-7rsqp\" (UID: \"79f13664-8af6-4e86-bf3f-8f2dcd645788\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-7rsqp"
Oct 30 00:47:21.117027 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:21.117005    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79f13664-8af6-4e86-bf3f-8f2dcd645788-istio-envoy\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-7rsqp\" (UID: \"79f13664-8af6-4e86-bf3f-8f2dcd645788\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-7rsqp"
Oct 30 00:47:21.117369 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:21.117352    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79f13664-8af6-4e86-bf3f-8f2dcd645788-istio-podinfo\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-7rsqp\" (UID: \"79f13664-8af6-4e86-bf3f-8f2dcd645788\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-7rsqp"
Oct 30 00:47:21.141628 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:21.141606    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdqnc\" (UniqueName: \"kubernetes.io/projected/79f13664-8af6-4e86-bf3f-8f2dcd645788-kube-api-access-cdqnc\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-7rsqp\" (UID: \"79f13664-8af6-4e86-bf3f-8f2dcd645788\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-7rsqp"
Oct 30 00:47:21.142825 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:21.142796    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79f13664-8af6-4e86-bf3f-8f2dcd645788-istio-token\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-7rsqp\" (UID: \"79f13664-8af6-4e86-bf3f-8f2dcd645788\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-7rsqp"
Oct 30 00:47:21.167468 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:47:21.167428    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod735d76ff_0add_4acc_b5e9_4ebeb27c0e9a.slice/crio-4df93890b733b3e6591f722aa6814b0d7eeff070f61a686c57331afb1bbe726f WatchSource:0}: Error finding container 4df93890b733b3e6591f722aa6814b0d7eeff070f61a686c57331afb1bbe726f: Status 404 returned error can't find the container with id 4df93890b733b3e6591f722aa6814b0d7eeff070f61a686c57331afb1bbe726f
Oct 30 00:47:21.169071 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:21.169047    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-xfr4r"]
Oct 30 00:47:21.170783 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:21.170745    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 00:47:21.170891 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:21.170829    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 00:47:21.170891 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:21.170871    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 00:47:21.188286 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:21.188249    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-7rsqp"
Oct 30 00:47:21.383698 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:21.383664    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-x8wgq"]
Oct 30 00:47:21.384307 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:47:21.383948    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc534def0_b8f3_4be1_8f47_72bffc1234e1.slice/crio-e0317a5d45bc42c47adbc0b2b854d0ce5d28c912eaa770656245b1acd14ed083 WatchSource:0}: Error finding container e0317a5d45bc42c47adbc0b2b854d0ce5d28c912eaa770656245b1acd14ed083: Status 404 returned error can't find the container with id e0317a5d45bc42c47adbc0b2b854d0ce5d28c912eaa770656245b1acd14ed083
Oct 30 00:47:21.387083 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:21.387050    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 00:47:21.387177 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:21.387124    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 00:47:21.387177 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:21.387156    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 00:47:21.541064 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:21.541040    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-7rsqp"]
Oct 30 00:47:21.542746 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:47:21.542708    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79f13664_8af6_4e86_bf3f_8f2dcd645788.slice/crio-5db85cf4f22afa67643679ba8abdce7e4bb78f9bfa2f9bda88821c74812b6dd5 WatchSource:0}: Error finding container 5db85cf4f22afa67643679ba8abdce7e4bb78f9bfa2f9bda88821c74812b6dd5: Status 404 returned error can't find the container with id 5db85cf4f22afa67643679ba8abdce7e4bb78f9bfa2f9bda88821c74812b6dd5
Oct 30 00:47:21.545635 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:21.545605    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 00:47:21.545739 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:21.545682    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 00:47:21.545802 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:21.545737    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 00:47:21.734754 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:21.734713    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-xfr4r" event={"ID":"735d76ff-0add-4acc-b5e9-4ebeb27c0e9a","Type":"ContainerStarted","Data":"6dd8fa824e1c5a0d12aab0bf1b534816167511bc151c9dc23fa04fef6e7e01c7"}
Oct 30 00:47:21.735232 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:21.734784    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-xfr4r" event={"ID":"735d76ff-0add-4acc-b5e9-4ebeb27c0e9a","Type":"ContainerStarted","Data":"4df93890b733b3e6591f722aa6814b0d7eeff070f61a686c57331afb1bbe726f"}
Oct 30 00:47:21.737515 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:21.737489    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eabf2670-bg57b"
Oct 30 00:47:21.737650 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:21.737493    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eabf2670-bg57b" event={"ID":"bf85f00d-1e7a-458c-8cc3-bc8eebc361c0","Type":"ContainerDied","Data":"2d5dcf22e8ca240f4a22948fb6970120deee5cf2a89f92f3381941a178954c1c"}
Oct 30 00:47:21.737650 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:21.737631    5002 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2d5dcf22e8ca240f4a22948fb6970120deee5cf2a89f92f3381941a178954c1c"
Oct 30 00:47:21.743772 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:21.743688    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-x8wgq" event={"ID":"c534def0-b8f3-4be1-8f47-72bffc1234e1","Type":"ContainerStarted","Data":"1f4486a5b827e8159aab1d7ebd1bac12a4071091b2a303abb7a71ed51131269d"}
Oct 30 00:47:21.743772 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:21.743744    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-x8wgq" event={"ID":"c534def0-b8f3-4be1-8f47-72bffc1234e1","Type":"ContainerStarted","Data":"e0317a5d45bc42c47adbc0b2b854d0ce5d28c912eaa770656245b1acd14ed083"}
Oct 30 00:47:21.746550 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:21.745962    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-7rsqp" event={"ID":"79f13664-8af6-4e86-bf3f-8f2dcd645788","Type":"ContainerStarted","Data":"d6e427cdce0e74b1ec0aca7c222adf7df59ae9bdaa1dbaf7b41417d71810200d"}
Oct 30 00:47:21.746550 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:21.746002    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-7rsqp" event={"ID":"79f13664-8af6-4e86-bf3f-8f2dcd645788","Type":"ContainerStarted","Data":"5db85cf4f22afa67643679ba8abdce7e4bb78f9bfa2f9bda88821c74812b6dd5"}
Oct 30 00:47:21.795630 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:21.795528    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-xfr4r" podStartSLOduration=1.795511593 podStartE2EDuration="1.795511593s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:47:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:47:21.789512391 +0000 UTC m=+937.405945296" watchObservedRunningTime="2026-10-30 00:47:21.795511593 +0000 UTC m=+937.411944472"
Oct 30 00:47:21.802361 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:21.802329    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-xfr4r"
Oct 30 00:47:21.855144 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:21.855092    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-7rsqp" podStartSLOduration=1.855078648 podStartE2EDuration="1.855078648s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:47:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:47:21.854671097 +0000 UTC m=+937.471103976" watchObservedRunningTime="2026-10-30 00:47:21.855078648 +0000 UTC m=+937.471511526"
Oct 30 00:47:21.905834 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:21.905783    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-x8wgq" podStartSLOduration=1.905769681 podStartE2EDuration="1.905769681s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:47:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:47:21.902815972 +0000 UTC m=+937.519248857" watchObservedRunningTime="2026-10-30 00:47:21.905769681 +0000 UTC m=+937.522202560"
Oct 30 00:47:22.081159 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:22.081070    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-x8wgq"
Oct 30 00:47:22.081544 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:22.081521    5002 patch_prober.go:29] interesting pod/e2e-worker-w6-021d6d-openshift-default-65c58b6947-x8wgq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body=
Oct 30 00:47:22.084413 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:22.081686    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-x8wgq" podUID="c534def0-b8f3-4be1-8f47-72bffc1234e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused"
Oct 30 00:47:22.198404 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:22.189744    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-7rsqp"
Oct 30 00:47:22.198404 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:22.197504    5002 patch_prober.go:29] interesting pod/e2e-worker-w2-1a11cf-openshift-default-748ddd859-7rsqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body=
Oct 30 00:47:22.198404 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:22.197568    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-7rsqp" podUID="79f13664-8af6-4e86-bf3f-8f2dcd645788" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused"
Oct 30 00:47:22.814545 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:22.814502    5002 patch_prober.go:29] interesting pod/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-xfr4r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body=
Oct 30 00:47:22.815144 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:22.815108    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-xfr4r" podUID="735d76ff-0add-4acc-b5e9-4ebeb27c0e9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused"
Oct 30 00:47:23.081457 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:23.078817    5002 patch_prober.go:29] interesting pod/e2e-worker-w6-021d6d-openshift-default-65c58b6947-x8wgq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body=
Oct 30 00:47:23.081457 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:23.078884    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-x8wgq" podUID="c534def0-b8f3-4be1-8f47-72bffc1234e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused"
Oct 30 00:47:23.198405 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:23.197963    5002 patch_prober.go:29] interesting pod/e2e-worker-w2-1a11cf-openshift-default-748ddd859-7rsqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body=
Oct 30 00:47:23.198405 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:23.198025    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-7rsqp" podUID="79f13664-8af6-4e86-bf3f-8f2dcd645788" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused"
Oct 30 00:47:23.825638 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:23.825604    5002 patch_prober.go:29] interesting pod/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-xfr4r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body=
Oct 30 00:47:23.826207 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:23.826177    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-xfr4r" podUID="735d76ff-0add-4acc-b5e9-4ebeb27c0e9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused"
Oct 30 00:47:24.082461 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:24.081404    5002 patch_prober.go:29] interesting pod/e2e-worker-w6-021d6d-openshift-default-65c58b6947-x8wgq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body=
Oct 30 00:47:24.082461 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:24.081476    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-x8wgq" podUID="c534def0-b8f3-4be1-8f47-72bffc1234e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused"
Oct 30 00:47:24.193403 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:24.192639    5002 patch_prober.go:29] interesting pod/e2e-worker-w2-1a11cf-openshift-default-748ddd859-7rsqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body=
Oct 30 00:47:24.193403 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:24.192714    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-7rsqp" podUID="79f13664-8af6-4e86-bf3f-8f2dcd645788" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused"
Oct 30 00:47:24.900423 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:24.835575    5002 patch_prober.go:29] interesting pod/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-xfr4r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body=
Oct 30 00:47:24.900423 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:24.835634    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-xfr4r" podUID="735d76ff-0add-4acc-b5e9-4ebeb27c0e9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused"
Oct 30 00:47:25.105521 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:25.104181    5002 patch_prober.go:29] interesting pod/e2e-worker-w6-021d6d-openshift-default-65c58b6947-x8wgq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body=
Oct 30 00:47:25.105521 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:25.104240    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-x8wgq" podUID="c534def0-b8f3-4be1-8f47-72bffc1234e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused"
Oct 30 00:47:25.407878 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:25.354518    5002 patch_prober.go:29] interesting pod/e2e-worker-w2-1a11cf-openshift-default-748ddd859-7rsqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body=
Oct 30 00:47:25.407878 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:25.354581    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-7rsqp" podUID="79f13664-8af6-4e86-bf3f-8f2dcd645788" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused"
Oct 30 00:47:25.837441 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:25.837317    5002 patch_prober.go:29] interesting pod/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-xfr4r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body=
Oct 30 00:47:25.837735 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:25.837690    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-xfr4r" podUID="735d76ff-0add-4acc-b5e9-4ebeb27c0e9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused"
Oct 30 00:47:25.866247 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:25.866187    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1a11cf/distinct-w2-1a11cf-kserve-854cf4cdf8-cddw6" podUID="32a03478-46eb-4363-97b6-51a32c4099c7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:47:25.896437 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:25.872273    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-021d6d/simulator-w6-021d6d-kserve-6cc97fdf86-dqtpw" podUID="50586f76-4f3b-4e7a-83ac-6c573faee415" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:47:25.947824 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:25.947755    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca5363/distinct-2-w4-ca5363-kserve-576576f9b5-xkkqc" podUID="5de8bebf-cfad-49da-9a98-08aedde4442c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:47:26.080881 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:26.080838    5002 patch_prober.go:29] interesting pod/e2e-worker-w6-021d6d-openshift-default-65c58b6947-x8wgq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body=
Oct 30 00:47:26.081150 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:26.081113    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-x8wgq" podUID="c534def0-b8f3-4be1-8f47-72bffc1234e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused"
Oct 30 00:47:26.169047 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:26.168956    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6f4fba/simulator-w5-6f4fba-kserve-5bf8dfddd5-wcwzh" podUID="3a2c2f8f-649c-4ab6-af98-5d87df0e511f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:47:26.206463 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:26.206427    5002 patch_prober.go:29] interesting pod/e2e-worker-w2-1a11cf-openshift-default-748ddd859-7rsqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body=
Oct 30 00:47:26.206736 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:26.206709    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-7rsqp" podUID="79f13664-8af6-4e86-bf3f-8f2dcd645788" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused"
Oct 30 00:47:26.236021 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:26.235979    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-29be73/simulator-w1-29be73-kserve-9958d76db-fks2t" podUID="fbb3c806-8967-4d39-84c6-9dc022fc5cae" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/ready\": context deadline exceeded"
Oct 30 00:47:26.237559 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:26.237511    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-52650f/premium-w0-52650f-kserve-c5bf66f9d-qq5dm" podUID="3455d983-b11b-4cee-bfce-d1f399cf9679" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:47:26.240674 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:26.240636    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-52650f/distinct-w0-52650f-kserve-d85899b5-vlhps" podUID="ea1850d9-abba-4977-a540-f6e0eaefb17d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:47:26.250663 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:26.249469    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1a11cf/distinct-2-w2-1a11cf-kserve-758bb6dd57-j5f2k" podUID="f2d66dea-9f0b-43f3-a774-bb065e88f378" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded"
Oct 30 00:47:26.339220 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:26.338173    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-29be73/premium-w1-29be73-kserve-7b777cfc8c-7kgp5" podUID="e6fabbdd-86ca-4cbf-85be-ef250575a355" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:47:26.665861 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:26.665812    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cefb27/simulator-w3-cefb27-kserve-6b669c9d9f-zx2vm" podUID="7121e81e-f903-42c0-a7a4-bc18d49194e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:47:26.820403 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:26.811340    5002 patch_prober.go:29] interesting pod/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-xfr4r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body=
Oct 30 00:47:26.820403 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:26.811481    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-xfr4r" podUID="735d76ff-0add-4acc-b5e9-4ebeb27c0e9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused"
Oct 30 00:47:26.894405 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:26.869489    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1a11cf/distinct-w2-1a11cf-kserve-854cf4cdf8-cddw6" podUID="32a03478-46eb-4363-97b6-51a32c4099c7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded"
Oct 30 00:47:26.894405 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:26.869949    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-1a11cf/distinct-w2-1a11cf-kserve-854cf4cdf8-cddw6"
Oct 30 00:47:26.955404 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:26.954798    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca5363/distinct-2-w4-ca5363-kserve-576576f9b5-xkkqc" podUID="5de8bebf-cfad-49da-9a98-08aedde4442c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:47:26.961422 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:26.960584    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ca5363/distinct-2-w4-ca5363-kserve-576576f9b5-xkkqc"
Oct 30 00:47:27.160651 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:27.151682    5002 patch_prober.go:29] interesting pod/e2e-worker-w6-021d6d-openshift-default-65c58b6947-x8wgq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body=
Oct 30 00:47:27.160651 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:27.151745    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-x8wgq" podUID="c534def0-b8f3-4be1-8f47-72bffc1234e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused"
Oct 30 00:47:27.190551 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:27.190517    5002 patch_prober.go:29] interesting pod/e2e-worker-w2-1a11cf-openshift-default-748ddd859-7rsqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body=
Oct 30 00:47:27.190815 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:27.190784    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-7rsqp" podUID="79f13664-8af6-4e86-bf3f-8f2dcd645788" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused"
Oct 30 00:47:27.239553 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:27.239447    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-29be73/simulator-w1-29be73-kserve-9958d76db-fks2t" podUID="fbb3c806-8967-4d39-84c6-9dc022fc5cae" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/ready\": context deadline exceeded"
Oct 30 00:47:27.240098 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:27.240075    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-29be73/simulator-w1-29be73-kserve-9958d76db-fks2t"
Oct 30 00:47:27.241344 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:27.241315    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-52650f/distinct-w0-52650f-kserve-d85899b5-vlhps" podUID="ea1850d9-abba-4977-a540-f6e0eaefb17d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:47:27.241796 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:27.241776    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-52650f/distinct-w0-52650f-kserve-d85899b5-vlhps"
Oct 30 00:47:27.243954 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:27.243929    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-52650f/premium-w0-52650f-kserve-c5bf66f9d-qq5dm" podUID="3455d983-b11b-4cee-bfce-d1f399cf9679" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:47:27.244344 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:27.244325    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-52650f/premium-w0-52650f-kserve-c5bf66f9d-qq5dm"
Oct 30 00:47:27.250492 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:27.250456    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1a11cf/distinct-2-w2-1a11cf-kserve-758bb6dd57-j5f2k" podUID="f2d66dea-9f0b-43f3-a774-bb065e88f378" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:47:27.250962 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:27.250928    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-1a11cf/distinct-2-w2-1a11cf-kserve-758bb6dd57-j5f2k"
Oct 30 00:47:27.271721 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:27.271686    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7d8z4"]
Oct 30 00:47:27.306403 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:27.302019    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ca5363/distinct-2-w4-ca5363-kserve-576576f9b5-xkkqc"
Oct 30 00:47:27.396406 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:27.349457    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-29be73/premium-w1-29be73-kserve-7b777cfc8c-7kgp5" podUID="e6fabbdd-86ca-4cbf-85be-ef250575a355" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": context deadline exceeded"
Oct 30 00:47:27.396406 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:27.349837    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-29be73/premium-w1-29be73-kserve-7b777cfc8c-7kgp5"
Oct 30 00:47:27.577004 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:27.575156    5002 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-cefb27/simulator-w3-cefb27-kserve-6b669c9d9f-zx2vm" podUID="7121e81e-f903-42c0-a7a4-bc18d49194e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:47:27.668406 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:27.668106    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cefb27/simulator-w3-cefb27-kserve-6b669c9d9f-zx2vm" podUID="7121e81e-f903-42c0-a7a4-bc18d49194e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/ready\": context deadline exceeded"
Oct 30 00:47:27.694412 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:27.668757    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-cefb27/simulator-w3-cefb27-kserve-6b669c9d9f-zx2vm"
Oct 30 00:47:27.744194 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:27.744160    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-29be73/premium-w1-29be73-kserve-7b777cfc8c-7kgp5"
Oct 30 00:47:27.803538 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:27.803180    5002 patch_prober.go:29] interesting pod/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-xfr4r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body=
Oct 30 00:47:27.803538 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:27.803233    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-xfr4r" podUID="735d76ff-0add-4acc-b5e9-4ebeb27c0e9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused"
Oct 30 00:47:27.915462 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:27.900458    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-021d6d/simulator-w6-021d6d-kserve-6cc97fdf86-dqtpw" podUID="50586f76-4f3b-4e7a-83ac-6c573faee415" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:47:27.924231 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:27.924200    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-cefb27/simulator-w3-cefb27-kserve-6b669c9d9f-zx2vm"
Oct 30 00:47:27.946760 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:27.894368    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1a11cf/distinct-w2-1a11cf-kserve-854cf4cdf8-cddw6" podUID="32a03478-46eb-4363-97b6-51a32c4099c7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:47:28.035313 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:28.034924    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-1a11cf/distinct-2-w2-1a11cf-kserve-758bb6dd57-j5f2k"
Oct 30 00:47:28.087402 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:28.082642    5002 patch_prober.go:29] interesting pod/e2e-worker-w6-021d6d-openshift-default-65c58b6947-x8wgq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body=
Oct 30 00:47:28.087402 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:28.082700    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-x8wgq" podUID="c534def0-b8f3-4be1-8f47-72bffc1234e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused"
Oct 30 00:47:28.190466 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:28.186151    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-1a11cf/distinct-w2-1a11cf-kserve-854cf4cdf8-cddw6"
Oct 30 00:47:28.206122 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:28.200588    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-52650f/distinct-w0-52650f-kserve-d85899b5-vlhps"
Oct 30 00:47:28.206122 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:28.202782    5002 patch_prober.go:29] interesting pod/e2e-worker-w2-1a11cf-openshift-default-748ddd859-7rsqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body=
Oct 30 00:47:28.206122 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:28.202828    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-7rsqp" podUID="79f13664-8af6-4e86-bf3f-8f2dcd645788" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused"
Oct 30 00:47:28.227567 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:28.216923    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-29be73/simulator-w1-29be73-kserve-9958d76db-fks2t"
Oct 30 00:47:28.245920 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:28.245522    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-52650f/premium-w0-52650f-kserve-c5bf66f9d-qq5dm" podUID="3455d983-b11b-4cee-bfce-d1f399cf9679" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": context deadline exceeded"
Oct 30 00:47:28.330213 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:28.330175    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-52650f/premium-w0-52650f-kserve-c5bf66f9d-qq5dm"
Oct 30 00:47:28.802887 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:28.802844    5002 patch_prober.go:29] interesting pod/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-xfr4r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body=
Oct 30 00:47:28.803111 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:28.802923    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-xfr4r" podUID="735d76ff-0add-4acc-b5e9-4ebeb27c0e9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused"
Oct 30 00:47:29.088460 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:29.087677    5002 patch_prober.go:29] interesting pod/e2e-worker-w6-021d6d-openshift-default-65c58b6947-x8wgq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body=
Oct 30 00:47:29.088460 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:29.087741    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-x8wgq" podUID="c534def0-b8f3-4be1-8f47-72bffc1234e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused"
Oct 30 00:47:29.195404 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:29.194767    5002 patch_prober.go:29] interesting pod/e2e-worker-w2-1a11cf-openshift-default-748ddd859-7rsqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body=
Oct 30 00:47:29.195404 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:29.194835    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-7rsqp" podUID="79f13664-8af6-4e86-bf3f-8f2dcd645788" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused"
Oct 30 00:47:30.190405 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:30.190165    5002 patch_prober.go:29] interesting pod/e2e-worker-w2-1a11cf-openshift-default-748ddd859-7rsqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body=
Oct 30 00:47:30.190405 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:30.190231    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-7rsqp" podUID="79f13664-8af6-4e86-bf3f-8f2dcd645788" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused"
Oct 30 00:47:30.252813 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:30.251969    5002 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-ca5363/distinct-2-w4-ca5363-kserve-576576f9b5-xkkqc" podUID="5de8bebf-cfad-49da-9a98-08aedde4442c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:47:30.301405 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:30.300504    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca5363/distinct-2-w4-ca5363-kserve-576576f9b5-xkkqc" podUID="5de8bebf-cfad-49da-9a98-08aedde4442c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:47:30.830525 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:30.830475    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-52650f/simulator-w0-52650f-kserve-58b64766c-94qxx" podUID="659c1425-5b67-4d01-bfc2-f1ba56b9a59b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:47:30.861407 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:30.856626    5002 patch_prober.go:29] interesting pod/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-xfr4r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 00:47:30.861407 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:30.856687    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-xfr4r" podUID="735d76ff-0add-4acc-b5e9-4ebeb27c0e9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:47:30.933406 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:30.930545    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cefb27/simulator-w3-cefb27-kserve-6b669c9d9f-zx2vm" podUID="7121e81e-f903-42c0-a7a4-bc18d49194e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:47:31.156154 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:31.155524    5002 patch_prober.go:29] interesting pod/e2e-worker-w6-021d6d-openshift-default-65c58b6947-x8wgq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 00:47:31.156154 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:31.155591    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-x8wgq" podUID="c534def0-b8f3-4be1-8f47-72bffc1234e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:47:31.156559 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:31.156532    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6f4fba/simulator-w5-6f4fba-kserve-5bf8dfddd5-wcwzh" podUID="3a2c2f8f-649c-4ab6-af98-5d87df0e511f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:47:31.211523 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:31.211462    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-52650f/distinct-w0-52650f-kserve-d85899b5-vlhps" podUID="ea1850d9-abba-4977-a540-f6e0eaefb17d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:47:31.247508 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:31.247454    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-52650f/premium-w0-52650f-kserve-c5bf66f9d-qq5dm" podUID="3455d983-b11b-4cee-bfce-d1f399cf9679" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": context deadline exceeded"
Oct 30 00:47:31.303777 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:31.303732    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca5363/distinct-2-w4-ca5363-kserve-576576f9b5-xkkqc" podUID="5de8bebf-cfad-49da-9a98-08aedde4442c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:47:31.304792 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:31.304766    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ca5363/distinct-2-w4-ca5363-kserve-576576f9b5-xkkqc"
Oct 30 00:47:31.571590 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:31.571527    5002 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-021d6d/simulator-w6-021d6d-kserve-6cc97fdf86-dqtpw" podUID="50586f76-4f3b-4e7a-83ac-6c573faee415" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:47:31.579225 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:31.579056    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-x8wgq"
Oct 30 00:47:31.580888 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:31.580758    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-x8wgq"
Oct 30 00:47:31.582108 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:31.582089    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-x8wgq"
Oct 30 00:47:31.759075 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:31.758098    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-29be73/premium-w1-29be73-kserve-7b777cfc8c-7kgp5" podUID="e6fabbdd-86ca-4cbf-85be-ef250575a355" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": context deadline exceeded"
Oct 30 00:47:31.841441 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:31.841334    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-52650f/simulator-w0-52650f-kserve-58b64766c-94qxx" podUID="659c1425-5b67-4d01-bfc2-f1ba56b9a59b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": context deadline exceeded"
Oct 30 00:47:31.842309 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:31.842287    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-52650f/simulator-w0-52650f-kserve-58b64766c-94qxx"
Oct 30 00:47:31.877448 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:31.877410    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-021d6d/simulator-w6-021d6d-kserve-6cc97fdf86-dqtpw" podUID="50586f76-4f3b-4e7a-83ac-6c573faee415" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:47:31.924516 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:31.915496    5002 patch_prober.go:29] interesting pod/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-xfr4r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 00:47:31.924516 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:31.915567    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-xfr4r" podUID="735d76ff-0add-4acc-b5e9-4ebeb27c0e9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:47:31.967550 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:31.964266    5002 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-52650f/premium-w0-52650f-kserve-c5bf66f9d-qq5dm" podUID="3455d983-b11b-4cee-bfce-d1f399cf9679" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:47:32.185492 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:32.160743    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6f4fba/simulator-w5-6f4fba-kserve-5bf8dfddd5-wcwzh" podUID="3a2c2f8f-649c-4ab6-af98-5d87df0e511f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:47:32.185492 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:32.161455    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-6f4fba/simulator-w5-6f4fba-kserve-5bf8dfddd5-wcwzh"
Oct 30 00:47:32.208737 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:32.208691    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1a11cf/distinct-2-w2-1a11cf-kserve-758bb6dd57-j5f2k" podUID="f2d66dea-9f0b-43f3-a774-bb065e88f378" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:47:32.209331 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:32.209300    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1a11cf/distinct-w2-1a11cf-kserve-854cf4cdf8-cddw6" podUID="32a03478-46eb-4363-97b6-51a32c4099c7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:47:32.213491 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:32.213458    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-52650f/distinct-w0-52650f-kserve-d85899b5-vlhps" podUID="ea1850d9-abba-4977-a540-f6e0eaefb17d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:47:32.222513 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:32.222483    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-52650f/distinct-w0-52650f-kserve-d85899b5-vlhps"
Oct 30 00:47:32.298550 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:32.283522    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-52650f/premium-w0-52650f-kserve-c5bf66f9d-qq5dm" podUID="3455d983-b11b-4cee-bfce-d1f399cf9679" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:47:32.298550 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:32.283884    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-52650f/premium-w0-52650f-kserve-c5bf66f9d-qq5dm"
Oct 30 00:47:32.318548 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:32.318508    5002 patch_prober.go:29] interesting pod/e2e-worker-w2-1a11cf-openshift-default-748ddd859-7rsqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 00:47:32.318850 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:32.318807    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-7rsqp" podUID="79f13664-8af6-4e86-bf3f-8f2dcd645788" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:47:32.319609 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:32.319188    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca5363/distinct-2-w4-ca5363-kserve-576576f9b5-xkkqc" podUID="5de8bebf-cfad-49da-9a98-08aedde4442c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:47:32.537624 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:32.534838    5002 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-52650f/distinct-w0-52650f-kserve-d85899b5-vlhps" podUID="ea1850d9-abba-4977-a540-f6e0eaefb17d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/health\": context deadline exceeded"
Oct 30 00:47:32.607997 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:32.607724    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-52650f/premium-w0-52650f-kserve-c5bf66f9d-qq5dm"
Oct 30 00:47:32.635022 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:32.634989    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-52650f/simulator-w0-52650f-kserve-58b64766c-94qxx"
Oct 30 00:47:32.635631 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:32.635605    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ca5363/distinct-2-w4-ca5363-kserve-576576f9b5-xkkqc"
Oct 30 00:47:32.639901 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:32.639822    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-6f4fba/simulator-w5-6f4fba-kserve-5bf8dfddd5-wcwzh"
Oct 30 00:47:32.681416 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:32.679870    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-7rsqp"
Oct 30 00:47:32.682941 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:32.682723    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-7rsqp"
Oct 30 00:47:32.683699 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:32.683682    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-7rsqp"
Oct 30 00:47:32.711907 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:32.711864    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-52650f/distinct-w0-52650f-kserve-d85899b5-vlhps"
Oct 30 00:47:32.958740 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:32.958701    5002 patch_prober.go:29] interesting pod/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-xfr4r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 00:47:32.959178 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:32.959142    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-xfr4r" podUID="735d76ff-0add-4acc-b5e9-4ebeb27c0e9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:47:33.623930 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:33.623884    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-xfr4r"
Oct 30 00:47:33.625174 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:33.625149    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-xfr4r"
Oct 30 00:47:33.626119 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:33.626097    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-xfr4r"
Oct 30 00:47:37.546595 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:37.546555    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16f76fb3-jmbfm"]
Oct 30 00:47:37.547123 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:37.547110    5002 cpu_manager.go:414] "containerMap: removing container" podUID="bf85f00d-1e7a-458c-8cc3-bc8eebc361c0" containerName="revoke-keys"
Oct 30 00:47:37.547169 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:37.547124    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf85f00d-1e7a-458c-8cc3-bc8eebc361c0" containerName="revoke-keys"
Oct 30 00:47:37.547240 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:37.547230    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf85f00d-1e7a-458c-8cc3-bc8eebc361c0" containerName="revoke-keys"
Oct 30 00:47:37.551081 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:37.551054    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16f76fb3-jmbfm"
Oct 30 00:47:37.570928 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:37.570895    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16f76fb3-jmbfm"]
Oct 30 00:47:37.580136 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:37.580109    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 00:47:37.580303 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:37.580138    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 00:47:37.580303 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:37.580215    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 00:47:37.583091 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:37.583067    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5phsh\""
Oct 30 00:47:37.694146 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:37.694102    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1acd774-f759-4d08-8b45-7f07976478fe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-16f76fb3-jmbfm\" (UID: \"a1acd774-f759-4d08-8b45-7f07976478fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16f76fb3-jmbfm"
Oct 30 00:47:37.694314 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:37.694278    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1acd774-f759-4d08-8b45-7f07976478fe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-16f76fb3-jmbfm\" (UID: \"a1acd774-f759-4d08-8b45-7f07976478fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16f76fb3-jmbfm"
Oct 30 00:47:37.694356 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:37.694312    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmtjd\" (UniqueName: \"kubernetes.io/projected/a1acd774-f759-4d08-8b45-7f07976478fe-kube-api-access-cmtjd\") pod \"maas-api-revoke-sub-16f76fb3-jmbfm\" (UID: \"a1acd774-f759-4d08-8b45-7f07976478fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16f76fb3-jmbfm"
Oct 30 00:47:37.795715 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:37.795683    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1acd774-f759-4d08-8b45-7f07976478fe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-16f76fb3-jmbfm\" (UID: \"a1acd774-f759-4d08-8b45-7f07976478fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16f76fb3-jmbfm"
Oct 30 00:47:37.795715 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:37.795727    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmtjd\" (UniqueName: \"kubernetes.io/projected/a1acd774-f759-4d08-8b45-7f07976478fe-kube-api-access-cmtjd\") pod \"maas-api-revoke-sub-16f76fb3-jmbfm\" (UID: \"a1acd774-f759-4d08-8b45-7f07976478fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16f76fb3-jmbfm"
Oct 30 00:47:37.795956 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:37.795829    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1acd774-f759-4d08-8b45-7f07976478fe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-16f76fb3-jmbfm\" (UID: \"a1acd774-f759-4d08-8b45-7f07976478fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16f76fb3-jmbfm"
Oct 30 00:47:37.796513 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:37.796488    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1acd774-f759-4d08-8b45-7f07976478fe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-16f76fb3-jmbfm\" (UID: \"a1acd774-f759-4d08-8b45-7f07976478fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16f76fb3-jmbfm"
Oct 30 00:47:37.798180 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:37.798125    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1acd774-f759-4d08-8b45-7f07976478fe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-16f76fb3-jmbfm\" (UID: \"a1acd774-f759-4d08-8b45-7f07976478fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16f76fb3-jmbfm"
Oct 30 00:47:37.820527 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:37.820502    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmtjd\" (UniqueName: \"kubernetes.io/projected/a1acd774-f759-4d08-8b45-7f07976478fe-kube-api-access-cmtjd\") pod \"maas-api-revoke-sub-16f76fb3-jmbfm\" (UID: \"a1acd774-f759-4d08-8b45-7f07976478fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16f76fb3-jmbfm"
Oct 30 00:47:37.869759 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:37.869729    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16f76fb3-jmbfm"
Oct 30 00:47:38.191614 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:38.191582    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16f76fb3-jmbfm"]
Oct 30 00:47:38.212255 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:38.212224    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16f76fb3-jmbfm" event={"ID":"a1acd774-f759-4d08-8b45-7f07976478fe","Type":"ContainerStarted","Data":"7a1877a0d507a7ccd0cc6646482a6a991ae1bdf4bef62d9bae79760a666f7995"}
Oct 30 00:47:39.219460 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:39.219427    5002 generic.go:356] "Generic (PLEG): container finished" podID="a1acd774-f759-4d08-8b45-7f07976478fe" containerID="510087b5d66aedeaefb7fd3863474957094da09e5f8e146d718256efcc730d33" exitCode=0
Oct 30 00:47:39.219856 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:39.219514    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16f76fb3-jmbfm" event={"ID":"a1acd774-f759-4d08-8b45-7f07976478fe","Type":"ContainerDied","Data":"510087b5d66aedeaefb7fd3863474957094da09e5f8e146d718256efcc730d33"}
Oct 30 00:47:40.412341 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:40.412306    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16f76fb3-jmbfm"
Oct 30 00:47:40.529791 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:40.529713    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1acd774-f759-4d08-8b45-7f07976478fe-kube-api-access-cmtjd\" (UniqueName: \"kubernetes.io/projected/a1acd774-f759-4d08-8b45-7f07976478fe-kube-api-access-cmtjd\") pod \"a1acd774-f759-4d08-8b45-7f07976478fe\" (UID: \"a1acd774-f759-4d08-8b45-7f07976478fe\") "
Oct 30 00:47:40.529928 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:40.529905    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1acd774-f759-4d08-8b45-7f07976478fe-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1acd774-f759-4d08-8b45-7f07976478fe-maas-api-openshift-ca\") pod \"a1acd774-f759-4d08-8b45-7f07976478fe\" (UID: \"a1acd774-f759-4d08-8b45-7f07976478fe\") "
Oct 30 00:47:40.529969 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:40.529942    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a1acd774-f759-4d08-8b45-7f07976478fe-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1acd774-f759-4d08-8b45-7f07976478fe-maas-api-xks-ca\") pod \"a1acd774-f759-4d08-8b45-7f07976478fe\" (UID: \"a1acd774-f759-4d08-8b45-7f07976478fe\") "
Oct 30 00:47:40.530228 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:40.530204    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1acd774-f759-4d08-8b45-7f07976478fe-maas-api-openshift-ca" pod "a1acd774-f759-4d08-8b45-7f07976478fe" (UID: "a1acd774-f759-4d08-8b45-7f07976478fe"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:47:40.531919 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:40.531896    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1acd774-f759-4d08-8b45-7f07976478fe-kube-api-access-cmtjd" pod "a1acd774-f759-4d08-8b45-7f07976478fe" (UID: "a1acd774-f759-4d08-8b45-7f07976478fe"). InnerVolumeSpecName "kube-api-access-cmtjd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:47:40.532017 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:40.531988    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1acd774-f759-4d08-8b45-7f07976478fe-maas-api-xks-ca" pod "a1acd774-f759-4d08-8b45-7f07976478fe" (UID: "a1acd774-f759-4d08-8b45-7f07976478fe"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:47:40.631007 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:40.630976    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1acd774-f759-4d08-8b45-7f07976478fe-maas-api-openshift-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:47:40.631007 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:40.631004    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1acd774-f759-4d08-8b45-7f07976478fe-maas-api-xks-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:47:40.631007 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:40.631014    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cmtjd\" (UniqueName: \"kubernetes.io/projected/a1acd774-f759-4d08-8b45-7f07976478fe-kube-api-access-cmtjd\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:47:41.233268 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:41.233230    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16f76fb3-jmbfm" event={"ID":"a1acd774-f759-4d08-8b45-7f07976478fe","Type":"ContainerDied","Data":"7a1877a0d507a7ccd0cc6646482a6a991ae1bdf4bef62d9bae79760a666f7995"}
Oct 30 00:47:41.233268 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:41.233263    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16f76fb3-jmbfm"
Oct 30 00:47:41.233516 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:41.233291    5002 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7a1877a0d507a7ccd0cc6646482a6a991ae1bdf4bef62d9bae79760a666f7995"
Oct 30 00:47:43.135472 ip-10-0-128-7 kubenswrapper[5002]: E1030 00:47:43.123335    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbjdq" containerName="extractor"
Oct 30 00:47:43.620414 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:43.619466    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-52650f/premium-w0-52650f-kserve-c5bf66f9d-qq5dm" podUID="3455d983-b11b-4cee-bfce-d1f399cf9679" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:47:43.647407 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:43.638451    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-52650f/simulator-w0-52650f-kserve-58b64766c-94qxx" podUID="659c1425-5b67-4d01-bfc2-f1ba56b9a59b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:47:43.665434 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:43.656807    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6f4fba/simulator-w5-6f4fba-kserve-5bf8dfddd5-wcwzh" podUID="3a2c2f8f-649c-4ab6-af98-5d87df0e511f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/ready\": context deadline exceeded"
Oct 30 00:47:43.753406 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:43.752462    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-29be73/premium-w1-29be73-kserve-7b777cfc8c-7kgp5" podUID="e6fabbdd-86ca-4cbf-85be-ef250575a355" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:47:44.039441 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:44.038496    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1a11cf/distinct-2-w2-1a11cf-kserve-758bb6dd57-j5f2k" podUID="f2d66dea-9f0b-43f3-a774-bb065e88f378" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:47:44.063572 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:44.063110    5002 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-29be73/simulator-w1-29be73-kserve-9958d76db-fks2t" podUID="fbb3c806-8967-4d39-84c6-9dc022fc5cae" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:47:44.169552 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:44.169499    5002 patch_prober.go:29] interesting pod/e2e-worker-w6-021d6d-openshift-default-65c58b6947-vqjxg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 00:47:44.170413 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:44.170102    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-vqjxg" podUID="00e1797d-0629-4b79-8073-eaa093f48f2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:47:44.222055 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:44.221490    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1a11cf/distinct-w2-1a11cf-kserve-854cf4cdf8-cddw6" podUID="32a03478-46eb-4363-97b6-51a32c4099c7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:47:44.235455 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:44.223449    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-29be73/simulator-w1-29be73-kserve-9958d76db-fks2t" podUID="fbb3c806-8967-4d39-84c6-9dc022fc5cae" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/ready\": context deadline exceeded"
Oct 30 00:47:44.864405 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:44.863776    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-021d6d/simulator-w6-021d6d-kserve-6cc97fdf86-dqtpw" podUID="50586f76-4f3b-4e7a-83ac-6c573faee415" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:47:44.941405 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:44.930533    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cefb27/simulator-w3-cefb27-kserve-6b669c9d9f-zx2vm" podUID="7121e81e-f903-42c0-a7a4-bc18d49194e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:47:44.997405 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:44.996804    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6f4fba/simulator-w5-6f4fba-kserve-5bf8dfddd5-wcwzh" podUID="3a2c2f8f-649c-4ab6-af98-5d87df0e511f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:47:45.120519 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:45.017053    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-29be73/premium-w1-29be73-kserve-7b777cfc8c-7kgp5" podUID="e6fabbdd-86ca-4cbf-85be-ef250575a355" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:47:45.120519 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:45.047209    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1a11cf/distinct-2-w2-1a11cf-kserve-758bb6dd57-j5f2k" podUID="f2d66dea-9f0b-43f3-a774-bb065e88f378" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:47:45.266471 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:45.266202    5002 patch_prober.go:29] interesting pod/e2e-worker-w2-1a11cf-openshift-default-748ddd859-s5rzw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 00:47:45.266471 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:45.266268    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-s5rzw" podUID="cc529c8b-d888-412d-b861-f3648aaa473d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:47:45.325196 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:45.325162    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-6f4fba/simulator-w5-6f4fba-kserve-5bf8dfddd5-wcwzh"
Oct 30 00:47:45.325459 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:45.325446    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-29be73/premium-w1-29be73-kserve-7b777cfc8c-7kgp5"
Oct 30 00:47:45.325600 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:45.325590    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-1a11cf/distinct-2-w2-1a11cf-kserve-758bb6dd57-j5f2k"
Oct 30 00:47:45.366598 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:45.366556    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca5363/distinct-2-w4-ca5363-kserve-576576f9b5-xkkqc" podUID="5de8bebf-cfad-49da-9a98-08aedde4442c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Oct 30 00:47:45.393477 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:45.367923    5002 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-52650f/simulator-w0-52650f-kserve-58b64766c-94qxx" podUID="659c1425-5b67-4d01-bfc2-f1ba56b9a59b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/health\": context deadline exceeded"
Oct 30 00:47:45.449896 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:45.449870    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-29be73/premium-w1-29be73-kserve-7b777cfc8c-7kgp5"
Oct 30 00:47:45.451498 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:45.451459    5002 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-1a11cf/distinct-w2-1a11cf-kserve-854cf4cdf8-cddw6" podUID="32a03478-46eb-4363-97b6-51a32c4099c7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:47:45.490399 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:45.490350    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-6f4fba/simulator-w5-6f4fba-kserve-5bf8dfddd5-wcwzh"
Oct 30 00:47:45.534837 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:45.531814    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-1a11cf/distinct-2-w2-1a11cf-kserve-758bb6dd57-j5f2k"
Oct 30 00:47:45.648729 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:45.648641    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-52650f/simulator-w0-52650f-kserve-58b64766c-94qxx" podUID="659c1425-5b67-4d01-bfc2-f1ba56b9a59b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:47:45.765017 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:45.764976    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-52650f/distinct-w0-52650f-kserve-d85899b5-vlhps" podUID="ea1850d9-abba-4977-a540-f6e0eaefb17d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:47:51.026739 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:51.026692    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-24zch"]
Oct 30 00:47:51.027264 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:51.027249    5002 cpu_manager.go:414] "containerMap: removing container" podUID="a1acd774-f759-4d08-8b45-7f07976478fe" containerName="revoke-keys"
Oct 30 00:47:51.027264 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:51.027264    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1acd774-f759-4d08-8b45-7f07976478fe" containerName="revoke-keys"
Oct 30 00:47:51.027422 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:51.027348    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1acd774-f759-4d08-8b45-7f07976478fe" containerName="revoke-keys"
Oct 30 00:47:51.032239 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:51.032213    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-24zch"
Oct 30 00:47:51.059745 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:51.059706    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-24zch"]
Oct 30 00:47:51.213355 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:51.213311    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35e8d90c-3415-41a3-8b4f-03a3c738cc99-credential-socket\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-24zch\" (UID: \"35e8d90c-3415-41a3-8b4f-03a3c738cc99\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-24zch"
Oct 30 00:47:51.213548 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:51.213442    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35e8d90c-3415-41a3-8b4f-03a3c738cc99-istio-envoy\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-24zch\" (UID: \"35e8d90c-3415-41a3-8b4f-03a3c738cc99\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-24zch"
Oct 30 00:47:51.213548 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:51.213485    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35e8d90c-3415-41a3-8b4f-03a3c738cc99-istio-data\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-24zch\" (UID: \"35e8d90c-3415-41a3-8b4f-03a3c738cc99\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-24zch"
Oct 30 00:47:51.213548 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:51.213503    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35e8d90c-3415-41a3-8b4f-03a3c738cc99-istio-token\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-24zch\" (UID: \"35e8d90c-3415-41a3-8b4f-03a3c738cc99\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-24zch"
Oct 30 00:47:51.213548 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:51.213526    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xz6rj\" (UniqueName: \"kubernetes.io/projected/35e8d90c-3415-41a3-8b4f-03a3c738cc99-kube-api-access-xz6rj\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-24zch\" (UID: \"35e8d90c-3415-41a3-8b4f-03a3c738cc99\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-24zch"
Oct 30 00:47:51.213687 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:51.213657    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35e8d90c-3415-41a3-8b4f-03a3c738cc99-istiod-ca-cert\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-24zch\" (UID: \"35e8d90c-3415-41a3-8b4f-03a3c738cc99\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-24zch"
Oct 30 00:47:51.213721 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:51.213699    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35e8d90c-3415-41a3-8b4f-03a3c738cc99-workload-socket\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-24zch\" (UID: \"35e8d90c-3415-41a3-8b4f-03a3c738cc99\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-24zch"
Oct 30 00:47:51.213754 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:51.213743    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35e8d90c-3415-41a3-8b4f-03a3c738cc99-istio-podinfo\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-24zch\" (UID: \"35e8d90c-3415-41a3-8b4f-03a3c738cc99\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-24zch"
Oct 30 00:47:51.213811 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:51.213797    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35e8d90c-3415-41a3-8b4f-03a3c738cc99-workload-certs\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-24zch\" (UID: \"35e8d90c-3415-41a3-8b4f-03a3c738cc99\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-24zch"
Oct 30 00:47:51.315639 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:51.315563    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35e8d90c-3415-41a3-8b4f-03a3c738cc99-istio-data\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-24zch\" (UID: \"35e8d90c-3415-41a3-8b4f-03a3c738cc99\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-24zch"
Oct 30 00:47:51.315639 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:51.315601    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35e8d90c-3415-41a3-8b4f-03a3c738cc99-istio-token\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-24zch\" (UID: \"35e8d90c-3415-41a3-8b4f-03a3c738cc99\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-24zch"
Oct 30 00:47:51.315639 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:51.315626    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xz6rj\" (UniqueName: \"kubernetes.io/projected/35e8d90c-3415-41a3-8b4f-03a3c738cc99-kube-api-access-xz6rj\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-24zch\" (UID: \"35e8d90c-3415-41a3-8b4f-03a3c738cc99\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-24zch"
Oct 30 00:47:51.315936 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:51.315660    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35e8d90c-3415-41a3-8b4f-03a3c738cc99-istiod-ca-cert\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-24zch\" (UID: \"35e8d90c-3415-41a3-8b4f-03a3c738cc99\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-24zch"
Oct 30 00:47:51.315936 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:51.315698    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35e8d90c-3415-41a3-8b4f-03a3c738cc99-workload-socket\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-24zch\" (UID: \"35e8d90c-3415-41a3-8b4f-03a3c738cc99\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-24zch"
Oct 30 00:47:51.315936 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:51.315748    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35e8d90c-3415-41a3-8b4f-03a3c738cc99-istio-podinfo\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-24zch\" (UID: \"35e8d90c-3415-41a3-8b4f-03a3c738cc99\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-24zch"
Oct 30 00:47:51.315936 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:51.315786    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35e8d90c-3415-41a3-8b4f-03a3c738cc99-workload-certs\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-24zch\" (UID: \"35e8d90c-3415-41a3-8b4f-03a3c738cc99\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-24zch"
Oct 30 00:47:51.315936 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:51.315895    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35e8d90c-3415-41a3-8b4f-03a3c738cc99-credential-socket\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-24zch\" (UID: \"35e8d90c-3415-41a3-8b4f-03a3c738cc99\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-24zch"
Oct 30 00:47:51.316195 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:51.315973    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35e8d90c-3415-41a3-8b4f-03a3c738cc99-istio-envoy\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-24zch\" (UID: \"35e8d90c-3415-41a3-8b4f-03a3c738cc99\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-24zch"
Oct 30 00:47:51.316195 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:51.316000    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35e8d90c-3415-41a3-8b4f-03a3c738cc99-istio-data\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-24zch\" (UID: \"35e8d90c-3415-41a3-8b4f-03a3c738cc99\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-24zch"
Oct 30 00:47:51.316195 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:51.316065    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35e8d90c-3415-41a3-8b4f-03a3c738cc99-workload-socket\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-24zch\" (UID: \"35e8d90c-3415-41a3-8b4f-03a3c738cc99\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-24zch"
Oct 30 00:47:51.316524 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:51.316499    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35e8d90c-3415-41a3-8b4f-03a3c738cc99-istiod-ca-cert\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-24zch\" (UID: \"35e8d90c-3415-41a3-8b4f-03a3c738cc99\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-24zch"
Oct 30 00:47:51.316922 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:51.316903    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35e8d90c-3415-41a3-8b4f-03a3c738cc99-workload-certs\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-24zch\" (UID: \"35e8d90c-3415-41a3-8b4f-03a3c738cc99\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-24zch"
Oct 30 00:47:51.317044 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:51.317022    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35e8d90c-3415-41a3-8b4f-03a3c738cc99-credential-socket\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-24zch\" (UID: \"35e8d90c-3415-41a3-8b4f-03a3c738cc99\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-24zch"
Oct 30 00:47:51.318644 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:51.318623    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35e8d90c-3415-41a3-8b4f-03a3c738cc99-istio-podinfo\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-24zch\" (UID: \"35e8d90c-3415-41a3-8b4f-03a3c738cc99\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-24zch"
Oct 30 00:47:51.319172 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:51.319148    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35e8d90c-3415-41a3-8b4f-03a3c738cc99-istio-envoy\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-24zch\" (UID: \"35e8d90c-3415-41a3-8b4f-03a3c738cc99\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-24zch"
Oct 30 00:47:51.340745 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:51.340721    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35e8d90c-3415-41a3-8b4f-03a3c738cc99-istio-token\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-24zch\" (UID: \"35e8d90c-3415-41a3-8b4f-03a3c738cc99\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-24zch"
Oct 30 00:47:51.340875 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:51.340810    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xz6rj\" (UniqueName: \"kubernetes.io/projected/35e8d90c-3415-41a3-8b4f-03a3c738cc99-kube-api-access-xz6rj\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-24zch\" (UID: \"35e8d90c-3415-41a3-8b4f-03a3c738cc99\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-24zch"
Oct 30 00:47:51.353705 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:51.353679    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-24zch"
Oct 30 00:47:51.671108 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:47:51.671075    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35e8d90c_3415_41a3_8b4f_03a3c738cc99.slice/crio-f62c7b40e212388685ad3b849283d69d825c1c47e2d33166df0b7e05d4f043b1 WatchSource:0}: Error finding container f62c7b40e212388685ad3b849283d69d825c1c47e2d33166df0b7e05d4f043b1: Status 404 returned error can't find the container with id f62c7b40e212388685ad3b849283d69d825c1c47e2d33166df0b7e05d4f043b1
Oct 30 00:47:51.673168 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:51.673143    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-24zch"]
Oct 30 00:47:51.673802 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:51.673771    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 00:47:51.673888 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:51.673841    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 00:47:51.673888 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:51.673871    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 00:47:52.588628 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:52.588589    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b586d6ef-6mxlr"]
Oct 30 00:47:52.593601 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:52.593573    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b586d6ef-6mxlr"
Oct 30 00:47:52.608591 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:52.608561    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b586d6ef-6mxlr"]
Oct 30 00:47:52.620991 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:52.619262    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 00:47:52.620991 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:52.619307    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 00:47:52.620991 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:52.619345    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 00:47:52.620991 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:52.619460    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5phsh\""
Oct 30 00:47:52.696334 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:52.696294    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-24zch" event={"ID":"35e8d90c-3415-41a3-8b4f-03a3c738cc99","Type":"ContainerStarted","Data":"6589a90942f3e31764a49a2d424e3f285e5224c7ba7c9a4a33e1f71345b0622e"}
Oct 30 00:47:52.696564 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:52.696356    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-24zch" event={"ID":"35e8d90c-3415-41a3-8b4f-03a3c738cc99","Type":"ContainerStarted","Data":"f62c7b40e212388685ad3b849283d69d825c1c47e2d33166df0b7e05d4f043b1"}
Oct 30 00:47:52.736295 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:52.736261    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cb5fca0-dafe-4619-8341-1ad59e427c64-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b586d6ef-6mxlr\" (UID: \"8cb5fca0-dafe-4619-8341-1ad59e427c64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b586d6ef-6mxlr"
Oct 30 00:47:52.736450 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:52.736362    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thv7s\" (UniqueName: \"kubernetes.io/projected/8cb5fca0-dafe-4619-8341-1ad59e427c64-kube-api-access-thv7s\") pod \"maas-api-revoke-sub-b586d6ef-6mxlr\" (UID: \"8cb5fca0-dafe-4619-8341-1ad59e427c64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b586d6ef-6mxlr"
Oct 30 00:47:52.736580 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:52.736555    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cb5fca0-dafe-4619-8341-1ad59e427c64-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b586d6ef-6mxlr\" (UID: \"8cb5fca0-dafe-4619-8341-1ad59e427c64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b586d6ef-6mxlr"
Oct 30 00:47:52.744797 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:52.744749    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-24zch" podStartSLOduration=2.7447362589999997 podStartE2EDuration="2.744736259s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:47:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:47:52.743149408 +0000 UTC m=+968.359582286" watchObservedRunningTime="2026-10-30 00:47:52.744736259 +0000 UTC m=+968.361169137"
Oct 30 00:47:52.837678 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:52.837637    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cb5fca0-dafe-4619-8341-1ad59e427c64-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b586d6ef-6mxlr\" (UID: \"8cb5fca0-dafe-4619-8341-1ad59e427c64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b586d6ef-6mxlr"
Oct 30 00:47:52.837861 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:52.837769    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cb5fca0-dafe-4619-8341-1ad59e427c64-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b586d6ef-6mxlr\" (UID: \"8cb5fca0-dafe-4619-8341-1ad59e427c64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b586d6ef-6mxlr"
Oct 30 00:47:52.837937 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:52.837880    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thv7s\" (UniqueName: \"kubernetes.io/projected/8cb5fca0-dafe-4619-8341-1ad59e427c64-kube-api-access-thv7s\") pod \"maas-api-revoke-sub-b586d6ef-6mxlr\" (UID: \"8cb5fca0-dafe-4619-8341-1ad59e427c64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b586d6ef-6mxlr"
Oct 30 00:47:52.838428 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:52.838403    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cb5fca0-dafe-4619-8341-1ad59e427c64-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b586d6ef-6mxlr\" (UID: \"8cb5fca0-dafe-4619-8341-1ad59e427c64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b586d6ef-6mxlr"
Oct 30 00:47:52.840089 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:52.840037    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cb5fca0-dafe-4619-8341-1ad59e427c64-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b586d6ef-6mxlr\" (UID: \"8cb5fca0-dafe-4619-8341-1ad59e427c64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b586d6ef-6mxlr"
Oct 30 00:47:52.887268 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:52.887197    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-thv7s\" (UniqueName: \"kubernetes.io/projected/8cb5fca0-dafe-4619-8341-1ad59e427c64-kube-api-access-thv7s\") pod \"maas-api-revoke-sub-b586d6ef-6mxlr\" (UID: \"8cb5fca0-dafe-4619-8341-1ad59e427c64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b586d6ef-6mxlr"
Oct 30 00:47:52.911579 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:52.911545    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b586d6ef-6mxlr"
Oct 30 00:47:53.240705 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:47:53.240674    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cb5fca0_dafe_4619_8341_1ad59e427c64.slice/crio-015db8c201aa82053c5d850ff0f8e0a4761967f7f747e86b89de494113c3daf9 WatchSource:0}: Error finding container 015db8c201aa82053c5d850ff0f8e0a4761967f7f747e86b89de494113c3daf9: Status 404 returned error can't find the container with id 015db8c201aa82053c5d850ff0f8e0a4761967f7f747e86b89de494113c3daf9
Oct 30 00:47:53.241669 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:53.241639    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b586d6ef-6mxlr"]
Oct 30 00:47:53.367404 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:53.361732    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-24zch"
Oct 30 00:47:53.367404 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:53.366844    5002 patch_prober.go:29] interesting pod/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-24zch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Oct 30 00:47:53.367404 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:53.366908    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-24zch" podUID="35e8d90c-3415-41a3-8b4f-03a3c738cc99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Oct 30 00:47:53.705108 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:53.705067    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b586d6ef-6mxlr" event={"ID":"8cb5fca0-dafe-4619-8341-1ad59e427c64","Type":"ContainerStarted","Data":"75c04f95617817d91a0ca70e23eac377d3467d5d890ea544d89e63b4e05807a6"}
Oct 30 00:47:53.705510 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:53.705131    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b586d6ef-6mxlr" event={"ID":"8cb5fca0-dafe-4619-8341-1ad59e427c64","Type":"ContainerStarted","Data":"015db8c201aa82053c5d850ff0f8e0a4761967f7f747e86b89de494113c3daf9"}
Oct 30 00:47:53.782576 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:53.773122    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b586d6ef-6mxlr" podStartSLOduration=1.7730980509999998 podStartE2EDuration="1.773098051s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:47:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:47:53.74769906 +0000 UTC m=+969.364131950" watchObservedRunningTime="2026-10-30 00:47:53.773098051 +0000 UTC m=+969.389530934"
Oct 30 00:47:54.366328 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:54.361779    5002 patch_prober.go:29] interesting pod/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-24zch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Oct 30 00:47:54.366328 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:54.361873    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-24zch" podUID="35e8d90c-3415-41a3-8b4f-03a3c738cc99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Oct 30 00:47:54.725094 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:54.719421    5002 generic.go:356] "Generic (PLEG): container finished" podID="8cb5fca0-dafe-4619-8341-1ad59e427c64" containerID="75c04f95617817d91a0ca70e23eac377d3467d5d890ea544d89e63b4e05807a6" exitCode=0
Oct 30 00:47:54.725094 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:54.720664    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b586d6ef-6mxlr" event={"ID":"8cb5fca0-dafe-4619-8341-1ad59e427c64","Type":"ContainerDied","Data":"75c04f95617817d91a0ca70e23eac377d3467d5d890ea544d89e63b4e05807a6"}
Oct 30 00:47:55.375429 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:55.374795    5002 patch_prober.go:29] interesting pod/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-24zch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Oct 30 00:47:55.375429 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:55.374894    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-24zch" podUID="35e8d90c-3415-41a3-8b4f-03a3c738cc99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Oct 30 00:47:55.961801 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:55.961766    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b586d6ef-6mxlr"
Oct 30 00:47:56.007801 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:56.007724    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8cb5fca0-dafe-4619-8341-1ad59e427c64-kube-api-access-thv7s\" (UniqueName: \"kubernetes.io/projected/8cb5fca0-dafe-4619-8341-1ad59e427c64-kube-api-access-thv7s\") pod \"8cb5fca0-dafe-4619-8341-1ad59e427c64\" (UID: \"8cb5fca0-dafe-4619-8341-1ad59e427c64\") "
Oct 30 00:47:56.008098 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:56.008085    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8cb5fca0-dafe-4619-8341-1ad59e427c64-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cb5fca0-dafe-4619-8341-1ad59e427c64-maas-api-openshift-ca\") pod \"8cb5fca0-dafe-4619-8341-1ad59e427c64\" (UID: \"8cb5fca0-dafe-4619-8341-1ad59e427c64\") "
Oct 30 00:47:56.008236 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:56.008221    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8cb5fca0-dafe-4619-8341-1ad59e427c64-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cb5fca0-dafe-4619-8341-1ad59e427c64-maas-api-xks-ca\") pod \"8cb5fca0-dafe-4619-8341-1ad59e427c64\" (UID: \"8cb5fca0-dafe-4619-8341-1ad59e427c64\") "
Oct 30 00:47:56.008411 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:56.008359    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8cb5fca0-dafe-4619-8341-1ad59e427c64-maas-api-openshift-ca" pod "8cb5fca0-dafe-4619-8341-1ad59e427c64" (UID: "8cb5fca0-dafe-4619-8341-1ad59e427c64"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:47:56.008903 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:56.008886    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cb5fca0-dafe-4619-8341-1ad59e427c64-maas-api-openshift-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:47:56.009922 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:56.009898    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cb5fca0-dafe-4619-8341-1ad59e427c64-kube-api-access-thv7s" pod "8cb5fca0-dafe-4619-8341-1ad59e427c64" (UID: "8cb5fca0-dafe-4619-8341-1ad59e427c64"). InnerVolumeSpecName "kube-api-access-thv7s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:47:56.012962 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:56.011729    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8cb5fca0-dafe-4619-8341-1ad59e427c64-maas-api-xks-ca" pod "8cb5fca0-dafe-4619-8341-1ad59e427c64" (UID: "8cb5fca0-dafe-4619-8341-1ad59e427c64"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:47:56.110050 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:56.110018    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-thv7s\" (UniqueName: \"kubernetes.io/projected/8cb5fca0-dafe-4619-8341-1ad59e427c64-kube-api-access-thv7s\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:47:56.110050 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:56.110048    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cb5fca0-dafe-4619-8341-1ad59e427c64-maas-api-xks-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:47:56.358752 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:56.355689    5002 patch_prober.go:29] interesting pod/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-24zch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Oct 30 00:47:56.358752 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:56.355765    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-24zch" podUID="35e8d90c-3415-41a3-8b4f-03a3c738cc99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Oct 30 00:47:56.736935 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:56.736908    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b586d6ef-6mxlr"
Oct 30 00:47:56.737155 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:56.736956    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b586d6ef-6mxlr" event={"ID":"8cb5fca0-dafe-4619-8341-1ad59e427c64","Type":"ContainerDied","Data":"015db8c201aa82053c5d850ff0f8e0a4761967f7f747e86b89de494113c3daf9"}
Oct 30 00:47:56.737155 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:56.737017    5002 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="015db8c201aa82053c5d850ff0f8e0a4761967f7f747e86b89de494113c3daf9"
Oct 30 00:47:57.354779 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:57.354740    5002 patch_prober.go:29] interesting pod/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-24zch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Oct 30 00:47:57.355344 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:57.355314    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-24zch" podUID="35e8d90c-3415-41a3-8b4f-03a3c738cc99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Oct 30 00:47:57.569404 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:57.569236    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7d8z4"]
Oct 30 00:47:58.354448 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:58.354376    5002 patch_prober.go:29] interesting pod/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-24zch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Oct 30 00:47:58.354661 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:47:58.354471    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-24zch" podUID="35e8d90c-3415-41a3-8b4f-03a3c738cc99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Oct 30 00:48:00.173404 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:00.166618    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1a11cf/distinct-w2-1a11cf-kserve-854cf4cdf8-cddw6" podUID="32a03478-46eb-4363-97b6-51a32c4099c7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:48:00.227307 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:00.227260    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-29be73/simulator-w1-29be73-kserve-9958d76db-fks2t" podUID="fbb3c806-8967-4d39-84c6-9dc022fc5cae" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:48:00.327676 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:00.326566    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca5363/distinct-2-w4-ca5363-kserve-576576f9b5-xkkqc" podUID="5de8bebf-cfad-49da-9a98-08aedde4442c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Oct 30 00:48:00.397112 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:00.396514    5002 patch_prober.go:29] interesting pod/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-24zch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 00:48:00.397112 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:00.396591    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-24zch" podUID="35e8d90c-3415-41a3-8b4f-03a3c738cc99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:48:00.457404 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:00.453930    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-29be73/premium-w1-29be73-kserve-7b777cfc8c-7kgp5" podUID="e6fabbdd-86ca-4cbf-85be-ef250575a355" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": context deadline exceeded"
Oct 30 00:48:00.541633 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:00.540494    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1a11cf/distinct-2-w2-1a11cf-kserve-758bb6dd57-j5f2k" podUID="f2d66dea-9f0b-43f3-a774-bb065e88f378" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:48:00.610861 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:00.610816    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-52650f/premium-w0-52650f-kserve-c5bf66f9d-qq5dm" podUID="3455d983-b11b-4cee-bfce-d1f399cf9679" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:48:00.649520 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:00.649470    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-52650f/simulator-w0-52650f-kserve-58b64766c-94qxx" podUID="659c1425-5b67-4d01-bfc2-f1ba56b9a59b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:48:00.786564 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:00.716530    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-52650f/distinct-w0-52650f-kserve-d85899b5-vlhps" podUID="ea1850d9-abba-4977-a540-f6e0eaefb17d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:48:00.857406 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:00.853489    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-021d6d/simulator-w6-021d6d-kserve-6cc97fdf86-dqtpw" podUID="50586f76-4f3b-4e7a-83ac-6c573faee415" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:48:00.948415 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:00.944660    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cefb27/simulator-w3-cefb27-kserve-6b669c9d9f-zx2vm" podUID="7121e81e-f903-42c0-a7a4-bc18d49194e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/ready\": context deadline exceeded"
Oct 30 00:48:01.167665 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:01.167542    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1a11cf/distinct-w2-1a11cf-kserve-854cf4cdf8-cddw6" podUID="32a03478-46eb-4363-97b6-51a32c4099c7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:48:01.168748 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:01.168720    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-1a11cf/distinct-w2-1a11cf-kserve-854cf4cdf8-cddw6"
Oct 30 00:48:01.225545 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:01.225503    5002 patch_prober.go:29] interesting pod/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-fjlsm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 00:48:01.226157 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:01.226106    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-fjlsm" podUID="9c20eb99-f057-44bd-94e2-1972b67aa7f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:48:01.277858 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:01.277004    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-29be73/simulator-w1-29be73-kserve-9958d76db-fks2t" podUID="fbb3c806-8967-4d39-84c6-9dc022fc5cae" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/ready\": context deadline exceeded"
Oct 30 00:48:01.277858 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:01.277440    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-29be73/simulator-w1-29be73-kserve-9958d76db-fks2t"
Oct 30 00:48:01.374529 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:01.374480    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca5363/distinct-2-w4-ca5363-kserve-576576f9b5-xkkqc" podUID="5de8bebf-cfad-49da-9a98-08aedde4442c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:48:01.383302 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:01.383264    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ca5363/distinct-2-w4-ca5363-kserve-576576f9b5-xkkqc"
Oct 30 00:48:01.528406 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:01.438494    5002 patch_prober.go:29] interesting pod/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-24zch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 00:48:01.528406 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:01.527614    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-24zch" podUID="35e8d90c-3415-41a3-8b4f-03a3c738cc99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:48:01.528406 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:01.498355    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-29be73/premium-w1-29be73-kserve-7b777cfc8c-7kgp5" podUID="e6fabbdd-86ca-4cbf-85be-ef250575a355" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:48:01.587500 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:01.540686    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-29be73/premium-w1-29be73-kserve-7b777cfc8c-7kgp5"
Oct 30 00:48:01.587500 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:01.542094    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1a11cf/distinct-2-w2-1a11cf-kserve-758bb6dd57-j5f2k" podUID="f2d66dea-9f0b-43f3-a774-bb065e88f378" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:48:01.587500 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:01.542460    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-1a11cf/distinct-2-w2-1a11cf-kserve-758bb6dd57-j5f2k"
Oct 30 00:48:01.590584 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:01.588081    5002 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-021d6d/simulator-w6-021d6d-kserve-6cc97fdf86-dqtpw" podUID="50586f76-4f3b-4e7a-83ac-6c573faee415" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:48:01.629432 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:01.627897    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-52650f/premium-w0-52650f-kserve-c5bf66f9d-qq5dm" podUID="3455d983-b11b-4cee-bfce-d1f399cf9679" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:48:01.629432 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:01.628346    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-52650f/premium-w0-52650f-kserve-c5bf66f9d-qq5dm"
Oct 30 00:48:01.784021 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:01.783939    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-52650f/simulator-w0-52650f-kserve-58b64766c-94qxx" podUID="659c1425-5b67-4d01-bfc2-f1ba56b9a59b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": context deadline exceeded"
Oct 30 00:48:01.785243 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:01.785224    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-52650f/simulator-w0-52650f-kserve-58b64766c-94qxx"
Oct 30 00:48:01.855610 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:01.855561    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-021d6d/simulator-w6-021d6d-kserve-6cc97fdf86-dqtpw" podUID="50586f76-4f3b-4e7a-83ac-6c573faee415" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:48:01.856610 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:01.856573    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-52650f/distinct-w0-52650f-kserve-d85899b5-vlhps" podUID="ea1850d9-abba-4977-a540-f6e0eaefb17d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:48:01.857546 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:01.857522    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-021d6d/simulator-w6-021d6d-kserve-6cc97fdf86-dqtpw"
Oct 30 00:48:01.857773 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:01.857755    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-52650f/distinct-w0-52650f-kserve-d85899b5-vlhps"
Oct 30 00:48:01.964411 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:01.950395    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cefb27/simulator-w3-cefb27-kserve-6b669c9d9f-zx2vm" podUID="7121e81e-f903-42c0-a7a4-bc18d49194e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:48:02.005416 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:01.965686    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-cefb27/simulator-w3-cefb27-kserve-6b669c9d9f-zx2vm"
Oct 30 00:48:02.005416 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:01.981012    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-1a11cf/distinct-w2-1a11cf-kserve-854cf4cdf8-cddw6"
Oct 30 00:48:02.134028 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:02.133493    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-cefb27/simulator-w3-cefb27-kserve-6b669c9d9f-zx2vm"
Oct 30 00:48:02.314223 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:02.314180    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-29be73/simulator-w1-29be73-kserve-9958d76db-fks2t" podUID="fbb3c806-8967-4d39-84c6-9dc022fc5cae" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:48:02.419940 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:02.385777    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca5363/distinct-2-w4-ca5363-kserve-576576f9b5-xkkqc" podUID="5de8bebf-cfad-49da-9a98-08aedde4442c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:48:02.467424 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:02.463474    5002 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-6f4fba/simulator-w5-6f4fba-kserve-5bf8dfddd5-wcwzh" podUID="3a2c2f8f-649c-4ab6-af98-5d87df0e511f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:48:02.467424 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:02.463779    5002 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-52650f/distinct-w0-52650f-kserve-d85899b5-vlhps" podUID="ea1850d9-abba-4977-a540-f6e0eaefb17d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/health\": context deadline exceeded"
Oct 30 00:48:02.543475 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:02.517900    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6f4fba/simulator-w5-6f4fba-kserve-5bf8dfddd5-wcwzh" podUID="3a2c2f8f-649c-4ab6-af98-5d87df0e511f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:48:02.543475 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:02.542447    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1a11cf/distinct-2-w2-1a11cf-kserve-758bb6dd57-j5f2k" podUID="f2d66dea-9f0b-43f3-a774-bb065e88f378" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:48:02.544244 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:02.543819    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-29be73/premium-w1-29be73-kserve-7b777cfc8c-7kgp5" podUID="e6fabbdd-86ca-4cbf-85be-ef250575a355" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:48:02.585577 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:02.585536    5002 patch_prober.go:29] interesting pod/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-24zch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 00:48:02.585892 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:02.585858    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-24zch" podUID="35e8d90c-3415-41a3-8b4f-03a3c738cc99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:48:02.646834 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:02.638794    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-52650f/premium-w0-52650f-kserve-c5bf66f9d-qq5dm" podUID="3455d983-b11b-4cee-bfce-d1f399cf9679" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:48:02.734700 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:02.733576    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ca5363/distinct-2-w4-ca5363-kserve-576576f9b5-xkkqc"
Oct 30 00:48:02.784957 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:02.784925    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-1a11cf/distinct-2-w2-1a11cf-kserve-758bb6dd57-j5f2k"
Oct 30 00:48:02.842403 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:02.800054    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-52650f/simulator-w0-52650f-kserve-58b64766c-94qxx" podUID="659c1425-5b67-4d01-bfc2-f1ba56b9a59b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:48:02.842861 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:02.842837    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-29be73/simulator-w1-29be73-kserve-9958d76db-fks2t"
Oct 30 00:48:02.862445 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:02.861464    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-021d6d/simulator-w6-021d6d-kserve-6cc97fdf86-dqtpw" podUID="50586f76-4f3b-4e7a-83ac-6c573faee415" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": context deadline exceeded"
Oct 30 00:48:02.862445 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:02.862148    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-52650f/distinct-w0-52650f-kserve-d85899b5-vlhps" podUID="ea1850d9-abba-4977-a540-f6e0eaefb17d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:48:02.933528 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:02.933500    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-52650f/premium-w0-52650f-kserve-c5bf66f9d-qq5dm"
Oct 30 00:48:02.942530 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:02.940077    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-52650f/distinct-w0-52650f-kserve-d85899b5-vlhps"
Oct 30 00:48:02.944464 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:02.944443    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-29be73/premium-w1-29be73-kserve-7b777cfc8c-7kgp5"
Oct 30 00:48:02.947525 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:02.945647    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-021d6d/simulator-w6-021d6d-kserve-6cc97fdf86-dqtpw"
Oct 30 00:48:02.963601 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:02.961956    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-52650f/simulator-w0-52650f-kserve-58b64766c-94qxx"
Oct 30 00:48:03.351554 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:03.351524    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-24zch"
Oct 30 00:48:03.352937 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:03.352913    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-24zch"
Oct 30 00:48:03.355641 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:03.355584    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-24zch"
Oct 30 00:48:12.861393 ip-10-0-128-7 kubenswrapper[5002]: E1030 00:48:12.861341    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbjdq" containerName="extractor"
Oct 30 00:48:22.789555 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:22.789499    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1a11cf/distinct-2-w2-1a11cf-kserve-758bb6dd57-j5f2k" podUID="f2d66dea-9f0b-43f3-a774-bb065e88f378" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:48:22.874406 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:22.873497    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-021d6d/simulator-w6-021d6d-kserve-6cc97fdf86-dqtpw" podUID="50586f76-4f3b-4e7a-83ac-6c573faee415" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:48:22.878402 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:22.874766    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-52650f/distinct-w0-52650f-kserve-d85899b5-vlhps" podUID="ea1850d9-abba-4977-a540-f6e0eaefb17d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:48:22.943577 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:22.943521    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-52650f/premium-w0-52650f-kserve-c5bf66f9d-qq5dm" podUID="3455d983-b11b-4cee-bfce-d1f399cf9679" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:48:22.964577 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:22.964521    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-52650f/simulator-w0-52650f-kserve-58b64766c-94qxx" podUID="659c1425-5b67-4d01-bfc2-f1ba56b9a59b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:48:22.977290 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:22.977244    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1a11cf/distinct-w2-1a11cf-kserve-854cf4cdf8-cddw6" podUID="32a03478-46eb-4363-97b6-51a32c4099c7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:48:23.143602 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:23.143495    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cefb27/simulator-w3-cefb27-kserve-6b669c9d9f-zx2vm" podUID="7121e81e-f903-42c0-a7a4-bc18d49194e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:48:23.327617 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:23.327549    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-29be73/simulator-w1-29be73-kserve-9958d76db-fks2t" podUID="fbb3c806-8967-4d39-84c6-9dc022fc5cae" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:48:23.532544 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:23.532490    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6f4fba/simulator-w5-6f4fba-kserve-5bf8dfddd5-wcwzh" podUID="3a2c2f8f-649c-4ab6-af98-5d87df0e511f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:48:23.595408 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:23.586488    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-29be73/premium-w1-29be73-kserve-7b777cfc8c-7kgp5" podUID="e6fabbdd-86ca-4cbf-85be-ef250575a355" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": context deadline exceeded"
Oct 30 00:48:23.790595 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:23.790484    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1a11cf/distinct-2-w2-1a11cf-kserve-758bb6dd57-j5f2k" podUID="f2d66dea-9f0b-43f3-a774-bb065e88f378" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:48:23.828405 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:23.808467    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-1a11cf/distinct-2-w2-1a11cf-kserve-758bb6dd57-j5f2k"
Oct 30 00:48:23.906838 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:23.906793    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-52650f/distinct-w0-52650f-kserve-d85899b5-vlhps" podUID="ea1850d9-abba-4977-a540-f6e0eaefb17d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:48:23.909981 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:23.909788    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-52650f/distinct-w0-52650f-kserve-d85899b5-vlhps"
Oct 30 00:48:23.912889 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:23.911248    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-021d6d/simulator-w6-021d6d-kserve-6cc97fdf86-dqtpw" podUID="50586f76-4f3b-4e7a-83ac-6c573faee415" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:48:23.912889 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:23.911629    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-021d6d/simulator-w6-021d6d-kserve-6cc97fdf86-dqtpw"
Oct 30 00:48:24.003626 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:23.958416    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-52650f/premium-w0-52650f-kserve-c5bf66f9d-qq5dm" podUID="3455d983-b11b-4cee-bfce-d1f399cf9679" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:48:24.047498 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:23.993469    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1a11cf/distinct-w2-1a11cf-kserve-854cf4cdf8-cddw6" podUID="32a03478-46eb-4363-97b6-51a32c4099c7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:48:24.047498 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:24.034437    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-52650f/premium-w0-52650f-kserve-c5bf66f9d-qq5dm"
Oct 30 00:48:24.047498 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:24.034504    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-1a11cf/distinct-w2-1a11cf-kserve-854cf4cdf8-cddw6"
Oct 30 00:48:24.090685 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:24.090643    5002 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-29be73/simulator-w1-29be73-kserve-9958d76db-fks2t" podUID="fbb3c806-8967-4d39-84c6-9dc022fc5cae" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:48:24.347103 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:24.346995    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-29be73/simulator-w1-29be73-kserve-9958d76db-fks2t" podUID="fbb3c806-8967-4d39-84c6-9dc022fc5cae" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:48:24.348210 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:24.348187    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-29be73/simulator-w1-29be73-kserve-9958d76db-fks2t"
Oct 30 00:48:24.542801 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:24.539342    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6f4fba/simulator-w5-6f4fba-kserve-5bf8dfddd5-wcwzh" podUID="3a2c2f8f-649c-4ab6-af98-5d87df0e511f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/ready\": context deadline exceeded"
Oct 30 00:48:24.587113 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:24.586712    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-29be73/premium-w1-29be73-kserve-7b777cfc8c-7kgp5" podUID="e6fabbdd-86ca-4cbf-85be-ef250575a355" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:48:24.859405 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:24.853246    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1a11cf/distinct-2-w2-1a11cf-kserve-758bb6dd57-j5f2k" podUID="f2d66dea-9f0b-43f3-a774-bb065e88f378" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:48:24.965436 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:24.956211    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-52650f/distinct-w0-52650f-kserve-d85899b5-vlhps" podUID="ea1850d9-abba-4977-a540-f6e0eaefb17d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded"
Oct 30 00:48:25.048405 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:25.016713    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-52650f/simulator-w0-52650f-kserve-58b64766c-94qxx" podUID="659c1425-5b67-4d01-bfc2-f1ba56b9a59b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:48:25.093784 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:25.093736    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1a11cf/distinct-w2-1a11cf-kserve-854cf4cdf8-cddw6" podUID="32a03478-46eb-4363-97b6-51a32c4099c7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:48:25.104649 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:25.104598    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-021d6d/simulator-w6-021d6d-kserve-6cc97fdf86-dqtpw" podUID="50586f76-4f3b-4e7a-83ac-6c573faee415" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": context deadline exceeded"
Oct 30 00:48:25.164492 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:25.163683    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-52650f/premium-w0-52650f-kserve-c5bf66f9d-qq5dm" podUID="3455d983-b11b-4cee-bfce-d1f399cf9679" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": context deadline exceeded"
Oct 30 00:48:25.176629 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:25.176586    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cefb27/simulator-w3-cefb27-kserve-6b669c9d9f-zx2vm" podUID="7121e81e-f903-42c0-a7a4-bc18d49194e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:48:25.319409 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:25.296681    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-6f4fba/simulator-w5-6f4fba-kserve-5bf8dfddd5-wcwzh"
Oct 30 00:48:25.319409 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:25.303953    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-29be73/premium-w1-29be73-kserve-7b777cfc8c-7kgp5"
Oct 30 00:48:25.319409 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:25.304073    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-29be73/simulator-w1-29be73-kserve-9958d76db-fks2t"
Oct 30 00:48:25.319409 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:25.304120    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-cefb27/simulator-w3-cefb27-kserve-6b669c9d9f-zx2vm"
Oct 30 00:48:25.319409 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:25.304180    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-6f4fba/simulator-w5-6f4fba-kserve-5bf8dfddd5-wcwzh"
Oct 30 00:48:25.319409 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:25.304234    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-52650f/distinct-w0-52650f-kserve-d85899b5-vlhps"
Oct 30 00:48:25.319409 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:25.313120    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-1a11cf/distinct-w2-1a11cf-kserve-854cf4cdf8-cddw6"
Oct 30 00:48:25.319409 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:25.315331    5002 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-52650f/simulator-w0-52650f-kserve-58b64766c-94qxx" podUID="659c1425-5b67-4d01-bfc2-f1ba56b9a59b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:48:25.319409 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:25.316795    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-1a11cf/distinct-2-w2-1a11cf-kserve-758bb6dd57-j5f2k"
Oct 30 00:48:25.319409 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:25.318030    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-29be73/premium-w1-29be73-kserve-7b777cfc8c-7kgp5"
Oct 30 00:48:25.320084 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:25.319555    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-cefb27/simulator-w3-cefb27-kserve-6b669c9d9f-zx2vm"
Oct 30 00:48:25.337504 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:25.337476    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-021d6d/simulator-w6-021d6d-kserve-6cc97fdf86-dqtpw"
Oct 30 00:48:25.339594 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:25.339569    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-52650f/premium-w0-52650f-kserve-c5bf66f9d-qq5dm"
Oct 30 00:48:25.372868 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:25.372808    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-655f3853-6rtdl"]
Oct 30 00:48:25.373881 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:25.373853    5002 cpu_manager.go:414] "containerMap: removing container" podUID="8cb5fca0-dafe-4619-8341-1ad59e427c64" containerName="revoke-keys"
Oct 30 00:48:25.374150 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:25.374133    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8cb5fca0-dafe-4619-8341-1ad59e427c64" containerName="revoke-keys"
Oct 30 00:48:25.374535 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:25.374517    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="8cb5fca0-dafe-4619-8341-1ad59e427c64" containerName="revoke-keys"
Oct 30 00:48:25.380320 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:25.380296    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-655f3853-6rtdl"
Oct 30 00:48:25.385688 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:25.385665    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-655f3853-6rtdl"]
Oct 30 00:48:25.399012 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:25.398977    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 00:48:25.399165 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:25.399048    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 00:48:25.399165 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:25.399139    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 00:48:25.399272 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:25.399236    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5phsh\""
Oct 30 00:48:25.419822 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:25.419757    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c4ea6fd-d9b4-4a04-b3cd-dd2707844a2c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-655f3853-6rtdl\" (UID: \"2c4ea6fd-d9b4-4a04-b3cd-dd2707844a2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-655f3853-6rtdl"
Oct 30 00:48:25.419951 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:25.419831    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c4ea6fd-d9b4-4a04-b3cd-dd2707844a2c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-655f3853-6rtdl\" (UID: \"2c4ea6fd-d9b4-4a04-b3cd-dd2707844a2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-655f3853-6rtdl"
Oct 30 00:48:25.419991 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:25.419959    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76fzt\" (UniqueName: \"kubernetes.io/projected/2c4ea6fd-d9b4-4a04-b3cd-dd2707844a2c-kube-api-access-76fzt\") pod \"maas-api-revoke-sub-655f3853-6rtdl\" (UID: \"2c4ea6fd-d9b4-4a04-b3cd-dd2707844a2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-655f3853-6rtdl"
Oct 30 00:48:25.520972 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:25.520937    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76fzt\" (UniqueName: \"kubernetes.io/projected/2c4ea6fd-d9b4-4a04-b3cd-dd2707844a2c-kube-api-access-76fzt\") pod \"maas-api-revoke-sub-655f3853-6rtdl\" (UID: \"2c4ea6fd-d9b4-4a04-b3cd-dd2707844a2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-655f3853-6rtdl"
Oct 30 00:48:25.521178 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:25.521154    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c4ea6fd-d9b4-4a04-b3cd-dd2707844a2c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-655f3853-6rtdl\" (UID: \"2c4ea6fd-d9b4-4a04-b3cd-dd2707844a2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-655f3853-6rtdl"
Oct 30 00:48:25.521272 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:25.521254    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c4ea6fd-d9b4-4a04-b3cd-dd2707844a2c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-655f3853-6rtdl\" (UID: \"2c4ea6fd-d9b4-4a04-b3cd-dd2707844a2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-655f3853-6rtdl"
Oct 30 00:48:25.522134 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:25.522106    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c4ea6fd-d9b4-4a04-b3cd-dd2707844a2c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-655f3853-6rtdl\" (UID: \"2c4ea6fd-d9b4-4a04-b3cd-dd2707844a2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-655f3853-6rtdl"
Oct 30 00:48:25.523846 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:25.523823    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c4ea6fd-d9b4-4a04-b3cd-dd2707844a2c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-655f3853-6rtdl\" (UID: \"2c4ea6fd-d9b4-4a04-b3cd-dd2707844a2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-655f3853-6rtdl"
Oct 30 00:48:25.546529 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:25.546499    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76fzt\" (UniqueName: \"kubernetes.io/projected/2c4ea6fd-d9b4-4a04-b3cd-dd2707844a2c-kube-api-access-76fzt\") pod \"maas-api-revoke-sub-655f3853-6rtdl\" (UID: \"2c4ea6fd-d9b4-4a04-b3cd-dd2707844a2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-655f3853-6rtdl"
Oct 30 00:48:25.719710 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:25.719678    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-655f3853-6rtdl"
Oct 30 00:48:26.040739 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:48:26.040707    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c4ea6fd_d9b4_4a04_b3cd_dd2707844a2c.slice/crio-8dd6cebccdfd5f22c869070771bb1e5d87848eef3aabab06445094a87973a593 WatchSource:0}: Error finding container 8dd6cebccdfd5f22c869070771bb1e5d87848eef3aabab06445094a87973a593: Status 404 returned error can't find the container with id 8dd6cebccdfd5f22c869070771bb1e5d87848eef3aabab06445094a87973a593
Oct 30 00:48:26.041085 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:26.040776    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-655f3853-6rtdl"]
Oct 30 00:48:26.384897 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:26.384799    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-655f3853-6rtdl" event={"ID":"2c4ea6fd-d9b4-4a04-b3cd-dd2707844a2c","Type":"ContainerStarted","Data":"8dd6cebccdfd5f22c869070771bb1e5d87848eef3aabab06445094a87973a593"}
Oct 30 00:48:27.398920 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:27.398877    5002 generic.go:356] "Generic (PLEG): container finished" podID="2c4ea6fd-d9b4-4a04-b3cd-dd2707844a2c" containerID="b845670c42362ad9fce39f5ea737b4b1a0a3ccc248c5696f4211b44100feff89" exitCode=0
Oct 30 00:48:27.399445 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:27.399133    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-655f3853-6rtdl" event={"ID":"2c4ea6fd-d9b4-4a04-b3cd-dd2707844a2c","Type":"ContainerDied","Data":"b845670c42362ad9fce39f5ea737b4b1a0a3ccc248c5696f4211b44100feff89"}
Oct 30 00:48:28.562317 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:28.562279    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-655f3853-6rtdl"
Oct 30 00:48:28.659570 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:28.659472    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c4ea6fd-d9b4-4a04-b3cd-dd2707844a2c-kube-api-access-76fzt\" (UniqueName: \"kubernetes.io/projected/2c4ea6fd-d9b4-4a04-b3cd-dd2707844a2c-kube-api-access-76fzt\") pod \"2c4ea6fd-d9b4-4a04-b3cd-dd2707844a2c\" (UID: \"2c4ea6fd-d9b4-4a04-b3cd-dd2707844a2c\") "
Oct 30 00:48:28.659570 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:28.659515    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c4ea6fd-d9b4-4a04-b3cd-dd2707844a2c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c4ea6fd-d9b4-4a04-b3cd-dd2707844a2c-maas-api-openshift-ca\") pod \"2c4ea6fd-d9b4-4a04-b3cd-dd2707844a2c\" (UID: \"2c4ea6fd-d9b4-4a04-b3cd-dd2707844a2c\") "
Oct 30 00:48:28.659799 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:28.659648    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2c4ea6fd-d9b4-4a04-b3cd-dd2707844a2c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c4ea6fd-d9b4-4a04-b3cd-dd2707844a2c-maas-api-xks-ca\") pod \"2c4ea6fd-d9b4-4a04-b3cd-dd2707844a2c\" (UID: \"2c4ea6fd-d9b4-4a04-b3cd-dd2707844a2c\") "
Oct 30 00:48:28.659935 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:28.659912    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c4ea6fd-d9b4-4a04-b3cd-dd2707844a2c-maas-api-openshift-ca" pod "2c4ea6fd-d9b4-4a04-b3cd-dd2707844a2c" (UID: "2c4ea6fd-d9b4-4a04-b3cd-dd2707844a2c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:48:28.660407 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:28.660354    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c4ea6fd-d9b4-4a04-b3cd-dd2707844a2c-maas-api-openshift-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:48:28.661833 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:28.661805    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c4ea6fd-d9b4-4a04-b3cd-dd2707844a2c-maas-api-xks-ca" pod "2c4ea6fd-d9b4-4a04-b3cd-dd2707844a2c" (UID: "2c4ea6fd-d9b4-4a04-b3cd-dd2707844a2c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:48:28.661923 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:28.661834    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c4ea6fd-d9b4-4a04-b3cd-dd2707844a2c-kube-api-access-76fzt" pod "2c4ea6fd-d9b4-4a04-b3cd-dd2707844a2c" (UID: "2c4ea6fd-d9b4-4a04-b3cd-dd2707844a2c"). InnerVolumeSpecName "kube-api-access-76fzt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:48:28.762143 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:28.762098    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c4ea6fd-d9b4-4a04-b3cd-dd2707844a2c-maas-api-xks-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:48:28.762143 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:28.762141    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76fzt\" (UniqueName: \"kubernetes.io/projected/2c4ea6fd-d9b4-4a04-b3cd-dd2707844a2c-kube-api-access-76fzt\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:48:29.083218 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:29.083179    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a14ef684-fl4gj"]
Oct 30 00:48:29.084053 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:29.084029    5002 cpu_manager.go:414] "containerMap: removing container" podUID="2c4ea6fd-d9b4-4a04-b3cd-dd2707844a2c" containerName="revoke-keys"
Oct 30 00:48:29.084053 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:29.084051    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c4ea6fd-d9b4-4a04-b3cd-dd2707844a2c" containerName="revoke-keys"
Oct 30 00:48:29.084250 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:29.084199    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c4ea6fd-d9b4-4a04-b3cd-dd2707844a2c" containerName="revoke-keys"
Oct 30 00:48:29.088276 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:29.088252    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a14ef684-fl4gj"
Oct 30 00:48:29.107611 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:29.107581    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a14ef684-fl4gj"]
Oct 30 00:48:29.168313 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:29.168256    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7245d167-0ff4-4064-892d-e4d91aed3ee0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a14ef684-fl4gj\" (UID: \"7245d167-0ff4-4064-892d-e4d91aed3ee0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a14ef684-fl4gj"
Oct 30 00:48:29.168540 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:29.168520    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7245d167-0ff4-4064-892d-e4d91aed3ee0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a14ef684-fl4gj\" (UID: \"7245d167-0ff4-4064-892d-e4d91aed3ee0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a14ef684-fl4gj"
Oct 30 00:48:29.168689 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:29.168665    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8vw2\" (UniqueName: \"kubernetes.io/projected/7245d167-0ff4-4064-892d-e4d91aed3ee0-kube-api-access-v8vw2\") pod \"maas-api-revoke-sub-a14ef684-fl4gj\" (UID: \"7245d167-0ff4-4064-892d-e4d91aed3ee0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a14ef684-fl4gj"
Oct 30 00:48:29.269754 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:29.269710    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7245d167-0ff4-4064-892d-e4d91aed3ee0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a14ef684-fl4gj\" (UID: \"7245d167-0ff4-4064-892d-e4d91aed3ee0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a14ef684-fl4gj"
Oct 30 00:48:29.269948 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:29.269814    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8vw2\" (UniqueName: \"kubernetes.io/projected/7245d167-0ff4-4064-892d-e4d91aed3ee0-kube-api-access-v8vw2\") pod \"maas-api-revoke-sub-a14ef684-fl4gj\" (UID: \"7245d167-0ff4-4064-892d-e4d91aed3ee0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a14ef684-fl4gj"
Oct 30 00:48:29.269948 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:29.269916    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7245d167-0ff4-4064-892d-e4d91aed3ee0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a14ef684-fl4gj\" (UID: \"7245d167-0ff4-4064-892d-e4d91aed3ee0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a14ef684-fl4gj"
Oct 30 00:48:29.270587 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:29.270560    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7245d167-0ff4-4064-892d-e4d91aed3ee0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a14ef684-fl4gj\" (UID: \"7245d167-0ff4-4064-892d-e4d91aed3ee0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a14ef684-fl4gj"
Oct 30 00:48:29.272158 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:29.272130    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7245d167-0ff4-4064-892d-e4d91aed3ee0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a14ef684-fl4gj\" (UID: \"7245d167-0ff4-4064-892d-e4d91aed3ee0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a14ef684-fl4gj"
Oct 30 00:48:29.295007 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:29.294982    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8vw2\" (UniqueName: \"kubernetes.io/projected/7245d167-0ff4-4064-892d-e4d91aed3ee0-kube-api-access-v8vw2\") pod \"maas-api-revoke-sub-a14ef684-fl4gj\" (UID: \"7245d167-0ff4-4064-892d-e4d91aed3ee0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a14ef684-fl4gj"
Oct 30 00:48:29.405257 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:29.405161    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a14ef684-fl4gj"
Oct 30 00:48:29.412274 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:29.412238    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-655f3853-6rtdl"
Oct 30 00:48:29.412412 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:29.412273    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-655f3853-6rtdl" event={"ID":"2c4ea6fd-d9b4-4a04-b3cd-dd2707844a2c","Type":"ContainerDied","Data":"8dd6cebccdfd5f22c869070771bb1e5d87848eef3aabab06445094a87973a593"}
Oct 30 00:48:29.412412 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:29.412321    5002 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8dd6cebccdfd5f22c869070771bb1e5d87848eef3aabab06445094a87973a593"
Oct 30 00:48:29.708037 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:48:29.708006    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7245d167_0ff4_4064_892d_e4d91aed3ee0.slice/crio-1a2c6a0aa950206e546c3e581c6110a2d88313ab75b8632cee2c2794e06bf1a2 WatchSource:0}: Error finding container 1a2c6a0aa950206e546c3e581c6110a2d88313ab75b8632cee2c2794e06bf1a2: Status 404 returned error can't find the container with id 1a2c6a0aa950206e546c3e581c6110a2d88313ab75b8632cee2c2794e06bf1a2
Oct 30 00:48:29.708842 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:29.708812    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a14ef684-fl4gj"]
Oct 30 00:48:30.420431 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:30.420334    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a14ef684-fl4gj" event={"ID":"7245d167-0ff4-4064-892d-e4d91aed3ee0","Type":"ContainerStarted","Data":"830eba8dee9e1a4663aa2b46c5160b2a979fbf4b7e953ee008cd061682b7080d"}
Oct 30 00:48:30.420578 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:30.420439    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a14ef684-fl4gj" event={"ID":"7245d167-0ff4-4064-892d-e4d91aed3ee0","Type":"ContainerStarted","Data":"1a2c6a0aa950206e546c3e581c6110a2d88313ab75b8632cee2c2794e06bf1a2"}
Oct 30 00:48:30.469900 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:30.469838    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a14ef684-fl4gj" podStartSLOduration=1.469818973 podStartE2EDuration="1.469818973s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:48:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:48:30.465915015 +0000 UTC m=+1006.082347912" watchObservedRunningTime="2026-10-30 00:48:30.469818973 +0000 UTC m=+1006.086251853"
Oct 30 00:48:31.430084 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:31.430017    5002 generic.go:356] "Generic (PLEG): container finished" podID="7245d167-0ff4-4064-892d-e4d91aed3ee0" containerID="830eba8dee9e1a4663aa2b46c5160b2a979fbf4b7e953ee008cd061682b7080d" exitCode=0
Oct 30 00:48:31.430560 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:31.430098    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a14ef684-fl4gj" event={"ID":"7245d167-0ff4-4064-892d-e4d91aed3ee0","Type":"ContainerDied","Data":"830eba8dee9e1a4663aa2b46c5160b2a979fbf4b7e953ee008cd061682b7080d"}
Oct 30 00:48:32.601212 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:32.601183    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a14ef684-fl4gj"
Oct 30 00:48:32.718857 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:32.718772    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7245d167-0ff4-4064-892d-e4d91aed3ee0-kube-api-access-v8vw2\" (UniqueName: \"kubernetes.io/projected/7245d167-0ff4-4064-892d-e4d91aed3ee0-kube-api-access-v8vw2\") pod \"7245d167-0ff4-4064-892d-e4d91aed3ee0\" (UID: \"7245d167-0ff4-4064-892d-e4d91aed3ee0\") "
Oct 30 00:48:32.719035 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:32.718881    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7245d167-0ff4-4064-892d-e4d91aed3ee0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7245d167-0ff4-4064-892d-e4d91aed3ee0-maas-api-xks-ca\") pod \"7245d167-0ff4-4064-892d-e4d91aed3ee0\" (UID: \"7245d167-0ff4-4064-892d-e4d91aed3ee0\") "
Oct 30 00:48:32.719158 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:32.719143    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7245d167-0ff4-4064-892d-e4d91aed3ee0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7245d167-0ff4-4064-892d-e4d91aed3ee0-maas-api-openshift-ca\") pod \"7245d167-0ff4-4064-892d-e4d91aed3ee0\" (UID: \"7245d167-0ff4-4064-892d-e4d91aed3ee0\") "
Oct 30 00:48:32.719480 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:32.719445    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7245d167-0ff4-4064-892d-e4d91aed3ee0-maas-api-openshift-ca" pod "7245d167-0ff4-4064-892d-e4d91aed3ee0" (UID: "7245d167-0ff4-4064-892d-e4d91aed3ee0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:48:32.719621 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:32.719605    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7245d167-0ff4-4064-892d-e4d91aed3ee0-maas-api-openshift-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:48:32.720953 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:32.720935    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7245d167-0ff4-4064-892d-e4d91aed3ee0-maas-api-xks-ca" pod "7245d167-0ff4-4064-892d-e4d91aed3ee0" (UID: "7245d167-0ff4-4064-892d-e4d91aed3ee0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:48:32.721018 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:32.720964    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7245d167-0ff4-4064-892d-e4d91aed3ee0-kube-api-access-v8vw2" pod "7245d167-0ff4-4064-892d-e4d91aed3ee0" (UID: "7245d167-0ff4-4064-892d-e4d91aed3ee0"). InnerVolumeSpecName "kube-api-access-v8vw2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:48:32.820792 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:32.820757    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8vw2\" (UniqueName: \"kubernetes.io/projected/7245d167-0ff4-4064-892d-e4d91aed3ee0-kube-api-access-v8vw2\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:48:32.820792 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:32.820788    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7245d167-0ff4-4064-892d-e4d91aed3ee0-maas-api-xks-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:48:33.443764 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:33.443734    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a14ef684-fl4gj"
Oct 30 00:48:33.443764 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:33.443753    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a14ef684-fl4gj" event={"ID":"7245d167-0ff4-4064-892d-e4d91aed3ee0","Type":"ContainerDied","Data":"1a2c6a0aa950206e546c3e581c6110a2d88313ab75b8632cee2c2794e06bf1a2"}
Oct 30 00:48:33.443981 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:33.443812    5002 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1a2c6a0aa950206e546c3e581c6110a2d88313ab75b8632cee2c2794e06bf1a2"
Oct 30 00:48:34.151681 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:34.151646    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4932e2d1-hlzs8"]
Oct 30 00:48:34.152445 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:34.152424    5002 cpu_manager.go:414] "containerMap: removing container" podUID="7245d167-0ff4-4064-892d-e4d91aed3ee0" containerName="revoke-keys"
Oct 30 00:48:34.152563 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:34.152446    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7245d167-0ff4-4064-892d-e4d91aed3ee0" containerName="revoke-keys"
Oct 30 00:48:34.152623 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:34.152603    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="7245d167-0ff4-4064-892d-e4d91aed3ee0" containerName="revoke-keys"
Oct 30 00:48:34.156213 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:34.156192    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4932e2d1-hlzs8"
Oct 30 00:48:34.172751 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:34.172717    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 00:48:34.175533 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:34.175496    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4932e2d1-hlzs8"]
Oct 30 00:48:34.184689 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:34.184658    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5phsh\""
Oct 30 00:48:34.185429 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:34.185407    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 00:48:34.185556 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:34.185484    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 00:48:34.236534 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:34.236502    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwggx\" (UniqueName: \"kubernetes.io/projected/9d059dbd-6736-48b6-9a97-7c54269ee2e9-kube-api-access-zwggx\") pod \"maas-api-revoke-sub-4932e2d1-hlzs8\" (UID: \"9d059dbd-6736-48b6-9a97-7c54269ee2e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4932e2d1-hlzs8"
Oct 30 00:48:34.236679 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:34.236546    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d059dbd-6736-48b6-9a97-7c54269ee2e9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4932e2d1-hlzs8\" (UID: \"9d059dbd-6736-48b6-9a97-7c54269ee2e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4932e2d1-hlzs8"
Oct 30 00:48:34.236679 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:34.236666    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d059dbd-6736-48b6-9a97-7c54269ee2e9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4932e2d1-hlzs8\" (UID: \"9d059dbd-6736-48b6-9a97-7c54269ee2e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4932e2d1-hlzs8"
Oct 30 00:48:34.337903 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:34.337867    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwggx\" (UniqueName: \"kubernetes.io/projected/9d059dbd-6736-48b6-9a97-7c54269ee2e9-kube-api-access-zwggx\") pod \"maas-api-revoke-sub-4932e2d1-hlzs8\" (UID: \"9d059dbd-6736-48b6-9a97-7c54269ee2e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4932e2d1-hlzs8"
Oct 30 00:48:34.338087 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:34.337915    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d059dbd-6736-48b6-9a97-7c54269ee2e9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4932e2d1-hlzs8\" (UID: \"9d059dbd-6736-48b6-9a97-7c54269ee2e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4932e2d1-hlzs8"
Oct 30 00:48:34.338087 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:34.337966    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d059dbd-6736-48b6-9a97-7c54269ee2e9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4932e2d1-hlzs8\" (UID: \"9d059dbd-6736-48b6-9a97-7c54269ee2e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4932e2d1-hlzs8"
Oct 30 00:48:34.338671 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:34.338636    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d059dbd-6736-48b6-9a97-7c54269ee2e9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4932e2d1-hlzs8\" (UID: \"9d059dbd-6736-48b6-9a97-7c54269ee2e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4932e2d1-hlzs8"
Oct 30 00:48:34.340421 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:34.340359    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d059dbd-6736-48b6-9a97-7c54269ee2e9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4932e2d1-hlzs8\" (UID: \"9d059dbd-6736-48b6-9a97-7c54269ee2e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4932e2d1-hlzs8"
Oct 30 00:48:34.362648 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:34.362615    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwggx\" (UniqueName: \"kubernetes.io/projected/9d059dbd-6736-48b6-9a97-7c54269ee2e9-kube-api-access-zwggx\") pod \"maas-api-revoke-sub-4932e2d1-hlzs8\" (UID: \"9d059dbd-6736-48b6-9a97-7c54269ee2e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4932e2d1-hlzs8"
Oct 30 00:48:34.470229 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:34.470196    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4932e2d1-hlzs8"
Oct 30 00:48:34.782101 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:48:34.782062    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d059dbd_6736_48b6_9a97_7c54269ee2e9.slice/crio-fae41955caef86304409428294248f8cacf4f0fc8d7eedcd62f6bfbbe1843ab8 WatchSource:0}: Error finding container fae41955caef86304409428294248f8cacf4f0fc8d7eedcd62f6bfbbe1843ab8: Status 404 returned error can't find the container with id fae41955caef86304409428294248f8cacf4f0fc8d7eedcd62f6bfbbe1843ab8
Oct 30 00:48:34.782356 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:34.782334    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4932e2d1-hlzs8"]
Oct 30 00:48:35.460276 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:35.460241    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4932e2d1-hlzs8" event={"ID":"9d059dbd-6736-48b6-9a97-7c54269ee2e9","Type":"ContainerStarted","Data":"b49d2e7601a7b6e1316fba4311502facbca84ede0e3d7b1d333e648f1044dd46"}
Oct 30 00:48:35.460804 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:35.460310    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4932e2d1-hlzs8" event={"ID":"9d059dbd-6736-48b6-9a97-7c54269ee2e9","Type":"ContainerStarted","Data":"fae41955caef86304409428294248f8cacf4f0fc8d7eedcd62f6bfbbe1843ab8"}
Oct 30 00:48:35.504137 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:35.504084    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4932e2d1-hlzs8" podStartSLOduration=1.5040696630000001 podStartE2EDuration="1.504069663s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:48:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:48:35.502607847 +0000 UTC m=+1011.119040746" watchObservedRunningTime="2026-10-30 00:48:35.504069663 +0000 UTC m=+1011.120502541"
Oct 30 00:48:36.468276 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:36.468245    5002 generic.go:356] "Generic (PLEG): container finished" podID="9d059dbd-6736-48b6-9a97-7c54269ee2e9" containerID="b49d2e7601a7b6e1316fba4311502facbca84ede0e3d7b1d333e648f1044dd46" exitCode=0
Oct 30 00:48:36.468754 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:36.468333    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4932e2d1-hlzs8" event={"ID":"9d059dbd-6736-48b6-9a97-7c54269ee2e9","Type":"ContainerDied","Data":"b49d2e7601a7b6e1316fba4311502facbca84ede0e3d7b1d333e648f1044dd46"}
Oct 30 00:48:37.657325 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:37.657296    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4932e2d1-hlzs8"
Oct 30 00:48:37.795995 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:37.795897    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d059dbd-6736-48b6-9a97-7c54269ee2e9-kube-api-access-zwggx\" (UniqueName: \"kubernetes.io/projected/9d059dbd-6736-48b6-9a97-7c54269ee2e9-kube-api-access-zwggx\") pod \"9d059dbd-6736-48b6-9a97-7c54269ee2e9\" (UID: \"9d059dbd-6736-48b6-9a97-7c54269ee2e9\") "
Oct 30 00:48:37.795995 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:37.795948    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d059dbd-6736-48b6-9a97-7c54269ee2e9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d059dbd-6736-48b6-9a97-7c54269ee2e9-maas-api-openshift-ca\") pod \"9d059dbd-6736-48b6-9a97-7c54269ee2e9\" (UID: \"9d059dbd-6736-48b6-9a97-7c54269ee2e9\") "
Oct 30 00:48:37.796193 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:37.796024    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d059dbd-6736-48b6-9a97-7c54269ee2e9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d059dbd-6736-48b6-9a97-7c54269ee2e9-maas-api-xks-ca\") pod \"9d059dbd-6736-48b6-9a97-7c54269ee2e9\" (UID: \"9d059dbd-6736-48b6-9a97-7c54269ee2e9\") "
Oct 30 00:48:37.796344 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:37.796318    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d059dbd-6736-48b6-9a97-7c54269ee2e9-maas-api-openshift-ca" pod "9d059dbd-6736-48b6-9a97-7c54269ee2e9" (UID: "9d059dbd-6736-48b6-9a97-7c54269ee2e9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:48:37.796667 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:37.796644    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d059dbd-6736-48b6-9a97-7c54269ee2e9-maas-api-openshift-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:48:37.798250 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:37.798219    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d059dbd-6736-48b6-9a97-7c54269ee2e9-maas-api-xks-ca" pod "9d059dbd-6736-48b6-9a97-7c54269ee2e9" (UID: "9d059dbd-6736-48b6-9a97-7c54269ee2e9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:48:37.798250 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:37.798221    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d059dbd-6736-48b6-9a97-7c54269ee2e9-kube-api-access-zwggx" pod "9d059dbd-6736-48b6-9a97-7c54269ee2e9" (UID: "9d059dbd-6736-48b6-9a97-7c54269ee2e9"). InnerVolumeSpecName "kube-api-access-zwggx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:48:37.897755 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:37.897719    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zwggx\" (UniqueName: \"kubernetes.io/projected/9d059dbd-6736-48b6-9a97-7c54269ee2e9-kube-api-access-zwggx\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:48:37.897755 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:37.897750    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d059dbd-6736-48b6-9a97-7c54269ee2e9-maas-api-xks-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:48:38.482190 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:38.482162    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4932e2d1-hlzs8"
Oct 30 00:48:38.482190 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:38.482172    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4932e2d1-hlzs8" event={"ID":"9d059dbd-6736-48b6-9a97-7c54269ee2e9","Type":"ContainerDied","Data":"fae41955caef86304409428294248f8cacf4f0fc8d7eedcd62f6bfbbe1843ab8"}
Oct 30 00:48:38.482490 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:38.482255    5002 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fae41955caef86304409428294248f8cacf4f0fc8d7eedcd62f6bfbbe1843ab8"
Oct 30 00:48:39.264492 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:39.264459    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0ea9289-t5f56"]
Oct 30 00:48:39.265003 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:39.264989    5002 cpu_manager.go:414] "containerMap: removing container" podUID="9d059dbd-6736-48b6-9a97-7c54269ee2e9" containerName="revoke-keys"
Oct 30 00:48:39.265047 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:39.265004    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d059dbd-6736-48b6-9a97-7c54269ee2e9" containerName="revoke-keys"
Oct 30 00:48:39.265116 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:39.265107    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d059dbd-6736-48b6-9a97-7c54269ee2e9" containerName="revoke-keys"
Oct 30 00:48:39.268562 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:39.268543    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0ea9289-t5f56"
Oct 30 00:48:39.288694 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:39.288665    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0ea9289-t5f56"]
Oct 30 00:48:39.298213 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:39.298184    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 00:48:39.298456 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:39.298227    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5phsh\""
Oct 30 00:48:39.298456 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:39.298274    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 00:48:39.300122 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:39.300083    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 00:48:39.415423 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:39.415350    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vr72g\" (UniqueName: \"kubernetes.io/projected/ba2785d1-40fc-4081-ab4b-71f251ffed23-kube-api-access-vr72g\") pod \"maas-api-revoke-sub-f0ea9289-t5f56\" (UID: \"ba2785d1-40fc-4081-ab4b-71f251ffed23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0ea9289-t5f56"
Oct 30 00:48:39.415603 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:39.415534    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba2785d1-40fc-4081-ab4b-71f251ffed23-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f0ea9289-t5f56\" (UID: \"ba2785d1-40fc-4081-ab4b-71f251ffed23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0ea9289-t5f56"
Oct 30 00:48:39.415645 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:39.415629    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba2785d1-40fc-4081-ab4b-71f251ffed23-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f0ea9289-t5f56\" (UID: \"ba2785d1-40fc-4081-ab4b-71f251ffed23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0ea9289-t5f56"
Oct 30 00:48:39.516721 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:39.516629    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba2785d1-40fc-4081-ab4b-71f251ffed23-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f0ea9289-t5f56\" (UID: \"ba2785d1-40fc-4081-ab4b-71f251ffed23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0ea9289-t5f56"
Oct 30 00:48:39.516875 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:39.516759    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vr72g\" (UniqueName: \"kubernetes.io/projected/ba2785d1-40fc-4081-ab4b-71f251ffed23-kube-api-access-vr72g\") pod \"maas-api-revoke-sub-f0ea9289-t5f56\" (UID: \"ba2785d1-40fc-4081-ab4b-71f251ffed23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0ea9289-t5f56"
Oct 30 00:48:39.516875 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:39.516818    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba2785d1-40fc-4081-ab4b-71f251ffed23-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f0ea9289-t5f56\" (UID: \"ba2785d1-40fc-4081-ab4b-71f251ffed23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0ea9289-t5f56"
Oct 30 00:48:39.517456 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:39.517433    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba2785d1-40fc-4081-ab4b-71f251ffed23-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f0ea9289-t5f56\" (UID: \"ba2785d1-40fc-4081-ab4b-71f251ffed23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0ea9289-t5f56"
Oct 30 00:48:39.518964 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:39.518936    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba2785d1-40fc-4081-ab4b-71f251ffed23-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f0ea9289-t5f56\" (UID: \"ba2785d1-40fc-4081-ab4b-71f251ffed23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0ea9289-t5f56"
Oct 30 00:48:39.541660 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:39.541636    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vr72g\" (UniqueName: \"kubernetes.io/projected/ba2785d1-40fc-4081-ab4b-71f251ffed23-kube-api-access-vr72g\") pod \"maas-api-revoke-sub-f0ea9289-t5f56\" (UID: \"ba2785d1-40fc-4081-ab4b-71f251ffed23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0ea9289-t5f56"
Oct 30 00:48:39.583722 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:39.583689    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0ea9289-t5f56"
Oct 30 00:48:39.891018 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:48:39.890984    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba2785d1_40fc_4081_ab4b_71f251ffed23.slice/crio-c7ab3571beeb5356ebc2c24f4d1b9759feacea23cfcec24c43777771a4fec96e WatchSource:0}: Error finding container c7ab3571beeb5356ebc2c24f4d1b9759feacea23cfcec24c43777771a4fec96e: Status 404 returned error can't find the container with id c7ab3571beeb5356ebc2c24f4d1b9759feacea23cfcec24c43777771a4fec96e
Oct 30 00:48:39.891489 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:39.891463    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0ea9289-t5f56"]
Oct 30 00:48:40.500354 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:40.500310    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0ea9289-t5f56" event={"ID":"ba2785d1-40fc-4081-ab4b-71f251ffed23","Type":"ContainerStarted","Data":"d5f162f015609b4bd5ea36e6e2b021b373b068a5b8583b3c405da39041cb6ea2"}
Oct 30 00:48:40.500875 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:40.500408    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0ea9289-t5f56" event={"ID":"ba2785d1-40fc-4081-ab4b-71f251ffed23","Type":"ContainerStarted","Data":"c7ab3571beeb5356ebc2c24f4d1b9759feacea23cfcec24c43777771a4fec96e"}
Oct 30 00:48:40.545930 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:40.545875    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0ea9289-t5f56" podStartSLOduration=1.545861989 podStartE2EDuration="1.545861989s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:48:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:48:40.542787943 +0000 UTC m=+1016.159220822" watchObservedRunningTime="2026-10-30 00:48:40.545861989 +0000 UTC m=+1016.162294867"
Oct 30 00:48:41.509372 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:41.509340    5002 generic.go:356] "Generic (PLEG): container finished" podID="ba2785d1-40fc-4081-ab4b-71f251ffed23" containerID="d5f162f015609b4bd5ea36e6e2b021b373b068a5b8583b3c405da39041cb6ea2" exitCode=0
Oct 30 00:48:41.509752 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:41.509424    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0ea9289-t5f56" event={"ID":"ba2785d1-40fc-4081-ab4b-71f251ffed23","Type":"ContainerDied","Data":"d5f162f015609b4bd5ea36e6e2b021b373b068a5b8583b3c405da39041cb6ea2"}
Oct 30 00:48:41.927597 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:41.927522    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bedf5b47-7qtsj"]
Oct 30 00:48:41.931634 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:41.931617    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bedf5b47-7qtsj"
Oct 30 00:48:41.950325 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:41.950291    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bedf5b47-7qtsj"]
Oct 30 00:48:42.048544 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:42.048502    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vlqf\" (UniqueName: \"kubernetes.io/projected/c2ae039e-d1d3-4190-8781-f58a8f778eee-kube-api-access-7vlqf\") pod \"maas-api-revoke-sub-bedf5b47-7qtsj\" (UID: \"c2ae039e-d1d3-4190-8781-f58a8f778eee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bedf5b47-7qtsj"
Oct 30 00:48:42.048722 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:42.048582    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2ae039e-d1d3-4190-8781-f58a8f778eee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bedf5b47-7qtsj\" (UID: \"c2ae039e-d1d3-4190-8781-f58a8f778eee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bedf5b47-7qtsj"
Oct 30 00:48:42.048722 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:42.048664    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2ae039e-d1d3-4190-8781-f58a8f778eee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bedf5b47-7qtsj\" (UID: \"c2ae039e-d1d3-4190-8781-f58a8f778eee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bedf5b47-7qtsj"
Oct 30 00:48:42.149980 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:42.149935    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2ae039e-d1d3-4190-8781-f58a8f778eee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bedf5b47-7qtsj\" (UID: \"c2ae039e-d1d3-4190-8781-f58a8f778eee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bedf5b47-7qtsj"
Oct 30 00:48:42.150169 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:42.149999    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2ae039e-d1d3-4190-8781-f58a8f778eee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bedf5b47-7qtsj\" (UID: \"c2ae039e-d1d3-4190-8781-f58a8f778eee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bedf5b47-7qtsj"
Oct 30 00:48:42.150238 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:42.150215    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vlqf\" (UniqueName: \"kubernetes.io/projected/c2ae039e-d1d3-4190-8781-f58a8f778eee-kube-api-access-7vlqf\") pod \"maas-api-revoke-sub-bedf5b47-7qtsj\" (UID: \"c2ae039e-d1d3-4190-8781-f58a8f778eee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bedf5b47-7qtsj"
Oct 30 00:48:42.150814 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:42.150790    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2ae039e-d1d3-4190-8781-f58a8f778eee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bedf5b47-7qtsj\" (UID: \"c2ae039e-d1d3-4190-8781-f58a8f778eee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bedf5b47-7qtsj"
Oct 30 00:48:42.152970 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:42.152944    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2ae039e-d1d3-4190-8781-f58a8f778eee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bedf5b47-7qtsj\" (UID: \"c2ae039e-d1d3-4190-8781-f58a8f778eee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bedf5b47-7qtsj"
Oct 30 00:48:42.175184 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:42.175157    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vlqf\" (UniqueName: \"kubernetes.io/projected/c2ae039e-d1d3-4190-8781-f58a8f778eee-kube-api-access-7vlqf\") pod \"maas-api-revoke-sub-bedf5b47-7qtsj\" (UID: \"c2ae039e-d1d3-4190-8781-f58a8f778eee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bedf5b47-7qtsj"
Oct 30 00:48:42.244873 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:42.244841    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bedf5b47-7qtsj"
Oct 30 00:48:42.552862 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:48:42.552818    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2ae039e_d1d3_4190_8781_f58a8f778eee.slice/crio-6ecbf2f922bcdf7539fd459e71ed0558d328004db1a43de1bb474cd0733a51c8 WatchSource:0}: Error finding container 6ecbf2f922bcdf7539fd459e71ed0558d328004db1a43de1bb474cd0733a51c8: Status 404 returned error can't find the container with id 6ecbf2f922bcdf7539fd459e71ed0558d328004db1a43de1bb474cd0733a51c8
Oct 30 00:48:42.553557 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:42.553525    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bedf5b47-7qtsj"]
Oct 30 00:48:42.670604 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:42.670578    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0ea9289-t5f56"
Oct 30 00:48:42.710470 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:42.710439    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7d8z4"]
Oct 30 00:48:42.760885 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:42.760857    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba2785d1-40fc-4081-ab4b-71f251ffed23-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba2785d1-40fc-4081-ab4b-71f251ffed23-maas-api-openshift-ca\") pod \"ba2785d1-40fc-4081-ab4b-71f251ffed23\" (UID: \"ba2785d1-40fc-4081-ab4b-71f251ffed23\") "
Oct 30 00:48:42.761050 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:42.760948    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba2785d1-40fc-4081-ab4b-71f251ffed23-kube-api-access-vr72g\" (UniqueName: \"kubernetes.io/projected/ba2785d1-40fc-4081-ab4b-71f251ffed23-kube-api-access-vr72g\") pod \"ba2785d1-40fc-4081-ab4b-71f251ffed23\" (UID: \"ba2785d1-40fc-4081-ab4b-71f251ffed23\") "
Oct 30 00:48:42.761050 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:42.761030    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ba2785d1-40fc-4081-ab4b-71f251ffed23-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba2785d1-40fc-4081-ab4b-71f251ffed23-maas-api-xks-ca\") pod \"ba2785d1-40fc-4081-ab4b-71f251ffed23\" (UID: \"ba2785d1-40fc-4081-ab4b-71f251ffed23\") "
Oct 30 00:48:42.761285 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:42.761259    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba2785d1-40fc-4081-ab4b-71f251ffed23-maas-api-openshift-ca" pod "ba2785d1-40fc-4081-ab4b-71f251ffed23" (UID: "ba2785d1-40fc-4081-ab4b-71f251ffed23"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:48:42.761443 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:42.761421    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba2785d1-40fc-4081-ab4b-71f251ffed23-maas-api-openshift-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:48:42.763056 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:42.763035    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba2785d1-40fc-4081-ab4b-71f251ffed23-kube-api-access-vr72g" pod "ba2785d1-40fc-4081-ab4b-71f251ffed23" (UID: "ba2785d1-40fc-4081-ab4b-71f251ffed23"). InnerVolumeSpecName "kube-api-access-vr72g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:48:42.763137 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:42.763114    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba2785d1-40fc-4081-ab4b-71f251ffed23-maas-api-xks-ca" pod "ba2785d1-40fc-4081-ab4b-71f251ffed23" (UID: "ba2785d1-40fc-4081-ab4b-71f251ffed23"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:48:42.862470 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:42.862431    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba2785d1-40fc-4081-ab4b-71f251ffed23-maas-api-xks-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:48:42.862470 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:42.862471    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vr72g\" (UniqueName: \"kubernetes.io/projected/ba2785d1-40fc-4081-ab4b-71f251ffed23-kube-api-access-vr72g\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:48:42.868463 ip-10-0-128-7 kubenswrapper[5002]: E1030 00:48:42.868369    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbjdq" containerName="extractor"
Oct 30 00:48:43.526662 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:43.525899    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bedf5b47-7qtsj" event={"ID":"c2ae039e-d1d3-4190-8781-f58a8f778eee","Type":"ContainerStarted","Data":"042df1b81a33a7400feaae8b97e2c4fda608860736e198543d7dd492020384e2"}
Oct 30 00:48:43.526662 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:43.525962    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bedf5b47-7qtsj" event={"ID":"c2ae039e-d1d3-4190-8781-f58a8f778eee","Type":"ContainerStarted","Data":"6ecbf2f922bcdf7539fd459e71ed0558d328004db1a43de1bb474cd0733a51c8"}
Oct 30 00:48:43.527592 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:43.527566    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0ea9289-t5f56"
Oct 30 00:48:43.527715 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:43.527565    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0ea9289-t5f56" event={"ID":"ba2785d1-40fc-4081-ab4b-71f251ffed23","Type":"ContainerDied","Data":"c7ab3571beeb5356ebc2c24f4d1b9759feacea23cfcec24c43777771a4fec96e"}
Oct 30 00:48:43.527770 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:43.527725    5002 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c7ab3571beeb5356ebc2c24f4d1b9759feacea23cfcec24c43777771a4fec96e"
Oct 30 00:48:43.586511 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:43.583406    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bedf5b47-7qtsj" podStartSLOduration=2.583370692 podStartE2EDuration="2.583370692s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:48:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:48:43.578696531 +0000 UTC m=+1019.195129410" watchObservedRunningTime="2026-10-30 00:48:43.583370692 +0000 UTC m=+1019.199803573"
Oct 30 00:48:44.666405 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:44.655503    5002 generic.go:356] "Generic (PLEG): container finished" podID="c2ae039e-d1d3-4190-8781-f58a8f778eee" containerID="042df1b81a33a7400feaae8b97e2c4fda608860736e198543d7dd492020384e2" exitCode=0
Oct 30 00:48:44.706403 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:44.702908    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bedf5b47-7qtsj" event={"ID":"c2ae039e-d1d3-4190-8781-f58a8f778eee","Type":"ContainerDied","Data":"042df1b81a33a7400feaae8b97e2c4fda608860736e198543d7dd492020384e2"}
Oct 30 00:48:45.109406 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:45.108567    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1a11cf/distinct-w2-1a11cf-kserve-854cf4cdf8-cddw6" podUID="32a03478-46eb-4363-97b6-51a32c4099c7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:48:45.123404 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:45.109809    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-021d6d/simulator-w6-021d6d-kserve-6cc97fdf86-dqtpw" podUID="50586f76-4f3b-4e7a-83ac-6c573faee415" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:48:45.268406 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:45.268002    5002 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-52650f/simulator-w0-52650f-kserve-58b64766c-94qxx" podUID="659c1425-5b67-4d01-bfc2-f1ba56b9a59b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:48:45.321405 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:45.320610    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cefb27/simulator-w3-cefb27-kserve-6b669c9d9f-zx2vm" podUID="7121e81e-f903-42c0-a7a4-bc18d49194e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:48:45.356406 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:45.355659    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-52650f/premium-w0-52650f-kserve-c5bf66f9d-qq5dm" podUID="3455d983-b11b-4cee-bfce-d1f399cf9679" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": context deadline exceeded"
Oct 30 00:48:45.418463 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:45.393769    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca5363/distinct-2-w4-ca5363-kserve-576576f9b5-xkkqc" podUID="5de8bebf-cfad-49da-9a98-08aedde4442c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:48:45.418463 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:45.394137    5002 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-1a11cf/distinct-w2-1a11cf-kserve-854cf4cdf8-cddw6" podUID="32a03478-46eb-4363-97b6-51a32c4099c7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/health\": context deadline exceeded"
Oct 30 00:48:45.528406 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:45.498518    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6f4fba/simulator-w5-6f4fba-kserve-5bf8dfddd5-wcwzh" podUID="3a2c2f8f-649c-4ab6-af98-5d87df0e511f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:48:46.334403 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:46.330662    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cefb27/simulator-w3-cefb27-kserve-6b669c9d9f-zx2vm" podUID="7121e81e-f903-42c0-a7a4-bc18d49194e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:48:46.334403 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:46.331053    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-29be73/premium-w1-29be73-kserve-7b777cfc8c-7kgp5" podUID="e6fabbdd-86ca-4cbf-85be-ef250575a355" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": context deadline exceeded"
Oct 30 00:48:46.334403 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:46.331491    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-cefb27/simulator-w3-cefb27-kserve-6b669c9d9f-zx2vm"
Oct 30 00:48:46.334403 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:46.333674    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1a11cf/distinct-w2-1a11cf-kserve-854cf4cdf8-cddw6" podUID="32a03478-46eb-4363-97b6-51a32c4099c7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:48:46.334403 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:46.333955    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-021d6d/simulator-w6-021d6d-kserve-6cc97fdf86-dqtpw" podUID="50586f76-4f3b-4e7a-83ac-6c573faee415" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": context deadline exceeded"
Oct 30 00:48:46.440517 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:46.357020    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-52650f/distinct-w0-52650f-kserve-d85899b5-vlhps" podUID="ea1850d9-abba-4977-a540-f6e0eaefb17d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded"
Oct 30 00:48:46.440517 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:46.357410    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-1a11cf/distinct-w2-1a11cf-kserve-854cf4cdf8-cddw6"
Oct 30 00:48:46.440517 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:46.357473    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-021d6d/simulator-w6-021d6d-kserve-6cc97fdf86-dqtpw"
Oct 30 00:48:46.440517 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:46.401479    5002 patch_prober.go:29] interesting pod/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-fjlsm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 00:48:46.440517 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:46.401541    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-fjlsm" podUID="9c20eb99-f057-44bd-94e2-1972b67aa7f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:48:46.440517 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:46.414466    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca5363/distinct-2-w4-ca5363-kserve-576576f9b5-xkkqc" podUID="5de8bebf-cfad-49da-9a98-08aedde4442c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:48:46.440517 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:46.414901    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ca5363/distinct-2-w4-ca5363-kserve-576576f9b5-xkkqc"
Oct 30 00:48:46.898459 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:46.896921    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1a11cf/distinct-2-w2-1a11cf-kserve-758bb6dd57-j5f2k" podUID="f2d66dea-9f0b-43f3-a774-bb065e88f378" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:48:47.014405 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:46.989414    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bedf5b47-7qtsj"
Oct 30 00:48:47.051728 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:47.049214    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2ae039e-d1d3-4190-8781-f58a8f778eee-kube-api-access-7vlqf\" (UniqueName: \"kubernetes.io/projected/c2ae039e-d1d3-4190-8781-f58a8f778eee-kube-api-access-7vlqf\") pod \"c2ae039e-d1d3-4190-8781-f58a8f778eee\" (UID: \"c2ae039e-d1d3-4190-8781-f58a8f778eee\") "
Oct 30 00:48:47.051728 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:47.049488    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c2ae039e-d1d3-4190-8781-f58a8f778eee-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2ae039e-d1d3-4190-8781-f58a8f778eee-maas-api-xks-ca\") pod \"c2ae039e-d1d3-4190-8781-f58a8f778eee\" (UID: \"c2ae039e-d1d3-4190-8781-f58a8f778eee\") "
Oct 30 00:48:47.051728 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:47.049534    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2ae039e-d1d3-4190-8781-f58a8f778eee-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2ae039e-d1d3-4190-8781-f58a8f778eee-maas-api-openshift-ca\") pod \"c2ae039e-d1d3-4190-8781-f58a8f778eee\" (UID: \"c2ae039e-d1d3-4190-8781-f58a8f778eee\") "
Oct 30 00:48:47.051728 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:47.051668    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-52650f/simulator-w0-52650f-kserve-58b64766c-94qxx" podUID="659c1425-5b67-4d01-bfc2-f1ba56b9a59b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:48:47.124031 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:47.100646    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2ae039e-d1d3-4190-8781-f58a8f778eee-maas-api-openshift-ca" pod "c2ae039e-d1d3-4190-8781-f58a8f778eee" (UID: "c2ae039e-d1d3-4190-8781-f58a8f778eee"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:48:47.184510 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:47.154410    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2ae039e-d1d3-4190-8781-f58a8f778eee-maas-api-openshift-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:48:47.211433 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:47.184930    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2ae039e-d1d3-4190-8781-f58a8f778eee-kube-api-access-7vlqf" pod "c2ae039e-d1d3-4190-8781-f58a8f778eee" (UID: "c2ae039e-d1d3-4190-8781-f58a8f778eee"). InnerVolumeSpecName "kube-api-access-7vlqf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:48:47.302596 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:47.302546    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-29be73/simulator-w1-29be73-kserve-9958d76db-fks2t" podUID="fbb3c806-8967-4d39-84c6-9dc022fc5cae" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:48:47.360679 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:47.360625    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-29be73/premium-w1-29be73-kserve-7b777cfc8c-7kgp5" podUID="e6fabbdd-86ca-4cbf-85be-ef250575a355" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:48:47.361717 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:47.361691    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-29be73/premium-w1-29be73-kserve-7b777cfc8c-7kgp5"
Oct 30 00:48:47.367416 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:47.364082    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-52650f/premium-w0-52650f-kserve-c5bf66f9d-qq5dm" podUID="3455d983-b11b-4cee-bfce-d1f399cf9679" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:48:47.381396 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:47.374227    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-52650f/distinct-w0-52650f-kserve-d85899b5-vlhps" podUID="ea1850d9-abba-4977-a540-f6e0eaefb17d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:48:47.381396 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:47.375529    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cefb27/simulator-w3-cefb27-kserve-6b669c9d9f-zx2vm" podUID="7121e81e-f903-42c0-a7a4-bc18d49194e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/ready\": context deadline exceeded"
Oct 30 00:48:47.381396 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:47.376091    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1a11cf/distinct-w2-1a11cf-kserve-854cf4cdf8-cddw6" podUID="32a03478-46eb-4363-97b6-51a32c4099c7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded"
Oct 30 00:48:47.381396 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:47.376636    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-021d6d/simulator-w6-021d6d-kserve-6cc97fdf86-dqtpw" podUID="50586f76-4f3b-4e7a-83ac-6c573faee415" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": context deadline exceeded"
Oct 30 00:48:47.381396 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:47.377731    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-52650f/distinct-w0-52650f-kserve-d85899b5-vlhps"
Oct 30 00:48:47.414055 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:47.401410    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7vlqf\" (UniqueName: \"kubernetes.io/projected/c2ae039e-d1d3-4190-8781-f58a8f778eee-kube-api-access-7vlqf\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:48:47.416543 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:47.416504    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bedf5b47-7qtsj" event={"ID":"c2ae039e-d1d3-4190-8781-f58a8f778eee","Type":"ContainerDied","Data":"6ecbf2f922bcdf7539fd459e71ed0558d328004db1a43de1bb474cd0733a51c8"}
Oct 30 00:48:47.416775 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:47.416756    5002 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6ecbf2f922bcdf7539fd459e71ed0558d328004db1a43de1bb474cd0733a51c8"
Oct 30 00:48:47.417100 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:47.417082    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bedf5b47-7qtsj"
Oct 30 00:48:47.418108 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:47.418084    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca5363/distinct-2-w4-ca5363-kserve-576576f9b5-xkkqc" podUID="5de8bebf-cfad-49da-9a98-08aedde4442c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:48:47.498590 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:47.498479    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6f4fba/simulator-w5-6f4fba-kserve-5bf8dfddd5-wcwzh" podUID="3a2c2f8f-649c-4ab6-af98-5d87df0e511f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:48:47.548885 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:47.548840    5002 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-cefb27/simulator-w3-cefb27-kserve-6b669c9d9f-zx2vm" podUID="7121e81e-f903-42c0-a7a4-bc18d49194e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:48:47.572773 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:47.572742    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-021d6d/simulator-w6-021d6d-kserve-6cc97fdf86-dqtpw"
Oct 30 00:48:47.613441 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:47.604052    5002 patch_prober.go:29] interesting pod/e2e-worker-w6-021d6d-openshift-default-65c58b6947-x8wgq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 00:48:47.633396 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:47.633335    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-x8wgq" podUID="c534def0-b8f3-4be1-8f47-72bffc1234e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:48:47.896828 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:47.896728    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-29be73/premium-w1-29be73-kserve-7b777cfc8c-7kgp5"
Oct 30 00:48:47.897148 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:47.897134    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-cefb27/simulator-w3-cefb27-kserve-6b669c9d9f-zx2vm"
Oct 30 00:48:47.927402 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:47.917774    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2ae039e-d1d3-4190-8781-f58a8f778eee-maas-api-xks-ca" pod "c2ae039e-d1d3-4190-8781-f58a8f778eee" (UID: "c2ae039e-d1d3-4190-8781-f58a8f778eee"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:48:47.951966 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:47.951932    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ca5363/distinct-2-w4-ca5363-kserve-576576f9b5-xkkqc"
Oct 30 00:48:47.953602 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:47.953537    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-52650f/distinct-w0-52650f-kserve-d85899b5-vlhps"
Oct 30 00:48:47.953864 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:47.953835    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1a11cf/distinct-2-w2-1a11cf-kserve-758bb6dd57-j5f2k" podUID="f2d66dea-9f0b-43f3-a774-bb065e88f378" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded"
Oct 30 00:48:47.978071 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:47.976271    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-1a11cf/distinct-2-w2-1a11cf-kserve-758bb6dd57-j5f2k"
Oct 30 00:48:47.996126 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:47.996097    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2ae039e-d1d3-4190-8781-f58a8f778eee-maas-api-xks-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:48:48.073900 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:48.073868    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-1a11cf/distinct-2-w2-1a11cf-kserve-758bb6dd57-j5f2k"
Oct 30 00:48:48.187678 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:48.187650    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-1a11cf/distinct-w2-1a11cf-kserve-854cf4cdf8-cddw6"
Oct 30 00:48:51.182319 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:51.182282    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-b7vtt"]
Oct 30 00:48:51.182971 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:51.182876    5002 cpu_manager.go:414] "containerMap: removing container" podUID="c2ae039e-d1d3-4190-8781-f58a8f778eee" containerName="revoke-keys"
Oct 30 00:48:51.182971 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:51.182890    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2ae039e-d1d3-4190-8781-f58a8f778eee" containerName="revoke-keys"
Oct 30 00:48:51.182971 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:51.182908    5002 cpu_manager.go:414] "containerMap: removing container" podUID="ba2785d1-40fc-4081-ab4b-71f251ffed23" containerName="revoke-keys"
Oct 30 00:48:51.182971 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:51.182913    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba2785d1-40fc-4081-ab4b-71f251ffed23" containerName="revoke-keys"
Oct 30 00:48:51.183192 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:51.183009    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba2785d1-40fc-4081-ab4b-71f251ffed23" containerName="revoke-keys"
Oct 30 00:48:51.183192 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:51.183024    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2ae039e-d1d3-4190-8781-f58a8f778eee" containerName="revoke-keys"
Oct 30 00:48:51.186737 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:51.186712    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-b7vtt"
Oct 30 00:48:51.221508 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:51.221477    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-b7vtt"]
Oct 30 00:48:51.252372 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:51.252319    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43b2d112-3ccc-448f-8f03-e6e15fccfa4c-istiod-ca-cert\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-b7vtt\" (UID: \"43b2d112-3ccc-448f-8f03-e6e15fccfa4c\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-b7vtt"
Oct 30 00:48:51.252628 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:51.252462    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43b2d112-3ccc-448f-8f03-e6e15fccfa4c-istio-envoy\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-b7vtt\" (UID: \"43b2d112-3ccc-448f-8f03-e6e15fccfa4c\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-b7vtt"
Oct 30 00:48:51.252628 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:51.252498    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43b2d112-3ccc-448f-8f03-e6e15fccfa4c-istio-podinfo\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-b7vtt\" (UID: \"43b2d112-3ccc-448f-8f03-e6e15fccfa4c\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-b7vtt"
Oct 30 00:48:51.252628 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:51.252591    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vp2n4\" (UniqueName: \"kubernetes.io/projected/43b2d112-3ccc-448f-8f03-e6e15fccfa4c-kube-api-access-vp2n4\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-b7vtt\" (UID: \"43b2d112-3ccc-448f-8f03-e6e15fccfa4c\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-b7vtt"
Oct 30 00:48:51.252819 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:51.252797    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43b2d112-3ccc-448f-8f03-e6e15fccfa4c-credential-socket\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-b7vtt\" (UID: \"43b2d112-3ccc-448f-8f03-e6e15fccfa4c\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-b7vtt"
Oct 30 00:48:51.252985 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:51.252955    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43b2d112-3ccc-448f-8f03-e6e15fccfa4c-istio-token\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-b7vtt\" (UID: \"43b2d112-3ccc-448f-8f03-e6e15fccfa4c\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-b7vtt"
Oct 30 00:48:51.253126 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:51.253028    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43b2d112-3ccc-448f-8f03-e6e15fccfa4c-istio-data\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-b7vtt\" (UID: \"43b2d112-3ccc-448f-8f03-e6e15fccfa4c\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-b7vtt"
Oct 30 00:48:51.253196 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:51.253124    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43b2d112-3ccc-448f-8f03-e6e15fccfa4c-workload-certs\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-b7vtt\" (UID: \"43b2d112-3ccc-448f-8f03-e6e15fccfa4c\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-b7vtt"
Oct 30 00:48:51.253196 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:51.253168    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43b2d112-3ccc-448f-8f03-e6e15fccfa4c-workload-socket\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-b7vtt\" (UID: \"43b2d112-3ccc-448f-8f03-e6e15fccfa4c\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-b7vtt"
Oct 30 00:48:51.354411 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:51.354352    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43b2d112-3ccc-448f-8f03-e6e15fccfa4c-credential-socket\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-b7vtt\" (UID: \"43b2d112-3ccc-448f-8f03-e6e15fccfa4c\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-b7vtt"
Oct 30 00:48:51.354610 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:51.354419    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43b2d112-3ccc-448f-8f03-e6e15fccfa4c-istio-token\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-b7vtt\" (UID: \"43b2d112-3ccc-448f-8f03-e6e15fccfa4c\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-b7vtt"
Oct 30 00:48:51.354610 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:51.354450    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43b2d112-3ccc-448f-8f03-e6e15fccfa4c-istio-data\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-b7vtt\" (UID: \"43b2d112-3ccc-448f-8f03-e6e15fccfa4c\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-b7vtt"
Oct 30 00:48:51.354610 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:51.354539    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43b2d112-3ccc-448f-8f03-e6e15fccfa4c-workload-certs\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-b7vtt\" (UID: \"43b2d112-3ccc-448f-8f03-e6e15fccfa4c\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-b7vtt"
Oct 30 00:48:51.354610 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:51.354581    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43b2d112-3ccc-448f-8f03-e6e15fccfa4c-workload-socket\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-b7vtt\" (UID: \"43b2d112-3ccc-448f-8f03-e6e15fccfa4c\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-b7vtt"
Oct 30 00:48:51.354840 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:51.354635    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43b2d112-3ccc-448f-8f03-e6e15fccfa4c-istiod-ca-cert\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-b7vtt\" (UID: \"43b2d112-3ccc-448f-8f03-e6e15fccfa4c\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-b7vtt"
Oct 30 00:48:51.354840 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:51.354671    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43b2d112-3ccc-448f-8f03-e6e15fccfa4c-istio-envoy\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-b7vtt\" (UID: \"43b2d112-3ccc-448f-8f03-e6e15fccfa4c\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-b7vtt"
Oct 30 00:48:51.354840 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:51.354701    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43b2d112-3ccc-448f-8f03-e6e15fccfa4c-istio-podinfo\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-b7vtt\" (UID: \"43b2d112-3ccc-448f-8f03-e6e15fccfa4c\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-b7vtt"
Oct 30 00:48:51.354840 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:51.354750    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vp2n4\" (UniqueName: \"kubernetes.io/projected/43b2d112-3ccc-448f-8f03-e6e15fccfa4c-kube-api-access-vp2n4\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-b7vtt\" (UID: \"43b2d112-3ccc-448f-8f03-e6e15fccfa4c\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-b7vtt"
Oct 30 00:48:51.355060 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:51.354984    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43b2d112-3ccc-448f-8f03-e6e15fccfa4c-workload-certs\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-b7vtt\" (UID: \"43b2d112-3ccc-448f-8f03-e6e15fccfa4c\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-b7vtt"
Oct 30 00:48:51.355471 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:51.355446    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43b2d112-3ccc-448f-8f03-e6e15fccfa4c-istio-data\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-b7vtt\" (UID: \"43b2d112-3ccc-448f-8f03-e6e15fccfa4c\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-b7vtt"
Oct 30 00:48:51.355833 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:51.355808    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43b2d112-3ccc-448f-8f03-e6e15fccfa4c-istiod-ca-cert\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-b7vtt\" (UID: \"43b2d112-3ccc-448f-8f03-e6e15fccfa4c\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-b7vtt"
Oct 30 00:48:51.356108 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:51.356085    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43b2d112-3ccc-448f-8f03-e6e15fccfa4c-workload-socket\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-b7vtt\" (UID: \"43b2d112-3ccc-448f-8f03-e6e15fccfa4c\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-b7vtt"
Oct 30 00:48:51.356247 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:51.356227    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43b2d112-3ccc-448f-8f03-e6e15fccfa4c-credential-socket\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-b7vtt\" (UID: \"43b2d112-3ccc-448f-8f03-e6e15fccfa4c\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-b7vtt"
Oct 30 00:48:51.357599 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:51.357573    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43b2d112-3ccc-448f-8f03-e6e15fccfa4c-istio-envoy\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-b7vtt\" (UID: \"43b2d112-3ccc-448f-8f03-e6e15fccfa4c\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-b7vtt"
Oct 30 00:48:51.357840 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:51.357819    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43b2d112-3ccc-448f-8f03-e6e15fccfa4c-istio-podinfo\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-b7vtt\" (UID: \"43b2d112-3ccc-448f-8f03-e6e15fccfa4c\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-b7vtt"
Oct 30 00:48:51.379609 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:51.379579    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vp2n4\" (UniqueName: \"kubernetes.io/projected/43b2d112-3ccc-448f-8f03-e6e15fccfa4c-kube-api-access-vp2n4\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-b7vtt\" (UID: \"43b2d112-3ccc-448f-8f03-e6e15fccfa4c\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-b7vtt"
Oct 30 00:48:51.381537 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:51.381509    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43b2d112-3ccc-448f-8f03-e6e15fccfa4c-istio-token\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-b7vtt\" (UID: \"43b2d112-3ccc-448f-8f03-e6e15fccfa4c\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-b7vtt"
Oct 30 00:48:51.502951 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:51.502860    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-b7vtt"
Oct 30 00:48:51.830225 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:51.829266    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-b7vtt"]
Oct 30 00:48:51.830225 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:48:51.829601    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43b2d112_3ccc_448f_8f03_e6e15fccfa4c.slice/crio-06a16911f2e8f75727e9eabe19613b5235935d82281aec19f9e10395718bb2b1 WatchSource:0}: Error finding container 06a16911f2e8f75727e9eabe19613b5235935d82281aec19f9e10395718bb2b1: Status 404 returned error can't find the container with id 06a16911f2e8f75727e9eabe19613b5235935d82281aec19f9e10395718bb2b1
Oct 30 00:48:51.832571 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:51.832528    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 00:48:51.832673 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:51.832620    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 00:48:51.832673 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:51.832663    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 00:48:52.275743 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:52.275702    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-34f9d3da-sm7z5"]
Oct 30 00:48:52.281410 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:52.281374    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34f9d3da-sm7z5"
Oct 30 00:48:52.300335 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:52.300305    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 00:48:52.300588 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:52.300556    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 00:48:52.300675 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:52.300626    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 00:48:52.302275 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:52.302239    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-34f9d3da-sm7z5"]
Oct 30 00:48:52.304756 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:52.304732    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5phsh\""
Oct 30 00:48:52.372445 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:52.370899    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/065123cf-d6c8-4282-81b6-7d4fb04c490a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-34f9d3da-sm7z5\" (UID: \"065123cf-d6c8-4282-81b6-7d4fb04c490a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34f9d3da-sm7z5"
Oct 30 00:48:52.372445 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:52.371120    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlcv8\" (UniqueName: \"kubernetes.io/projected/065123cf-d6c8-4282-81b6-7d4fb04c490a-kube-api-access-hlcv8\") pod \"maas-api-revoke-sub-34f9d3da-sm7z5\" (UID: \"065123cf-d6c8-4282-81b6-7d4fb04c490a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34f9d3da-sm7z5"
Oct 30 00:48:52.372445 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:52.371197    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/065123cf-d6c8-4282-81b6-7d4fb04c490a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-34f9d3da-sm7z5\" (UID: \"065123cf-d6c8-4282-81b6-7d4fb04c490a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34f9d3da-sm7z5"
Oct 30 00:48:52.475598 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:52.475433    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlcv8\" (UniqueName: \"kubernetes.io/projected/065123cf-d6c8-4282-81b6-7d4fb04c490a-kube-api-access-hlcv8\") pod \"maas-api-revoke-sub-34f9d3da-sm7z5\" (UID: \"065123cf-d6c8-4282-81b6-7d4fb04c490a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34f9d3da-sm7z5"
Oct 30 00:48:52.475598 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:52.475538    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/065123cf-d6c8-4282-81b6-7d4fb04c490a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-34f9d3da-sm7z5\" (UID: \"065123cf-d6c8-4282-81b6-7d4fb04c490a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34f9d3da-sm7z5"
Oct 30 00:48:52.475892 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:52.475627    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/065123cf-d6c8-4282-81b6-7d4fb04c490a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-34f9d3da-sm7z5\" (UID: \"065123cf-d6c8-4282-81b6-7d4fb04c490a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34f9d3da-sm7z5"
Oct 30 00:48:52.476720 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:52.476399    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/065123cf-d6c8-4282-81b6-7d4fb04c490a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-34f9d3da-sm7z5\" (UID: \"065123cf-d6c8-4282-81b6-7d4fb04c490a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34f9d3da-sm7z5"
Oct 30 00:48:52.479827 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:52.479802    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/065123cf-d6c8-4282-81b6-7d4fb04c490a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-34f9d3da-sm7z5\" (UID: \"065123cf-d6c8-4282-81b6-7d4fb04c490a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34f9d3da-sm7z5"
Oct 30 00:48:52.480938 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:52.480909    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-b7vtt" event={"ID":"43b2d112-3ccc-448f-8f03-e6e15fccfa4c","Type":"ContainerStarted","Data":"ce3946fa6c193d416d428d807339ac323f37e7f1d1aabaa78c3c38a8524ae8dd"}
Oct 30 00:48:52.481045 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:52.480974    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-b7vtt" event={"ID":"43b2d112-3ccc-448f-8f03-e6e15fccfa4c","Type":"ContainerStarted","Data":"06a16911f2e8f75727e9eabe19613b5235935d82281aec19f9e10395718bb2b1"}
Oct 30 00:48:52.501218 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:52.501182    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlcv8\" (UniqueName: \"kubernetes.io/projected/065123cf-d6c8-4282-81b6-7d4fb04c490a-kube-api-access-hlcv8\") pod \"maas-api-revoke-sub-34f9d3da-sm7z5\" (UID: \"065123cf-d6c8-4282-81b6-7d4fb04c490a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34f9d3da-sm7z5"
Oct 30 00:48:52.503363 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:52.503339    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-b7vtt"
Oct 30 00:48:52.505124 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:52.505100    5002 patch_prober.go:29] interesting pod/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-b7vtt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Oct 30 00:48:52.505232 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:52.505175    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-b7vtt" podUID="43b2d112-3ccc-448f-8f03-e6e15fccfa4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Oct 30 00:48:52.529069 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:52.528982    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-b7vtt" podStartSLOduration=1.5289701039999999 podStartE2EDuration="1.528970104s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:48:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:48:52.52777582 +0000 UTC m=+1028.144208701" watchObservedRunningTime="2026-10-30 00:48:52.528970104 +0000 UTC m=+1028.145402982"
Oct 30 00:48:52.600508 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:52.600473    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34f9d3da-sm7z5"
Oct 30 00:48:52.934850 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:48:52.934823    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod065123cf_d6c8_4282_81b6_7d4fb04c490a.slice/crio-8a3e9760447e7f171aaac58c485464c6ee1fb8210000fdc7a8ec1f94992fe263 WatchSource:0}: Error finding container 8a3e9760447e7f171aaac58c485464c6ee1fb8210000fdc7a8ec1f94992fe263: Status 404 returned error can't find the container with id 8a3e9760447e7f171aaac58c485464c6ee1fb8210000fdc7a8ec1f94992fe263
Oct 30 00:48:52.936274 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:52.936249    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-34f9d3da-sm7z5"]
Oct 30 00:48:53.495614 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:53.495506    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34f9d3da-sm7z5" event={"ID":"065123cf-d6c8-4282-81b6-7d4fb04c490a","Type":"ContainerStarted","Data":"ce0bf28697e53c08afa8e664a0b806a1f15a282be73cfbd3c6817f1d8ea07722"}
Oct 30 00:48:53.495614 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:53.495581    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34f9d3da-sm7z5" event={"ID":"065123cf-d6c8-4282-81b6-7d4fb04c490a","Type":"ContainerStarted","Data":"8a3e9760447e7f171aaac58c485464c6ee1fb8210000fdc7a8ec1f94992fe263"}
Oct 30 00:48:53.503263 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:53.503236    5002 patch_prober.go:29] interesting pod/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-b7vtt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Oct 30 00:48:53.503408 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:53.503312    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-b7vtt" podUID="43b2d112-3ccc-448f-8f03-e6e15fccfa4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Oct 30 00:48:53.542577 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:53.542516    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34f9d3da-sm7z5" podStartSLOduration=1.542498082 podStartE2EDuration="1.542498082s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:48:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:48:53.539012596 +0000 UTC m=+1029.155445478" watchObservedRunningTime="2026-10-30 00:48:53.542498082 +0000 UTC m=+1029.158930982"
Oct 30 00:48:54.504332 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:54.504205    5002 patch_prober.go:29] interesting pod/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-b7vtt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Oct 30 00:48:54.504332 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:54.504251    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-b7vtt" podUID="43b2d112-3ccc-448f-8f03-e6e15fccfa4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Oct 30 00:48:54.506067 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:54.506038    5002 generic.go:356] "Generic (PLEG): container finished" podID="065123cf-d6c8-4282-81b6-7d4fb04c490a" containerID="ce0bf28697e53c08afa8e664a0b806a1f15a282be73cfbd3c6817f1d8ea07722" exitCode=0
Oct 30 00:48:54.506345 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:54.506317    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34f9d3da-sm7z5" event={"ID":"065123cf-d6c8-4282-81b6-7d4fb04c490a","Type":"ContainerDied","Data":"ce0bf28697e53c08afa8e664a0b806a1f15a282be73cfbd3c6817f1d8ea07722"}
Oct 30 00:48:55.503767 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:55.503728    5002 patch_prober.go:29] interesting pod/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-b7vtt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Oct 30 00:48:55.503940 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:55.503790    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-b7vtt" podUID="43b2d112-3ccc-448f-8f03-e6e15fccfa4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Oct 30 00:48:55.790932 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:55.790832    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34f9d3da-sm7z5"
Oct 30 00:48:55.831969 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:55.831936    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/065123cf-d6c8-4282-81b6-7d4fb04c490a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/065123cf-d6c8-4282-81b6-7d4fb04c490a-maas-api-xks-ca\") pod \"065123cf-d6c8-4282-81b6-7d4fb04c490a\" (UID: \"065123cf-d6c8-4282-81b6-7d4fb04c490a\") "
Oct 30 00:48:55.832189 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:55.831998    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/065123cf-d6c8-4282-81b6-7d4fb04c490a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/065123cf-d6c8-4282-81b6-7d4fb04c490a-maas-api-openshift-ca\") pod \"065123cf-d6c8-4282-81b6-7d4fb04c490a\" (UID: \"065123cf-d6c8-4282-81b6-7d4fb04c490a\") "
Oct 30 00:48:55.832189 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:55.832024    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/065123cf-d6c8-4282-81b6-7d4fb04c490a-kube-api-access-hlcv8\" (UniqueName: \"kubernetes.io/projected/065123cf-d6c8-4282-81b6-7d4fb04c490a-kube-api-access-hlcv8\") pod \"065123cf-d6c8-4282-81b6-7d4fb04c490a\" (UID: \"065123cf-d6c8-4282-81b6-7d4fb04c490a\") "
Oct 30 00:48:55.832445 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:55.832418    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/065123cf-d6c8-4282-81b6-7d4fb04c490a-maas-api-openshift-ca" pod "065123cf-d6c8-4282-81b6-7d4fb04c490a" (UID: "065123cf-d6c8-4282-81b6-7d4fb04c490a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:48:55.833420 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:55.832683    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/065123cf-d6c8-4282-81b6-7d4fb04c490a-maas-api-openshift-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:48:55.834612 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:55.834586    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/065123cf-d6c8-4282-81b6-7d4fb04c490a-maas-api-xks-ca" pod "065123cf-d6c8-4282-81b6-7d4fb04c490a" (UID: "065123cf-d6c8-4282-81b6-7d4fb04c490a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:48:55.835560 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:55.835531    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/065123cf-d6c8-4282-81b6-7d4fb04c490a-kube-api-access-hlcv8" pod "065123cf-d6c8-4282-81b6-7d4fb04c490a" (UID: "065123cf-d6c8-4282-81b6-7d4fb04c490a"). InnerVolumeSpecName "kube-api-access-hlcv8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:48:55.933992 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:55.933950    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/065123cf-d6c8-4282-81b6-7d4fb04c490a-maas-api-xks-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:48:55.933992 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:55.933987    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hlcv8\" (UniqueName: \"kubernetes.io/projected/065123cf-d6c8-4282-81b6-7d4fb04c490a-kube-api-access-hlcv8\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:48:56.503775 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:56.503710    5002 patch_prober.go:29] interesting pod/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-b7vtt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Oct 30 00:48:56.504103 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:56.503786    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-b7vtt" podUID="43b2d112-3ccc-448f-8f03-e6e15fccfa4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Oct 30 00:48:56.522731 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:56.522690    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34f9d3da-sm7z5" event={"ID":"065123cf-d6c8-4282-81b6-7d4fb04c490a","Type":"ContainerDied","Data":"8a3e9760447e7f171aaac58c485464c6ee1fb8210000fdc7a8ec1f94992fe263"}
Oct 30 00:48:56.522731 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:56.522716    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34f9d3da-sm7z5"
Oct 30 00:48:56.522731 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:56.522738    5002 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8a3e9760447e7f171aaac58c485464c6ee1fb8210000fdc7a8ec1f94992fe263"
Oct 30 00:48:57.503744 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:57.503709    5002 patch_prober.go:29] interesting pod/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-b7vtt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Oct 30 00:48:57.504243 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:57.503770    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-b7vtt" podUID="43b2d112-3ccc-448f-8f03-e6e15fccfa4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Oct 30 00:48:58.505449 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:58.505373    5002 patch_prober.go:29] interesting pod/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-b7vtt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Oct 30 00:48:58.505933 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:58.505490    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-b7vtt" podUID="43b2d112-3ccc-448f-8f03-e6e15fccfa4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Oct 30 00:48:59.508938 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:59.508907    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-b7vtt"
Oct 30 00:48:59.509343 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:59.509187    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-b7vtt"
Oct 30 00:48:59.509918 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:48:59.509900    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-b7vtt"
Oct 30 00:49:07.304534 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:07.304478    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1a11cf/distinct-w2-1a11cf-kserve-854cf4cdf8-cddw6" podUID="32a03478-46eb-4363-97b6-51a32c4099c7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:49:07.345591 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:07.345530    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-52650f/premium-w0-52650f-kserve-c5bf66f9d-qq5dm" podUID="3455d983-b11b-4cee-bfce-d1f399cf9679" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:49:07.382533 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:07.382474    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-021d6d/simulator-w6-021d6d-kserve-6cc97fdf86-dqtpw" podUID="50586f76-4f3b-4e7a-83ac-6c573faee415" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:49:07.423538 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:07.423483    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca5363/distinct-2-w4-ca5363-kserve-576576f9b5-xkkqc" podUID="5de8bebf-cfad-49da-9a98-08aedde4442c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Oct 30 00:49:07.490635 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:07.490567    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6f4fba/simulator-w5-6f4fba-kserve-5bf8dfddd5-wcwzh" podUID="3a2c2f8f-649c-4ab6-af98-5d87df0e511f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/ready\": context deadline exceeded"
Oct 30 00:49:07.530700 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:07.530655    5002 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-cefb27/simulator-w3-cefb27-kserve-6b669c9d9f-zx2vm" podUID="7121e81e-f903-42c0-a7a4-bc18d49194e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/health\": context deadline exceeded"
Oct 30 00:49:07.910583 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:07.910543    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-29be73/premium-w1-29be73-kserve-7b777cfc8c-7kgp5" podUID="e6fabbdd-86ca-4cbf-85be-ef250575a355" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": context deadline exceeded"
Oct 30 00:49:07.957537 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:07.957491    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-52650f/distinct-w0-52650f-kserve-d85899b5-vlhps" podUID="ea1850d9-abba-4977-a540-f6e0eaefb17d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded"
Oct 30 00:49:08.433918 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:08.431515    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca5363/distinct-2-w4-ca5363-kserve-576576f9b5-xkkqc" podUID="5de8bebf-cfad-49da-9a98-08aedde4442c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:49:08.433918 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:08.432330    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ca5363/distinct-2-w4-ca5363-kserve-576576f9b5-xkkqc"
Oct 30 00:49:08.510505 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:08.510461    5002 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-29be73/premium-w1-29be73-kserve-7b777cfc8c-7kgp5" podUID="e6fabbdd-86ca-4cbf-85be-ef250575a355" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:49:09.027720 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:09.027677    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-29be73/premium-w1-29be73-kserve-7b777cfc8c-7kgp5" podUID="e6fabbdd-86ca-4cbf-85be-ef250575a355" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": context deadline exceeded"
Oct 30 00:49:09.031016 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:09.030987    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-29be73/premium-w1-29be73-kserve-7b777cfc8c-7kgp5"
Oct 30 00:49:09.037401 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:09.034018    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-52650f/simulator-w0-52650f-kserve-58b64766c-94qxx" podUID="659c1425-5b67-4d01-bfc2-f1ba56b9a59b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:49:09.037401 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:09.034756    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-29be73/simulator-w1-29be73-kserve-9958d76db-fks2t" podUID="fbb3c806-8967-4d39-84c6-9dc022fc5cae" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/ready\": context deadline exceeded"
Oct 30 00:49:09.111577 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:09.111507    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1a11cf/distinct-2-w2-1a11cf-kserve-758bb6dd57-j5f2k" podUID="f2d66dea-9f0b-43f3-a774-bb065e88f378" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded"
Oct 30 00:49:09.314504 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:09.300479    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1a11cf/distinct-w2-1a11cf-kserve-854cf4cdf8-cddw6" podUID="32a03478-46eb-4363-97b6-51a32c4099c7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded"
Oct 30 00:49:09.342846 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:09.342789    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-52650f/premium-w0-52650f-kserve-c5bf66f9d-qq5dm" podUID="3455d983-b11b-4cee-bfce-d1f399cf9679" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:49:09.389547 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:09.389502    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-021d6d/simulator-w6-021d6d-kserve-6cc97fdf86-dqtpw" podUID="50586f76-4f3b-4e7a-83ac-6c573faee415" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:49:09.441409 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:09.431701    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca5363/distinct-2-w4-ca5363-kserve-576576f9b5-xkkqc" podUID="5de8bebf-cfad-49da-9a98-08aedde4442c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:49:09.500628 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:09.500549    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6f4fba/simulator-w5-6f4fba-kserve-5bf8dfddd5-wcwzh" podUID="3a2c2f8f-649c-4ab6-af98-5d87df0e511f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:49:10.022160 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:10.022112    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cefb27/simulator-w3-cefb27-kserve-6b669c9d9f-zx2vm" podUID="7121e81e-f903-42c0-a7a4-bc18d49194e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:49:10.047406 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:10.037933    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-29be73/premium-w1-29be73-kserve-7b777cfc8c-7kgp5" podUID="e6fabbdd-86ca-4cbf-85be-ef250575a355" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": context deadline exceeded"
Oct 30 00:49:10.047406 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:10.046529    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-29be73/simulator-w1-29be73-kserve-9958d76db-fks2t" podUID="fbb3c806-8967-4d39-84c6-9dc022fc5cae" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:49:10.047406 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:10.046843    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-52650f/simulator-w0-52650f-kserve-58b64766c-94qxx" podUID="659c1425-5b67-4d01-bfc2-f1ba56b9a59b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": context deadline exceeded"
Oct 30 00:49:10.047406 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:10.047118    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-29be73/simulator-w1-29be73-kserve-9958d76db-fks2t"
Oct 30 00:49:10.047406 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:10.047175    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-52650f/simulator-w0-52650f-kserve-58b64766c-94qxx"
Oct 30 00:49:10.086553 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:10.086511    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-52650f/distinct-w0-52650f-kserve-d85899b5-vlhps" podUID="ea1850d9-abba-4977-a540-f6e0eaefb17d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:49:10.131165 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:10.128462    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1a11cf/distinct-2-w2-1a11cf-kserve-758bb6dd57-j5f2k" podUID="f2d66dea-9f0b-43f3-a774-bb065e88f378" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded"
Oct 30 00:49:10.131165 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:10.128930    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-1a11cf/distinct-2-w2-1a11cf-kserve-758bb6dd57-j5f2k"
Oct 30 00:49:10.291471 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:10.274472    5002 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-ca5363/distinct-2-w4-ca5363-kserve-576576f9b5-xkkqc" podUID="5de8bebf-cfad-49da-9a98-08aedde4442c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:49:10.462405 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:10.427505    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1a11cf/distinct-w2-1a11cf-kserve-854cf4cdf8-cddw6" podUID="32a03478-46eb-4363-97b6-51a32c4099c7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded"
Oct 30 00:49:10.463555 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:10.436687    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-52650f/premium-w0-52650f-kserve-c5bf66f9d-qq5dm" podUID="3455d983-b11b-4cee-bfce-d1f399cf9679" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:49:10.490457 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:10.463405    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca5363/distinct-2-w4-ca5363-kserve-576576f9b5-xkkqc" podUID="5de8bebf-cfad-49da-9a98-08aedde4442c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:49:10.548462 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:10.503117    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-1a11cf/distinct-w2-1a11cf-kserve-854cf4cdf8-cddw6"
Oct 30 00:49:10.548462 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:10.519662    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-52650f/premium-w0-52650f-kserve-c5bf66f9d-qq5dm"
Oct 30 00:49:10.548462 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:10.509369    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6f4fba/simulator-w5-6f4fba-kserve-5bf8dfddd5-wcwzh" podUID="3a2c2f8f-649c-4ab6-af98-5d87df0e511f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/ready\": context deadline exceeded"
Oct 30 00:49:10.802465 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:10.783354    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-6f4fba/simulator-w5-6f4fba-kserve-5bf8dfddd5-wcwzh"
Oct 30 00:49:10.880405 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:10.878767    5002 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-1a11cf/distinct-2-w2-1a11cf-kserve-758bb6dd57-j5f2k" podUID="f2d66dea-9f0b-43f3-a774-bb065e88f378" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:49:10.977791 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:10.977756    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-29be73/premium-w1-29be73-kserve-7b777cfc8c-7kgp5"
Oct 30 00:49:10.978093 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:10.978072    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-6f4fba/simulator-w5-6f4fba-kserve-5bf8dfddd5-wcwzh"
Oct 30 00:49:10.982501 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:10.982479    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-1a11cf/distinct-2-w2-1a11cf-kserve-758bb6dd57-j5f2k"
Oct 30 00:49:10.982698 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:10.982684    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-52650f/premium-w0-52650f-kserve-c5bf66f9d-qq5dm"
Oct 30 00:49:11.026648 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:11.012138    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ca5363/distinct-2-w4-ca5363-kserve-576576f9b5-xkkqc"
Oct 30 00:49:11.026648 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:11.025310    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-29be73/simulator-w1-29be73-kserve-9958d76db-fks2t"
Oct 30 00:49:11.031915 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:11.031885    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-52650f/simulator-w0-52650f-kserve-58b64766c-94qxx"
Oct 30 00:49:11.057824 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:11.057060    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-1a11cf/distinct-w2-1a11cf-kserve-854cf4cdf8-cddw6"
Oct 30 00:49:12.815648 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:12.815606    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j5x7c"]
Oct 30 00:49:12.818357 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:12.818330    5002 cpu_manager.go:414] "containerMap: removing container" podUID="065123cf-d6c8-4282-81b6-7d4fb04c490a" containerName="revoke-keys"
Oct 30 00:49:12.818357 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:12.818354    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="065123cf-d6c8-4282-81b6-7d4fb04c490a" containerName="revoke-keys"
Oct 30 00:49:12.818591 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:12.818545    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="065123cf-d6c8-4282-81b6-7d4fb04c490a" containerName="revoke-keys"
Oct 30 00:49:12.824448 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:12.824417    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j5x7c"
Oct 30 00:49:12.845544 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:12.845512    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j5x7c"]
Oct 30 00:49:12.858727 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:12.851774    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-k4cpq\""
Oct 30 00:49:12.858727 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:12.852560    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Oct 30 00:49:12.858727 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:12.852618    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Oct 30 00:49:12.858727 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:12.852689    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Oct 30 00:49:12.925569 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:12.925452    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/48f3cd2a-244d-4849-888b-e4b9abd0ce9c-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-j5x7c\" (UID: \"48f3cd2a-244d-4849-888b-e4b9abd0ce9c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j5x7c"
Oct 30 00:49:12.947536 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:12.947499    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8z4n\" (UniqueName: \"kubernetes.io/projected/48f3cd2a-244d-4849-888b-e4b9abd0ce9c-kube-api-access-d8z4n\") pod \"istiod-openshift-gateway-7ddd9cfdc8-j5x7c\" (UID: \"48f3cd2a-244d-4849-888b-e4b9abd0ce9c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j5x7c"
Oct 30 00:49:12.947809 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:12.947784    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/48f3cd2a-244d-4849-888b-e4b9abd0ce9c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-j5x7c\" (UID: \"48f3cd2a-244d-4849-888b-e4b9abd0ce9c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j5x7c"
Oct 30 00:49:12.948063 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:12.948044    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/48f3cd2a-244d-4849-888b-e4b9abd0ce9c-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-j5x7c\" (UID: \"48f3cd2a-244d-4849-888b-e4b9abd0ce9c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j5x7c"
Oct 30 00:49:12.948261 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:12.948244    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48f3cd2a-244d-4849-888b-e4b9abd0ce9c-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-j5x7c\" (UID: \"48f3cd2a-244d-4849-888b-e4b9abd0ce9c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j5x7c"
Oct 30 00:49:12.948443 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:12.948366    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/48f3cd2a-244d-4849-888b-e4b9abd0ce9c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-j5x7c\" (UID: \"48f3cd2a-244d-4849-888b-e4b9abd0ce9c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j5x7c"
Oct 30 00:49:12.948583 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:12.948567    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/48f3cd2a-244d-4849-888b-e4b9abd0ce9c-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-j5x7c\" (UID: \"48f3cd2a-244d-4849-888b-e4b9abd0ce9c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j5x7c"
Oct 30 00:49:12.975400 ip-10-0-128-7 kubenswrapper[5002]: E1030 00:49:12.971894    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbjdq" containerName="extractor"
Oct 30 00:49:13.051401 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:13.050257    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/48f3cd2a-244d-4849-888b-e4b9abd0ce9c-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-j5x7c\" (UID: \"48f3cd2a-244d-4849-888b-e4b9abd0ce9c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j5x7c"
Oct 30 00:49:13.051401 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:13.050364    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48f3cd2a-244d-4849-888b-e4b9abd0ce9c-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-j5x7c\" (UID: \"48f3cd2a-244d-4849-888b-e4b9abd0ce9c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j5x7c"
Oct 30 00:49:13.051401 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:13.050421    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/48f3cd2a-244d-4849-888b-e4b9abd0ce9c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-j5x7c\" (UID: \"48f3cd2a-244d-4849-888b-e4b9abd0ce9c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j5x7c"
Oct 30 00:49:13.051401 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:13.050453    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/48f3cd2a-244d-4849-888b-e4b9abd0ce9c-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-j5x7c\" (UID: \"48f3cd2a-244d-4849-888b-e4b9abd0ce9c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j5x7c"
Oct 30 00:49:13.051401 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:13.050507    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/48f3cd2a-244d-4849-888b-e4b9abd0ce9c-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-j5x7c\" (UID: \"48f3cd2a-244d-4849-888b-e4b9abd0ce9c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j5x7c"
Oct 30 00:49:13.051401 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:13.050611    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8z4n\" (UniqueName: \"kubernetes.io/projected/48f3cd2a-244d-4849-888b-e4b9abd0ce9c-kube-api-access-d8z4n\") pod \"istiod-openshift-gateway-7ddd9cfdc8-j5x7c\" (UID: \"48f3cd2a-244d-4849-888b-e4b9abd0ce9c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j5x7c"
Oct 30 00:49:13.051401 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:13.050641    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/48f3cd2a-244d-4849-888b-e4b9abd0ce9c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-j5x7c\" (UID: \"48f3cd2a-244d-4849-888b-e4b9abd0ce9c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j5x7c"
Oct 30 00:49:13.067040 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:13.065275    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/48f3cd2a-244d-4849-888b-e4b9abd0ce9c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-j5x7c\" (UID: \"48f3cd2a-244d-4849-888b-e4b9abd0ce9c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j5x7c"
Oct 30 00:49:13.102949 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:13.077165    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/48f3cd2a-244d-4849-888b-e4b9abd0ce9c-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-j5x7c\" (UID: \"48f3cd2a-244d-4849-888b-e4b9abd0ce9c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j5x7c"
Oct 30 00:49:13.102949 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:13.090017    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/48f3cd2a-244d-4849-888b-e4b9abd0ce9c-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-j5x7c\" (UID: \"48f3cd2a-244d-4849-888b-e4b9abd0ce9c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j5x7c"
Oct 30 00:49:13.104732 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:13.104254    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/48f3cd2a-244d-4849-888b-e4b9abd0ce9c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-j5x7c\" (UID: \"48f3cd2a-244d-4849-888b-e4b9abd0ce9c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j5x7c"
Oct 30 00:49:13.115418 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:13.110726    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48f3cd2a-244d-4849-888b-e4b9abd0ce9c-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-j5x7c\" (UID: \"48f3cd2a-244d-4849-888b-e4b9abd0ce9c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j5x7c"
Oct 30 00:49:13.117186 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:13.116361    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/48f3cd2a-244d-4849-888b-e4b9abd0ce9c-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-j5x7c\" (UID: \"48f3cd2a-244d-4849-888b-e4b9abd0ce9c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j5x7c"
Oct 30 00:49:13.119357 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:13.117550    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8z4n\" (UniqueName: \"kubernetes.io/projected/48f3cd2a-244d-4849-888b-e4b9abd0ce9c-kube-api-access-d8z4n\") pod \"istiod-openshift-gateway-7ddd9cfdc8-j5x7c\" (UID: \"48f3cd2a-244d-4849-888b-e4b9abd0ce9c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j5x7c"
Oct 30 00:49:13.187403 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:13.186830    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j5x7c"
Oct 30 00:49:13.513568 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:49:13.513537    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48f3cd2a_244d_4849_888b_e4b9abd0ce9c.slice/crio-21bbb170c5858234e2f1a9497722ea60d249074ebb4e42d88a8c179e480f6519 WatchSource:0}: Error finding container 21bbb170c5858234e2f1a9497722ea60d249074ebb4e42d88a8c179e480f6519: Status 404 returned error can't find the container with id 21bbb170c5858234e2f1a9497722ea60d249074ebb4e42d88a8c179e480f6519
Oct 30 00:49:13.514165 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:13.514141    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j5x7c"]
Oct 30 00:49:13.516182 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:13.516165    5002 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 00:49:14.335739 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:14.335700    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j5x7c" event={"ID":"48f3cd2a-244d-4849-888b-e4b9abd0ce9c","Type":"ContainerStarted","Data":"21bbb170c5858234e2f1a9497722ea60d249074ebb4e42d88a8c179e480f6519"}
Oct 30 00:49:21.311547 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:21.311509    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 00:49:21.311970 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:21.311586    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 00:49:21.427753 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:21.427705    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j5x7c" event={"ID":"48f3cd2a-244d-4849-888b-e4b9abd0ce9c","Type":"ContainerStarted","Data":"e89bb890f9827e0d0829ffcc2d3e13dd949cf5c8f1c782aeda11d7ca662cf402"}
Oct 30 00:49:21.427924 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:21.427801    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j5x7c"
Oct 30 00:49:21.497680 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:21.497618    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j5x7c" podStartSLOduration=1.703821151 podStartE2EDuration="9.497598432s" totalImagesPullingTime="7.793777281s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:49:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:49:21.48336902 +0000 UTC m=+1057.099801901" watchObservedRunningTime="2026-10-30 00:49:21.497598432 +0000 UTC m=+1057.114031314"
Oct 30 00:49:22.438464 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:22.438427    5002 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-j5x7c container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 00:49:22.438946 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:22.438478    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j5x7c" podUID="48f3cd2a-244d-4849-888b-e4b9abd0ce9c" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 00:49:22.982145 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:22.982112    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7d8z4"]
Oct 30 00:49:25.439605 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:25.438085    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j5x7c"
Oct 30 00:49:25.969539 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:25.969483    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-52650f/distinct-w0-52650f-kserve-d85899b5-vlhps" podUID="ea1850d9-abba-4977-a540-f6e0eaefb17d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:49:26.008858 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:26.008812    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1a11cf/distinct-2-w2-1a11cf-kserve-758bb6dd57-j5f2k" podUID="f2d66dea-9f0b-43f3-a774-bb065e88f378" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:49:26.010088 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:26.009094    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6f4fba/simulator-w5-6f4fba-kserve-5bf8dfddd5-wcwzh" podUID="3a2c2f8f-649c-4ab6-af98-5d87df0e511f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:49:26.044404 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:26.035051    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-52650f/simulator-w0-52650f-kserve-58b64766c-94qxx" podUID="659c1425-5b67-4d01-bfc2-f1ba56b9a59b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:49:26.088405 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:26.084525    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1a11cf/distinct-w2-1a11cf-kserve-854cf4cdf8-cddw6" podUID="32a03478-46eb-4363-97b6-51a32c4099c7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded"
Oct 30 00:49:26.591912 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:26.591865    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca5363/distinct-2-w4-ca5363-kserve-576576f9b5-xkkqc" podUID="5de8bebf-cfad-49da-9a98-08aedde4442c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:49:26.897469 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:26.886504    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cefb27/simulator-w3-cefb27-kserve-6b669c9d9f-zx2vm" podUID="7121e81e-f903-42c0-a7a4-bc18d49194e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:49:27.016416 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:27.014469    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1a11cf/distinct-2-w2-1a11cf-kserve-758bb6dd57-j5f2k" podUID="f2d66dea-9f0b-43f3-a774-bb065e88f378" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:49:27.016416 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:27.015206    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-1a11cf/distinct-2-w2-1a11cf-kserve-758bb6dd57-j5f2k"
Oct 30 00:49:27.016416 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:27.016349    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6f4fba/simulator-w5-6f4fba-kserve-5bf8dfddd5-wcwzh" podUID="3a2c2f8f-649c-4ab6-af98-5d87df0e511f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:49:27.016731 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:27.016680    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-6f4fba/simulator-w5-6f4fba-kserve-5bf8dfddd5-wcwzh"
Oct 30 00:49:27.298325 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:27.296574    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-6f4fba/simulator-w5-6f4fba-kserve-5bf8dfddd5-wcwzh"
Oct 30 00:49:27.304148 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:27.303983    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-1a11cf/distinct-2-w2-1a11cf-kserve-758bb6dd57-j5f2k"
Oct 30 00:49:42.862898 ip-10-0-128-7 kubenswrapper[5002]: E1030 00:49:42.862849    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbjdq" containerName="extractor"
Oct 30 00:49:50.719405 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:50.719220    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-j9nfb"]
Oct 30 00:49:50.750625 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:50.750591    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-5tcsp"]
Oct 30 00:49:50.753727 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:50.753700    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-j9nfb"
Oct 30 00:49:50.762686 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:50.762657    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-5tcsp"
Oct 30 00:49:50.792310 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:50.792270    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-j9nfb"]
Oct 30 00:49:50.943419 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:50.919357    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62422ac3-d47e-40a0-8997-90a3fba2a424-istiod-ca-cert\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-5tcsp\" (UID: \"62422ac3-d47e-40a0-8997-90a3fba2a424\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-5tcsp"
Oct 30 00:49:50.948775 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:50.948649    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62422ac3-d47e-40a0-8997-90a3fba2a424-workload-certs\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-5tcsp\" (UID: \"62422ac3-d47e-40a0-8997-90a3fba2a424\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-5tcsp"
Oct 30 00:49:50.963532 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:50.963493    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c72b5051-4287-4298-b179-4e39b2535eb5-istio-token\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-j9nfb\" (UID: \"c72b5051-4287-4298-b179-4e39b2535eb5\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-j9nfb"
Oct 30 00:49:50.963780 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:50.963758    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62422ac3-d47e-40a0-8997-90a3fba2a424-istio-envoy\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-5tcsp\" (UID: \"62422ac3-d47e-40a0-8997-90a3fba2a424\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-5tcsp"
Oct 30 00:49:50.963908 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:50.963892    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62422ac3-d47e-40a0-8997-90a3fba2a424-istio-token\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-5tcsp\" (UID: \"62422ac3-d47e-40a0-8997-90a3fba2a424\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-5tcsp"
Oct 30 00:49:50.964058 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:50.964042    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62422ac3-d47e-40a0-8997-90a3fba2a424-istio-data\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-5tcsp\" (UID: \"62422ac3-d47e-40a0-8997-90a3fba2a424\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-5tcsp"
Oct 30 00:49:50.964191 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:50.964171    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mplhb\" (UniqueName: \"kubernetes.io/projected/c72b5051-4287-4298-b179-4e39b2535eb5-kube-api-access-mplhb\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-j9nfb\" (UID: \"c72b5051-4287-4298-b179-4e39b2535eb5\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-j9nfb"
Oct 30 00:49:50.964319 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:50.964302    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62422ac3-d47e-40a0-8997-90a3fba2a424-credential-socket\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-5tcsp\" (UID: \"62422ac3-d47e-40a0-8997-90a3fba2a424\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-5tcsp"
Oct 30 00:49:50.964485 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:50.964466    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c72b5051-4287-4298-b179-4e39b2535eb5-istio-podinfo\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-j9nfb\" (UID: \"c72b5051-4287-4298-b179-4e39b2535eb5\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-j9nfb"
Oct 30 00:49:50.973471 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:50.964629    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c72b5051-4287-4298-b179-4e39b2535eb5-istio-envoy\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-j9nfb\" (UID: \"c72b5051-4287-4298-b179-4e39b2535eb5\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-j9nfb"
Oct 30 00:49:50.983406 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:50.973703    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c72b5051-4287-4298-b179-4e39b2535eb5-istio-data\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-j9nfb\" (UID: \"c72b5051-4287-4298-b179-4e39b2535eb5\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-j9nfb"
Oct 30 00:49:50.983406 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:50.973801    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5nvh\" (UniqueName: \"kubernetes.io/projected/62422ac3-d47e-40a0-8997-90a3fba2a424-kube-api-access-n5nvh\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-5tcsp\" (UID: \"62422ac3-d47e-40a0-8997-90a3fba2a424\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-5tcsp"
Oct 30 00:49:50.983406 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:50.973856    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c72b5051-4287-4298-b179-4e39b2535eb5-workload-socket\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-j9nfb\" (UID: \"c72b5051-4287-4298-b179-4e39b2535eb5\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-j9nfb"
Oct 30 00:49:50.983406 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:50.973911    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c72b5051-4287-4298-b179-4e39b2535eb5-workload-certs\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-j9nfb\" (UID: \"c72b5051-4287-4298-b179-4e39b2535eb5\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-j9nfb"
Oct 30 00:49:50.983406 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:50.973951    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c72b5051-4287-4298-b179-4e39b2535eb5-istiod-ca-cert\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-j9nfb\" (UID: \"c72b5051-4287-4298-b179-4e39b2535eb5\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-j9nfb"
Oct 30 00:49:50.983406 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:50.974001    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62422ac3-d47e-40a0-8997-90a3fba2a424-istio-podinfo\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-5tcsp\" (UID: \"62422ac3-d47e-40a0-8997-90a3fba2a424\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-5tcsp"
Oct 30 00:49:50.983406 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:50.974121    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c72b5051-4287-4298-b179-4e39b2535eb5-credential-socket\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-j9nfb\" (UID: \"c72b5051-4287-4298-b179-4e39b2535eb5\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-j9nfb"
Oct 30 00:49:50.983406 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:50.974248    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62422ac3-d47e-40a0-8997-90a3fba2a424-workload-socket\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-5tcsp\" (UID: \"62422ac3-d47e-40a0-8997-90a3fba2a424\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-5tcsp"
Oct 30 00:49:51.127406 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:51.121363    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62422ac3-d47e-40a0-8997-90a3fba2a424-workload-socket\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-5tcsp\" (UID: \"62422ac3-d47e-40a0-8997-90a3fba2a424\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-5tcsp"
Oct 30 00:49:51.127406 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:51.121436    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62422ac3-d47e-40a0-8997-90a3fba2a424-istiod-ca-cert\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-5tcsp\" (UID: \"62422ac3-d47e-40a0-8997-90a3fba2a424\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-5tcsp"
Oct 30 00:49:51.127406 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:51.121470    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62422ac3-d47e-40a0-8997-90a3fba2a424-workload-certs\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-5tcsp\" (UID: \"62422ac3-d47e-40a0-8997-90a3fba2a424\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-5tcsp"
Oct 30 00:49:51.127406 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:51.121504    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c72b5051-4287-4298-b179-4e39b2535eb5-istio-token\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-j9nfb\" (UID: \"c72b5051-4287-4298-b179-4e39b2535eb5\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-j9nfb"
Oct 30 00:49:51.127406 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:51.125838    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62422ac3-d47e-40a0-8997-90a3fba2a424-istiod-ca-cert\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-5tcsp\" (UID: \"62422ac3-d47e-40a0-8997-90a3fba2a424\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-5tcsp"
Oct 30 00:49:51.157403 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:51.130598    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62422ac3-d47e-40a0-8997-90a3fba2a424-workload-certs\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-5tcsp\" (UID: \"62422ac3-d47e-40a0-8997-90a3fba2a424\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-5tcsp"
Oct 30 00:49:51.157403 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:51.154497    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62422ac3-d47e-40a0-8997-90a3fba2a424-istio-envoy\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-5tcsp\" (UID: \"62422ac3-d47e-40a0-8997-90a3fba2a424\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-5tcsp"
Oct 30 00:49:51.157403 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:51.154536    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62422ac3-d47e-40a0-8997-90a3fba2a424-istio-token\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-5tcsp\" (UID: \"62422ac3-d47e-40a0-8997-90a3fba2a424\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-5tcsp"
Oct 30 00:49:51.157403 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:51.154596    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62422ac3-d47e-40a0-8997-90a3fba2a424-istio-data\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-5tcsp\" (UID: \"62422ac3-d47e-40a0-8997-90a3fba2a424\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-5tcsp"
Oct 30 00:49:51.157403 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:51.154628    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mplhb\" (UniqueName: \"kubernetes.io/projected/c72b5051-4287-4298-b179-4e39b2535eb5-kube-api-access-mplhb\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-j9nfb\" (UID: \"c72b5051-4287-4298-b179-4e39b2535eb5\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-j9nfb"
Oct 30 00:49:51.157403 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:51.154660    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62422ac3-d47e-40a0-8997-90a3fba2a424-credential-socket\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-5tcsp\" (UID: \"62422ac3-d47e-40a0-8997-90a3fba2a424\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-5tcsp"
Oct 30 00:49:51.157403 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:51.154714    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c72b5051-4287-4298-b179-4e39b2535eb5-istio-podinfo\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-j9nfb\" (UID: \"c72b5051-4287-4298-b179-4e39b2535eb5\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-j9nfb"
Oct 30 00:49:51.157403 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:51.154770    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c72b5051-4287-4298-b179-4e39b2535eb5-istio-envoy\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-j9nfb\" (UID: \"c72b5051-4287-4298-b179-4e39b2535eb5\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-j9nfb"
Oct 30 00:49:51.157403 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:51.154822    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c72b5051-4287-4298-b179-4e39b2535eb5-istio-data\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-j9nfb\" (UID: \"c72b5051-4287-4298-b179-4e39b2535eb5\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-j9nfb"
Oct 30 00:49:51.157403 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:51.154895    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5nvh\" (UniqueName: \"kubernetes.io/projected/62422ac3-d47e-40a0-8997-90a3fba2a424-kube-api-access-n5nvh\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-5tcsp\" (UID: \"62422ac3-d47e-40a0-8997-90a3fba2a424\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-5tcsp"
Oct 30 00:49:51.157403 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:51.154941    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c72b5051-4287-4298-b179-4e39b2535eb5-workload-socket\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-j9nfb\" (UID: \"c72b5051-4287-4298-b179-4e39b2535eb5\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-j9nfb"
Oct 30 00:49:51.157403 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:51.154990    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c72b5051-4287-4298-b179-4e39b2535eb5-workload-certs\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-j9nfb\" (UID: \"c72b5051-4287-4298-b179-4e39b2535eb5\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-j9nfb"
Oct 30 00:49:51.157403 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:51.155026    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c72b5051-4287-4298-b179-4e39b2535eb5-istiod-ca-cert\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-j9nfb\" (UID: \"c72b5051-4287-4298-b179-4e39b2535eb5\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-j9nfb"
Oct 30 00:49:51.157403 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:51.155068    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62422ac3-d47e-40a0-8997-90a3fba2a424-istio-podinfo\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-5tcsp\" (UID: \"62422ac3-d47e-40a0-8997-90a3fba2a424\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-5tcsp"
Oct 30 00:49:51.157403 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:51.155167    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c72b5051-4287-4298-b179-4e39b2535eb5-credential-socket\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-j9nfb\" (UID: \"c72b5051-4287-4298-b179-4e39b2535eb5\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-j9nfb"
Oct 30 00:49:51.185365 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:51.183992    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c72b5051-4287-4298-b179-4e39b2535eb5-credential-socket\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-j9nfb\" (UID: \"c72b5051-4287-4298-b179-4e39b2535eb5\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-j9nfb"
Oct 30 00:49:51.257136 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:51.257056    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62422ac3-d47e-40a0-8997-90a3fba2a424-workload-socket\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-5tcsp\" (UID: \"62422ac3-d47e-40a0-8997-90a3fba2a424\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-5tcsp"
Oct 30 00:49:51.305513 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:51.294416    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62422ac3-d47e-40a0-8997-90a3fba2a424-istio-data\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-5tcsp\" (UID: \"62422ac3-d47e-40a0-8997-90a3fba2a424\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-5tcsp"
Oct 30 00:49:51.307690 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:51.294899    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62422ac3-d47e-40a0-8997-90a3fba2a424-credential-socket\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-5tcsp\" (UID: \"62422ac3-d47e-40a0-8997-90a3fba2a424\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-5tcsp"
Oct 30 00:49:51.333126 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:51.317076    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c72b5051-4287-4298-b179-4e39b2535eb5-istio-envoy\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-j9nfb\" (UID: \"c72b5051-4287-4298-b179-4e39b2535eb5\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-j9nfb"
Oct 30 00:49:51.333347 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:51.317335    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c72b5051-4287-4298-b179-4e39b2535eb5-workload-socket\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-j9nfb\" (UID: \"c72b5051-4287-4298-b179-4e39b2535eb5\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-j9nfb"
Oct 30 00:49:51.333483 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:51.317550    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-5tcsp"]
Oct 30 00:49:51.333633 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:51.318209    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c72b5051-4287-4298-b179-4e39b2535eb5-istiod-ca-cert\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-j9nfb\" (UID: \"c72b5051-4287-4298-b179-4e39b2535eb5\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-j9nfb"
Oct 30 00:49:51.350684 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:51.350622    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mplhb\" (UniqueName: \"kubernetes.io/projected/c72b5051-4287-4298-b179-4e39b2535eb5-kube-api-access-mplhb\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-j9nfb\" (UID: \"c72b5051-4287-4298-b179-4e39b2535eb5\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-j9nfb"
Oct 30 00:49:51.353878 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:51.353806    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c72b5051-4287-4298-b179-4e39b2535eb5-istio-data\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-j9nfb\" (UID: \"c72b5051-4287-4298-b179-4e39b2535eb5\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-j9nfb"
Oct 30 00:49:51.354165 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:51.354130    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c72b5051-4287-4298-b179-4e39b2535eb5-workload-certs\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-j9nfb\" (UID: \"c72b5051-4287-4298-b179-4e39b2535eb5\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-j9nfb"
Oct 30 00:49:51.371307 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:51.371263    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62422ac3-d47e-40a0-8997-90a3fba2a424-istio-podinfo\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-5tcsp\" (UID: \"62422ac3-d47e-40a0-8997-90a3fba2a424\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-5tcsp"
Oct 30 00:49:51.373895 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:51.373868    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62422ac3-d47e-40a0-8997-90a3fba2a424-istio-token\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-5tcsp\" (UID: \"62422ac3-d47e-40a0-8997-90a3fba2a424\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-5tcsp"
Oct 30 00:49:51.402416 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:51.398465    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c72b5051-4287-4298-b179-4e39b2535eb5-istio-token\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-j9nfb\" (UID: \"c72b5051-4287-4298-b179-4e39b2535eb5\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-j9nfb"
Oct 30 00:49:51.402416 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:51.399637    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5nvh\" (UniqueName: \"kubernetes.io/projected/62422ac3-d47e-40a0-8997-90a3fba2a424-kube-api-access-n5nvh\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-5tcsp\" (UID: \"62422ac3-d47e-40a0-8997-90a3fba2a424\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-5tcsp"
Oct 30 00:49:51.425407 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:51.411704    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62422ac3-d47e-40a0-8997-90a3fba2a424-istio-envoy\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-5tcsp\" (UID: \"62422ac3-d47e-40a0-8997-90a3fba2a424\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-5tcsp"
Oct 30 00:49:51.426678 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:51.426132    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c72b5051-4287-4298-b179-4e39b2535eb5-istio-podinfo\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-j9nfb\" (UID: \"c72b5051-4287-4298-b179-4e39b2535eb5\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-j9nfb"
Oct 30 00:49:51.426678 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:51.426428    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-5tcsp"
Oct 30 00:49:51.526422 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:51.525878    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca5363/distinct-2-w4-ca5363-kserve-576576f9b5-xkkqc" podUID="5de8bebf-cfad-49da-9a98-08aedde4442c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Oct 30 00:49:51.589406 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:51.587487    5002 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-021d6d/simulator-w6-021d6d-kserve-6cc97fdf86-dqtpw" podUID="50586f76-4f3b-4e7a-83ac-6c573faee415" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:49:51.661252 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:51.660953    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-j9nfb"
Oct 30 00:49:51.933474 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:51.932358    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cefb27/simulator-w3-cefb27-kserve-6b669c9d9f-zx2vm" podUID="7121e81e-f903-42c0-a7a4-bc18d49194e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:49:51.973176 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:51.961027    5002 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-52650f/premium-w0-52650f-kserve-c5bf66f9d-qq5dm" podUID="3455d983-b11b-4cee-bfce-d1f399cf9679" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:49:52.042808 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:52.015437    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-52650f/premium-w0-52650f-kserve-c5bf66f9d-qq5dm" podUID="3455d983-b11b-4cee-bfce-d1f399cf9679" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:49:52.052854 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:52.052475    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-52650f/simulator-w0-52650f-kserve-58b64766c-94qxx" podUID="659c1425-5b67-4d01-bfc2-f1ba56b9a59b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:49:52.116356 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:52.115508    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1a11cf/distinct-w2-1a11cf-kserve-854cf4cdf8-cddw6" podUID="32a03478-46eb-4363-97b6-51a32c4099c7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:49:52.342274 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:52.317477    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6f4fba/simulator-w5-6f4fba-kserve-5bf8dfddd5-wcwzh" podUID="3a2c2f8f-649c-4ab6-af98-5d87df0e511f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/ready\": context deadline exceeded"
Oct 30 00:49:52.452068 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:52.437704    5002 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-6f4fba/simulator-w5-6f4fba-kserve-5bf8dfddd5-wcwzh" podUID="3a2c2f8f-649c-4ab6-af98-5d87df0e511f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:49:52.462855 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:52.458513    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 00:49:52.462855 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:52.458612    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 00:49:52.462855 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:52.458658    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 00:49:52.493408 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:52.484941    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-5tcsp"]
Oct 30 00:49:52.567411 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:52.566317    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca5363/distinct-2-w4-ca5363-kserve-576576f9b5-xkkqc" podUID="5de8bebf-cfad-49da-9a98-08aedde4442c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:49:52.747048 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:52.747010    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ca5363/distinct-2-w4-ca5363-kserve-576576f9b5-xkkqc"
Oct 30 00:49:52.945007 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:52.944948    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cefb27/simulator-w3-cefb27-kserve-6b669c9d9f-zx2vm" podUID="7121e81e-f903-42c0-a7a4-bc18d49194e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/ready\": context deadline exceeded"
Oct 30 00:49:52.971007 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:52.970542    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-cefb27/simulator-w3-cefb27-kserve-6b669c9d9f-zx2vm"
Oct 30 00:49:52.998052 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:52.974461    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-j9nfb"]
Oct 30 00:49:53.015791 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:52.979633    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-52650f/distinct-w0-52650f-kserve-d85899b5-vlhps" podUID="ea1850d9-abba-4977-a540-f6e0eaefb17d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:49:53.016089 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:53.016005    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-52650f/premium-w0-52650f-kserve-c5bf66f9d-qq5dm" podUID="3455d983-b11b-4cee-bfce-d1f399cf9679" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": context deadline exceeded"
Oct 30 00:49:53.020115 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:53.019194    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-52650f/premium-w0-52650f-kserve-c5bf66f9d-qq5dm"
Oct 30 00:49:53.039873 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:53.039826    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-29be73/simulator-w1-29be73-kserve-9958d76db-fks2t" podUID="fbb3c806-8967-4d39-84c6-9dc022fc5cae" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:49:53.045500 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:53.045449    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-29be73/premium-w1-29be73-kserve-7b777cfc8c-7kgp5" podUID="e6fabbdd-86ca-4cbf-85be-ef250575a355" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:49:53.081970 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:53.059800    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-52650f/simulator-w0-52650f-kserve-58b64766c-94qxx" podUID="659c1425-5b67-4d01-bfc2-f1ba56b9a59b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:49:53.081970 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:53.061213    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-52650f/simulator-w0-52650f-kserve-58b64766c-94qxx"
Oct 30 00:49:53.114414 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:53.107970    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 00:49:53.114738 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:53.114701    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 00:49:53.114862 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:53.114848    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 00:49:53.188490 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:53.167854    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1a11cf/distinct-w2-1a11cf-kserve-854cf4cdf8-cddw6" podUID="32a03478-46eb-4363-97b6-51a32c4099c7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded"
Oct 30 00:49:53.188490 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:53.168606    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-1a11cf/distinct-w2-1a11cf-kserve-854cf4cdf8-cddw6"
Oct 30 00:49:53.296939 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:53.286021    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-1a11cf/distinct-w2-1a11cf-kserve-854cf4cdf8-cddw6"
Oct 30 00:49:53.343365 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:53.325463    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6f4fba/simulator-w5-6f4fba-kserve-5bf8dfddd5-wcwzh" podUID="3a2c2f8f-649c-4ab6-af98-5d87df0e511f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:49:53.343365 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:53.325820    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1a11cf/distinct-2-w2-1a11cf-kserve-758bb6dd57-j5f2k" podUID="f2d66dea-9f0b-43f3-a774-bb065e88f378" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded"
Oct 30 00:49:53.343365 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:53.326278    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-6f4fba/simulator-w5-6f4fba-kserve-5bf8dfddd5-wcwzh"
Oct 30 00:49:53.363617 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:53.363007    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-6f4fba/simulator-w5-6f4fba-kserve-5bf8dfddd5-wcwzh"
Oct 30 00:49:53.581464 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:53.569569    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca5363/distinct-2-w4-ca5363-kserve-576576f9b5-xkkqc" podUID="5de8bebf-cfad-49da-9a98-08aedde4442c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:49:53.628355 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:53.618341    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-52650f/premium-w0-52650f-kserve-c5bf66f9d-qq5dm"
Oct 30 00:49:53.648612 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:53.648568    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-5tcsp" event={"ID":"62422ac3-d47e-40a0-8997-90a3fba2a424","Type":"ContainerStarted","Data":"81c4fe4b7128a339eaca861bf79401abd0a20577405c2177e9ef05fdf14ff4ac"}
Oct 30 00:49:53.651743 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:53.651708    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-j9nfb" event={"ID":"c72b5051-4287-4298-b179-4e39b2535eb5","Type":"ContainerStarted","Data":"5d1785dd7f6c048e6aeefa299860d74aa1bf0268712f6f72530738b92befc864"}
Oct 30 00:49:53.713945 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:53.693663    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ca5363/distinct-2-w4-ca5363-kserve-576576f9b5-xkkqc"
Oct 30 00:49:54.043469 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:53.950519    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cefb27/simulator-w3-cefb27-kserve-6b669c9d9f-zx2vm" podUID="7121e81e-f903-42c0-a7a4-bc18d49194e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:49:54.333633 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:54.331066    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-52650f/simulator-w0-52650f-kserve-58b64766c-94qxx" podUID="659c1425-5b67-4d01-bfc2-f1ba56b9a59b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:49:54.618201 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:54.618106    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-52650f/premium-w0-52650f-kserve-c5bf66f9d-qq5dm" podUID="3455d983-b11b-4cee-bfce-d1f399cf9679" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:49:54.767469 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:54.757509    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-cefb27/simulator-w3-cefb27-kserve-6b669c9d9f-zx2vm"
Oct 30 00:49:54.780899 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:54.780862    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-5tcsp" event={"ID":"62422ac3-d47e-40a0-8997-90a3fba2a424","Type":"ContainerStarted","Data":"8fc8dd904bcddbd6ed9a21b6d6a5543ab1cd58452f15e943d350738b78f81f16"}
Oct 30 00:49:54.796300 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:54.796264    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-j9nfb" event={"ID":"c72b5051-4287-4298-b179-4e39b2535eb5","Type":"ContainerStarted","Data":"7ee356c780ab592d8054cf3342ed989c829d92c2815ee050d38beec4a7d48e3b"}
Oct 30 00:49:54.805502 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:54.805469    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-52650f/simulator-w0-52650f-kserve-58b64766c-94qxx"
Oct 30 00:49:54.941828 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:54.941766    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-j9nfb" podStartSLOduration=4.941750855 podStartE2EDuration="4.941750855s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:49:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:49:54.936944725 +0000 UTC m=+1090.553377614" watchObservedRunningTime="2026-10-30 00:49:54.941750855 +0000 UTC m=+1090.558183730"
Oct 30 00:49:54.942863 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:54.942819    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-5tcsp" podStartSLOduration=4.942808041 podStartE2EDuration="4.942808041s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:49:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:49:54.888263811 +0000 UTC m=+1090.504696691" watchObservedRunningTime="2026-10-30 00:49:54.942808041 +0000 UTC m=+1090.559240919"
Oct 30 00:49:55.427520 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:55.427413    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-5tcsp"
Oct 30 00:49:55.428568 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:55.428532    5002 patch_prober.go:29] interesting pod/e2e-worker-w1-29be73-openshift-default-66467f5dcf-5tcsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body=
Oct 30 00:49:55.428714 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:55.428603    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-5tcsp" podUID="62422ac3-d47e-40a0-8997-90a3fba2a424" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused"
Oct 30 00:49:55.662738 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:55.662705    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-j9nfb"
Oct 30 00:49:55.664898 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:55.664571    5002 patch_prober.go:29] interesting pod/e2e-worker-w1-29be73-openshift-default-66467f5dcf-j9nfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body=
Oct 30 00:49:55.664898 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:55.664630    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-j9nfb" podUID="c72b5051-4287-4298-b179-4e39b2535eb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused"
Oct 30 00:49:56.427737 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:56.427704    5002 patch_prober.go:29] interesting pod/e2e-worker-w1-29be73-openshift-default-66467f5dcf-5tcsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body=
Oct 30 00:49:56.428372 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:56.428333    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-5tcsp" podUID="62422ac3-d47e-40a0-8997-90a3fba2a424" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused"
Oct 30 00:49:56.695465 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:56.685122    5002 patch_prober.go:29] interesting pod/e2e-worker-w1-29be73-openshift-default-66467f5dcf-j9nfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body=
Oct 30 00:49:56.695465 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:56.685196    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-j9nfb" podUID="c72b5051-4287-4298-b179-4e39b2535eb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused"
Oct 30 00:49:57.427749 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:57.427708    5002 patch_prober.go:29] interesting pod/e2e-worker-w1-29be73-openshift-default-66467f5dcf-5tcsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body=
Oct 30 00:49:57.428137 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:57.427763    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-5tcsp" podUID="62422ac3-d47e-40a0-8997-90a3fba2a424" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused"
Oct 30 00:49:57.665791 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:57.665751    5002 patch_prober.go:29] interesting pod/e2e-worker-w1-29be73-openshift-default-66467f5dcf-j9nfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body=
Oct 30 00:49:57.666059 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:57.666030    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-j9nfb" podUID="c72b5051-4287-4298-b179-4e39b2535eb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused"
Oct 30 00:49:58.427853 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:58.427818    5002 patch_prober.go:29] interesting pod/e2e-worker-w1-29be73-openshift-default-66467f5dcf-5tcsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body=
Oct 30 00:49:58.428571 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:58.427868    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-5tcsp" podUID="62422ac3-d47e-40a0-8997-90a3fba2a424" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused"
Oct 30 00:49:58.693467 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:58.677730    5002 patch_prober.go:29] interesting pod/e2e-worker-w1-29be73-openshift-default-66467f5dcf-j9nfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body=
Oct 30 00:49:58.693467 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:58.677789    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-j9nfb" podUID="c72b5051-4287-4298-b179-4e39b2535eb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused"
Oct 30 00:49:59.394403 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:59.393499    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cab9990d-cfgtj"]
Oct 30 00:49:59.399682 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:59.399656    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cab9990d-cfgtj"
Oct 30 00:49:59.415894 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:59.415835    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cab9990d-cfgtj"]
Oct 30 00:49:59.420690 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:59.420598    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 00:49:59.423802 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:59.423776    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 00:49:59.424124 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:59.424077    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5phsh\""
Oct 30 00:49:59.427712 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:59.427689    5002 patch_prober.go:29] interesting pod/e2e-worker-w1-29be73-openshift-default-66467f5dcf-5tcsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body=
Oct 30 00:49:59.427932 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:59.427856    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-5tcsp" podUID="62422ac3-d47e-40a0-8997-90a3fba2a424" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused"
Oct 30 00:49:59.429080 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:59.429051    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44b94cc1-4817-4b62-9aed-8ced2c0ebb83-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-cab9990d-cfgtj\" (UID: \"44b94cc1-4817-4b62-9aed-8ced2c0ebb83\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cab9990d-cfgtj"
Oct 30 00:49:59.429183 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:59.429157    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hsxk\" (UniqueName: \"kubernetes.io/projected/44b94cc1-4817-4b62-9aed-8ced2c0ebb83-kube-api-access-7hsxk\") pod \"maas-api-revoke-keys-e2e-migrate-cab9990d-cfgtj\" (UID: \"44b94cc1-4817-4b62-9aed-8ced2c0ebb83\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cab9990d-cfgtj"
Oct 30 00:49:59.429305 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:59.429287    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44b94cc1-4817-4b62-9aed-8ced2c0ebb83-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-cab9990d-cfgtj\" (UID: \"44b94cc1-4817-4b62-9aed-8ced2c0ebb83\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cab9990d-cfgtj"
Oct 30 00:49:59.433943 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:59.433919    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 00:49:59.531326 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:59.531281    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44b94cc1-4817-4b62-9aed-8ced2c0ebb83-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-cab9990d-cfgtj\" (UID: \"44b94cc1-4817-4b62-9aed-8ced2c0ebb83\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cab9990d-cfgtj"
Oct 30 00:49:59.531583 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:59.531374    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hsxk\" (UniqueName: \"kubernetes.io/projected/44b94cc1-4817-4b62-9aed-8ced2c0ebb83-kube-api-access-7hsxk\") pod \"maas-api-revoke-keys-e2e-migrate-cab9990d-cfgtj\" (UID: \"44b94cc1-4817-4b62-9aed-8ced2c0ebb83\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cab9990d-cfgtj"
Oct 30 00:49:59.531583 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:59.531498    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44b94cc1-4817-4b62-9aed-8ced2c0ebb83-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-cab9990d-cfgtj\" (UID: \"44b94cc1-4817-4b62-9aed-8ced2c0ebb83\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cab9990d-cfgtj"
Oct 30 00:49:59.532849 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:59.532783    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44b94cc1-4817-4b62-9aed-8ced2c0ebb83-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-cab9990d-cfgtj\" (UID: \"44b94cc1-4817-4b62-9aed-8ced2c0ebb83\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cab9990d-cfgtj"
Oct 30 00:49:59.545314 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:59.545280    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44b94cc1-4817-4b62-9aed-8ced2c0ebb83-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-cab9990d-cfgtj\" (UID: \"44b94cc1-4817-4b62-9aed-8ced2c0ebb83\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cab9990d-cfgtj"
Oct 30 00:49:59.620405 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:59.606488    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hsxk\" (UniqueName: \"kubernetes.io/projected/44b94cc1-4817-4b62-9aed-8ced2c0ebb83-kube-api-access-7hsxk\") pod \"maas-api-revoke-keys-e2e-migrate-cab9990d-cfgtj\" (UID: \"44b94cc1-4817-4b62-9aed-8ced2c0ebb83\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cab9990d-cfgtj"
Oct 30 00:49:59.680874 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:59.680786    5002 patch_prober.go:29] interesting pod/e2e-worker-w1-29be73-openshift-default-66467f5dcf-j9nfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body=
Oct 30 00:49:59.681188 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:59.681156    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-j9nfb" podUID="c72b5051-4287-4298-b179-4e39b2535eb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused"
Oct 30 00:49:59.736521 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:49:59.736485    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cab9990d-cfgtj"
Oct 30 00:50:00.222719 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:50:00.221594    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44b94cc1_4817_4b62_9aed_8ced2c0ebb83.slice/crio-14d0632549303821a1a0d013024d4706689534b3d58f8184390a65224f7d0036 WatchSource:0}: Error finding container 14d0632549303821a1a0d013024d4706689534b3d58f8184390a65224f7d0036: Status 404 returned error can't find the container with id 14d0632549303821a1a0d013024d4706689534b3d58f8184390a65224f7d0036
Oct 30 00:50:00.222719 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:00.222560    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cab9990d-cfgtj"]
Oct 30 00:50:00.428861 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:00.428808    5002 patch_prober.go:29] interesting pod/e2e-worker-w1-29be73-openshift-default-66467f5dcf-5tcsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body=
Oct 30 00:50:00.429272 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:00.428881    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-5tcsp" podUID="62422ac3-d47e-40a0-8997-90a3fba2a424" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused"
Oct 30 00:50:00.430342 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:00.430315    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-c85z8"]
Oct 30 00:50:00.436231 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:00.436205    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-c85z8"
Oct 30 00:50:00.507504 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:00.507475    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-cab9990d-gw-openshift-default-dockercfg-7h89s\""
Oct 30 00:50:00.535594 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:00.535561    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/792cb412-63fa-4eab-b70a-b6969a02897e-workload-certs\") pod \"e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-c85z8\" (UID: \"792cb412-63fa-4eab-b70a-b6969a02897e\") " pod="openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-c85z8"
Oct 30 00:50:00.535889 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:00.535866    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/792cb412-63fa-4eab-b70a-b6969a02897e-istio-token\") pod \"e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-c85z8\" (UID: \"792cb412-63fa-4eab-b70a-b6969a02897e\") " pod="openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-c85z8"
Oct 30 00:50:00.536164 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:00.536141    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/792cb412-63fa-4eab-b70a-b6969a02897e-istio-data\") pod \"e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-c85z8\" (UID: \"792cb412-63fa-4eab-b70a-b6969a02897e\") " pod="openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-c85z8"
Oct 30 00:50:00.536314 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:00.536297    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/792cb412-63fa-4eab-b70a-b6969a02897e-credential-socket\") pod \"e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-c85z8\" (UID: \"792cb412-63fa-4eab-b70a-b6969a02897e\") " pod="openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-c85z8"
Oct 30 00:50:00.536558 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:00.536540    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/792cb412-63fa-4eab-b70a-b6969a02897e-istio-podinfo\") pod \"e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-c85z8\" (UID: \"792cb412-63fa-4eab-b70a-b6969a02897e\") " pod="openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-c85z8"
Oct 30 00:50:00.536721 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:00.536706    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/792cb412-63fa-4eab-b70a-b6969a02897e-istio-envoy\") pod \"e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-c85z8\" (UID: \"792cb412-63fa-4eab-b70a-b6969a02897e\") " pod="openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-c85z8"
Oct 30 00:50:00.536877 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:00.536858    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/792cb412-63fa-4eab-b70a-b6969a02897e-istiod-ca-cert\") pod \"e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-c85z8\" (UID: \"792cb412-63fa-4eab-b70a-b6969a02897e\") " pod="openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-c85z8"
Oct 30 00:50:00.537011 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:00.536994    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2png\" (UniqueName: \"kubernetes.io/projected/792cb412-63fa-4eab-b70a-b6969a02897e-kube-api-access-k2png\") pod \"e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-c85z8\" (UID: \"792cb412-63fa-4eab-b70a-b6969a02897e\") " pod="openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-c85z8"
Oct 30 00:50:00.537122 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:00.537105    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/792cb412-63fa-4eab-b70a-b6969a02897e-workload-socket\") pod \"e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-c85z8\" (UID: \"792cb412-63fa-4eab-b70a-b6969a02897e\") " pod="openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-c85z8"
Oct 30 00:50:00.546306 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:00.546172    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-c85z8"]
Oct 30 00:50:00.638485 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:00.638445    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/792cb412-63fa-4eab-b70a-b6969a02897e-istio-data\") pod \"e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-c85z8\" (UID: \"792cb412-63fa-4eab-b70a-b6969a02897e\") " pod="openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-c85z8"
Oct 30 00:50:00.638485 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:00.638489    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/792cb412-63fa-4eab-b70a-b6969a02897e-credential-socket\") pod \"e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-c85z8\" (UID: \"792cb412-63fa-4eab-b70a-b6969a02897e\") " pod="openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-c85z8"
Oct 30 00:50:00.638743 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:00.638549    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/792cb412-63fa-4eab-b70a-b6969a02897e-istio-podinfo\") pod \"e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-c85z8\" (UID: \"792cb412-63fa-4eab-b70a-b6969a02897e\") " pod="openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-c85z8"
Oct 30 00:50:00.638743 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:00.638583    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/792cb412-63fa-4eab-b70a-b6969a02897e-istio-envoy\") pod \"e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-c85z8\" (UID: \"792cb412-63fa-4eab-b70a-b6969a02897e\") " pod="openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-c85z8"
Oct 30 00:50:00.638743 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:00.638614    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/792cb412-63fa-4eab-b70a-b6969a02897e-istiod-ca-cert\") pod \"e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-c85z8\" (UID: \"792cb412-63fa-4eab-b70a-b6969a02897e\") " pod="openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-c85z8"
Oct 30 00:50:00.638743 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:00.638640    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2png\" (UniqueName: \"kubernetes.io/projected/792cb412-63fa-4eab-b70a-b6969a02897e-kube-api-access-k2png\") pod \"e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-c85z8\" (UID: \"792cb412-63fa-4eab-b70a-b6969a02897e\") " pod="openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-c85z8"
Oct 30 00:50:00.638743 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:00.638658    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/792cb412-63fa-4eab-b70a-b6969a02897e-workload-socket\") pod \"e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-c85z8\" (UID: \"792cb412-63fa-4eab-b70a-b6969a02897e\") " pod="openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-c85z8"
Oct 30 00:50:00.638743 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:00.638699    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/792cb412-63fa-4eab-b70a-b6969a02897e-workload-certs\") pod \"e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-c85z8\" (UID: \"792cb412-63fa-4eab-b70a-b6969a02897e\") " pod="openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-c85z8"
Oct 30 00:50:00.638743 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:00.638723    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/792cb412-63fa-4eab-b70a-b6969a02897e-istio-token\") pod \"e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-c85z8\" (UID: \"792cb412-63fa-4eab-b70a-b6969a02897e\") " pod="openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-c85z8"
Oct 30 00:50:00.639409 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:00.639300    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/792cb412-63fa-4eab-b70a-b6969a02897e-credential-socket\") pod \"e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-c85z8\" (UID: \"792cb412-63fa-4eab-b70a-b6969a02897e\") " pod="openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-c85z8"
Oct 30 00:50:00.639654 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:00.639589    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/792cb412-63fa-4eab-b70a-b6969a02897e-workload-certs\") pod \"e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-c85z8\" (UID: \"792cb412-63fa-4eab-b70a-b6969a02897e\") " pod="openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-c85z8"
Oct 30 00:50:00.639654 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:00.639602    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/792cb412-63fa-4eab-b70a-b6969a02897e-istio-data\") pod \"e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-c85z8\" (UID: \"792cb412-63fa-4eab-b70a-b6969a02897e\") " pod="openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-c85z8"
Oct 30 00:50:00.639779 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:00.639651    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/792cb412-63fa-4eab-b70a-b6969a02897e-workload-socket\") pod \"e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-c85z8\" (UID: \"792cb412-63fa-4eab-b70a-b6969a02897e\") " pod="openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-c85z8"
Oct 30 00:50:00.640070 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:00.640044    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/792cb412-63fa-4eab-b70a-b6969a02897e-istiod-ca-cert\") pod \"e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-c85z8\" (UID: \"792cb412-63fa-4eab-b70a-b6969a02897e\") " pod="openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-c85z8"
Oct 30 00:50:00.641397 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:00.641352    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/792cb412-63fa-4eab-b70a-b6969a02897e-istio-envoy\") pod \"e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-c85z8\" (UID: \"792cb412-63fa-4eab-b70a-b6969a02897e\") " pod="openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-c85z8"
Oct 30 00:50:00.642704 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:00.642677    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/792cb412-63fa-4eab-b70a-b6969a02897e-istio-podinfo\") pod \"e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-c85z8\" (UID: \"792cb412-63fa-4eab-b70a-b6969a02897e\") " pod="openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-c85z8"
Oct 30 00:50:00.663503 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:00.663055    5002 patch_prober.go:29] interesting pod/e2e-worker-w1-29be73-openshift-default-66467f5dcf-j9nfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body=
Oct 30 00:50:00.663503 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:00.663114    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-j9nfb" podUID="c72b5051-4287-4298-b179-4e39b2535eb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused"
Oct 30 00:50:00.695595 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:00.695563    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2png\" (UniqueName: \"kubernetes.io/projected/792cb412-63fa-4eab-b70a-b6969a02897e-kube-api-access-k2png\") pod \"e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-c85z8\" (UID: \"792cb412-63fa-4eab-b70a-b6969a02897e\") " pod="openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-c85z8"
Oct 30 00:50:00.704314 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:00.704139    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/792cb412-63fa-4eab-b70a-b6969a02897e-istio-token\") pod \"e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-c85z8\" (UID: \"792cb412-63fa-4eab-b70a-b6969a02897e\") " pod="openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-c85z8"
Oct 30 00:50:00.779772 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:00.779674    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-c85z8"
Oct 30 00:50:00.929813 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:00.927785    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cab9990d-cfgtj" event={"ID":"44b94cc1-4817-4b62-9aed-8ced2c0ebb83","Type":"ContainerStarted","Data":"b7c5ed48c9a8360bc67c944348411ec45c2747e6629e5a2e7ca6a0417e81d457"}
Oct 30 00:50:00.929813 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:00.927837    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cab9990d-cfgtj" event={"ID":"44b94cc1-4817-4b62-9aed-8ced2c0ebb83","Type":"ContainerStarted","Data":"14d0632549303821a1a0d013024d4706689534b3d58f8184390a65224f7d0036"}
Oct 30 00:50:00.996489 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:00.977212    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cab9990d-cfgtj" podStartSLOduration=1.977192751 podStartE2EDuration="1.977192751s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:49:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:50:00.973810259 +0000 UTC m=+1096.590243155" watchObservedRunningTime="2026-10-30 00:50:00.977192751 +0000 UTC m=+1096.593625630"
Oct 30 00:50:01.171783 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:01.171684    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-c85z8"]
Oct 30 00:50:01.182373 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:50:01.182335    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod792cb412_63fa_4eab_b70a_b6969a02897e.slice/crio-f170a2cd29bb93de9bb7cba86067b7dbcefadec217b0ca9bb820fb6b62d69ca0 WatchSource:0}: Error finding container f170a2cd29bb93de9bb7cba86067b7dbcefadec217b0ca9bb820fb6b62d69ca0: Status 404 returned error can't find the container with id f170a2cd29bb93de9bb7cba86067b7dbcefadec217b0ca9bb820fb6b62d69ca0
Oct 30 00:50:01.186276 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:01.186230    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 00:50:01.186421 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:01.186329    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 00:50:01.186499 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:01.186435    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 00:50:01.427422 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:01.427313    5002 patch_prober.go:29] interesting pod/e2e-worker-w1-29be73-openshift-default-66467f5dcf-5tcsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body=
Oct 30 00:50:01.427422 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:01.427402    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-5tcsp" podUID="62422ac3-d47e-40a0-8997-90a3fba2a424" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused"
Oct 30 00:50:01.669404 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:01.665044    5002 patch_prober.go:29] interesting pod/e2e-worker-w1-29be73-openshift-default-66467f5dcf-j9nfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body=
Oct 30 00:50:01.669404 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:01.665121    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-j9nfb" podUID="c72b5051-4287-4298-b179-4e39b2535eb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused"
Oct 30 00:50:02.029405 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:02.023009    5002 generic.go:356] "Generic (PLEG): container finished" podID="44b94cc1-4817-4b62-9aed-8ced2c0ebb83" containerID="b7c5ed48c9a8360bc67c944348411ec45c2747e6629e5a2e7ca6a0417e81d457" exitCode=0
Oct 30 00:50:02.029405 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:02.023106    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cab9990d-cfgtj" event={"ID":"44b94cc1-4817-4b62-9aed-8ced2c0ebb83","Type":"ContainerDied","Data":"b7c5ed48c9a8360bc67c944348411ec45c2747e6629e5a2e7ca6a0417e81d457"}
Oct 30 00:50:02.029405 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:02.026180    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-c85z8" event={"ID":"792cb412-63fa-4eab-b70a-b6969a02897e","Type":"ContainerStarted","Data":"36e6c2f97204a877e63e47faf015d2016acf0464d823a74500d8a90be5afae3c"}
Oct 30 00:50:02.029405 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:02.026252    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-c85z8" event={"ID":"792cb412-63fa-4eab-b70a-b6969a02897e","Type":"ContainerStarted","Data":"f170a2cd29bb93de9bb7cba86067b7dbcefadec217b0ca9bb820fb6b62d69ca0"}
Oct 30 00:50:02.197300 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:02.195827    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-c85z8" podStartSLOduration=2.195807261 podStartE2EDuration="2.195807261s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:50:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:50:02.195204754 +0000 UTC m=+1097.811637635" watchObservedRunningTime="2026-10-30 00:50:02.195807261 +0000 UTC m=+1097.812240141"
Oct 30 00:50:02.443555 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:02.443522    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-5tcsp"
Oct 30 00:50:02.443980 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:02.443954    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-5tcsp"
Oct 30 00:50:02.444826 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:02.444800    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-5tcsp"
Oct 30 00:50:02.790461 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:02.788633    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-c85z8"
Oct 30 00:50:02.790928 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:02.790449    5002 patch_prober.go:29] interesting pod/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-c85z8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Oct 30 00:50:02.790928 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:02.790587    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-c85z8" podUID="792cb412-63fa-4eab-b70a-b6969a02897e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Oct 30 00:50:02.791668 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:02.791204    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-j9nfb"
Oct 30 00:50:02.793178 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:02.792689    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-j9nfb"
Oct 30 00:50:02.795102 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:02.795002    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-j9nfb"
Oct 30 00:50:03.288645 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:03.288620    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cab9990d-cfgtj"
Oct 30 00:50:03.332462 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:03.332324    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/44b94cc1-4817-4b62-9aed-8ced2c0ebb83-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44b94cc1-4817-4b62-9aed-8ced2c0ebb83-maas-api-xks-ca\") pod \"44b94cc1-4817-4b62-9aed-8ced2c0ebb83\" (UID: \"44b94cc1-4817-4b62-9aed-8ced2c0ebb83\") "
Oct 30 00:50:03.332676 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:03.332507    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44b94cc1-4817-4b62-9aed-8ced2c0ebb83-kube-api-access-7hsxk\" (UniqueName: \"kubernetes.io/projected/44b94cc1-4817-4b62-9aed-8ced2c0ebb83-kube-api-access-7hsxk\") pod \"44b94cc1-4817-4b62-9aed-8ced2c0ebb83\" (UID: \"44b94cc1-4817-4b62-9aed-8ced2c0ebb83\") "
Oct 30 00:50:03.332676 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:03.332574    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44b94cc1-4817-4b62-9aed-8ced2c0ebb83-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44b94cc1-4817-4b62-9aed-8ced2c0ebb83-maas-api-openshift-ca\") pod \"44b94cc1-4817-4b62-9aed-8ced2c0ebb83\" (UID: \"44b94cc1-4817-4b62-9aed-8ced2c0ebb83\") "
Oct 30 00:50:03.333495 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:03.333465    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44b94cc1-4817-4b62-9aed-8ced2c0ebb83-maas-api-openshift-ca" pod "44b94cc1-4817-4b62-9aed-8ced2c0ebb83" (UID: "44b94cc1-4817-4b62-9aed-8ced2c0ebb83"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:50:03.335278 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:03.335217    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44b94cc1-4817-4b62-9aed-8ced2c0ebb83-kube-api-access-7hsxk" pod "44b94cc1-4817-4b62-9aed-8ced2c0ebb83" (UID: "44b94cc1-4817-4b62-9aed-8ced2c0ebb83"). InnerVolumeSpecName "kube-api-access-7hsxk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:50:03.337649 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:03.337625    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44b94cc1-4817-4b62-9aed-8ced2c0ebb83-maas-api-xks-ca" pod "44b94cc1-4817-4b62-9aed-8ced2c0ebb83" (UID: "44b94cc1-4817-4b62-9aed-8ced2c0ebb83"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:50:03.434884 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:03.434848    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44b94cc1-4817-4b62-9aed-8ced2c0ebb83-maas-api-xks-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:50:03.434884 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:03.434880    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7hsxk\" (UniqueName: \"kubernetes.io/projected/44b94cc1-4817-4b62-9aed-8ced2c0ebb83-kube-api-access-7hsxk\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:50:03.434884 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:03.434891    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44b94cc1-4817-4b62-9aed-8ced2c0ebb83-maas-api-openshift-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:50:03.781284 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:03.781202    5002 patch_prober.go:29] interesting pod/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-c85z8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Oct 30 00:50:03.781284 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:03.781262    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-c85z8" podUID="792cb412-63fa-4eab-b70a-b6969a02897e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Oct 30 00:50:04.044857 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:04.044760    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cab9990d-cfgtj"
Oct 30 00:50:04.044857 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:04.044785    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cab9990d-cfgtj" event={"ID":"44b94cc1-4817-4b62-9aed-8ced2c0ebb83","Type":"ContainerDied","Data":"14d0632549303821a1a0d013024d4706689534b3d58f8184390a65224f7d0036"}
Oct 30 00:50:04.044857 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:04.044835    5002 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="14d0632549303821a1a0d013024d4706689534b3d58f8184390a65224f7d0036"
Oct 30 00:50:04.780898 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:04.780861    5002 patch_prober.go:29] interesting pod/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-c85z8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Oct 30 00:50:04.781098 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:04.781017    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-c85z8" podUID="792cb412-63fa-4eab-b70a-b6969a02897e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Oct 30 00:50:05.781650 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:05.781613    5002 patch_prober.go:29] interesting pod/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-c85z8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Oct 30 00:50:05.782047 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:05.781677    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-c85z8" podUID="792cb412-63fa-4eab-b70a-b6969a02897e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Oct 30 00:50:06.781518 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:06.781467    5002 patch_prober.go:29] interesting pod/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-c85z8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Oct 30 00:50:06.781785 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:06.781558    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-c85z8" podUID="792cb412-63fa-4eab-b70a-b6969a02897e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Oct 30 00:50:07.796474 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:07.796437    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-c85z8"
Oct 30 00:50:07.797272 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:07.797246    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-c85z8"
Oct 30 00:50:07.798202 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:07.798183    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-c85z8"
Oct 30 00:50:12.041070 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:12.039453    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98f2e7aa-7vs62"]
Oct 30 00:50:12.050195 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:12.050162    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98f2e7aa-7vs62"]
Oct 30 00:50:12.498626 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:12.498589    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b76dd9d-45bc-41f8-b2a9-1ecbae554e94" path="/var/lib/kubelet/pods/4b76dd9d-45bc-41f8-b2a9-1ecbae554e94/volumes"
Oct 30 00:50:12.862946 ip-10-0-128-7 kubenswrapper[5002]: E1030 00:50:12.862848    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbjdq" containerName="extractor"
Oct 30 00:50:14.911803 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:14.911768    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cab9990d-cfgtj"]
Oct 30 00:50:14.928578 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:14.928541    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cab9990d-cfgtj"]
Oct 30 00:50:15.069083 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:15.069035    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-c85z8"]
Oct 30 00:50:15.069876 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:15.069840    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-c85z8" podUID="792cb412-63fa-4eab-b70a-b6969a02897e" containerName="istio-proxy" containerID="cri-o://36e6c2f97204a877e63e47faf015d2016acf0464d823a74500d8a90be5afae3c" gracePeriod=30
Oct 30 00:50:16.498212 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:16.498175    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44b94cc1-4817-4b62-9aed-8ced2c0ebb83" path="/var/lib/kubelet/pods/44b94cc1-4817-4b62-9aed-8ced2c0ebb83/volumes"
Oct 30 00:50:20.314682 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:20.314647    5002 generic.go:356] "Generic (PLEG): container finished" podID="792cb412-63fa-4eab-b70a-b6969a02897e" containerID="36e6c2f97204a877e63e47faf015d2016acf0464d823a74500d8a90be5afae3c" exitCode=0
Oct 30 00:50:20.315060 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:20.314721    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-c85z8" event={"ID":"792cb412-63fa-4eab-b70a-b6969a02897e","Type":"ContainerDied","Data":"36e6c2f97204a877e63e47faf015d2016acf0464d823a74500d8a90be5afae3c"}
Oct 30 00:50:20.607007 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:20.606975    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-9cjs5"]
Oct 30 00:50:20.607600 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:20.607579    5002 cpu_manager.go:414] "containerMap: removing container" podUID="44b94cc1-4817-4b62-9aed-8ced2c0ebb83" containerName="revoke-keys"
Oct 30 00:50:20.607600 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:20.607597    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44b94cc1-4817-4b62-9aed-8ced2c0ebb83" containerName="revoke-keys"
Oct 30 00:50:20.607757 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:20.607691    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="44b94cc1-4817-4b62-9aed-8ced2c0ebb83" containerName="revoke-keys"
Oct 30 00:50:20.611622 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:20.611207    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-9cjs5"
Oct 30 00:50:20.644013 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:20.643975    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-9cjs5"]
Oct 30 00:50:20.685116 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:20.684946    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34603daa-226c-4698-9bb5-774255e781a6-workload-certs\") pod \"e2e-worker-w3-cefb27-openshift-default-577b649d97-9cjs5\" (UID: \"34603daa-226c-4698-9bb5-774255e781a6\") " pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-9cjs5"
Oct 30 00:50:20.685116 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:20.685014    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34603daa-226c-4698-9bb5-774255e781a6-istio-data\") pod \"e2e-worker-w3-cefb27-openshift-default-577b649d97-9cjs5\" (UID: \"34603daa-226c-4698-9bb5-774255e781a6\") " pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-9cjs5"
Oct 30 00:50:20.685116 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:20.685075    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34603daa-226c-4698-9bb5-774255e781a6-credential-socket\") pod \"e2e-worker-w3-cefb27-openshift-default-577b649d97-9cjs5\" (UID: \"34603daa-226c-4698-9bb5-774255e781a6\") " pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-9cjs5"
Oct 30 00:50:20.685116 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:20.685108    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34603daa-226c-4698-9bb5-774255e781a6-istio-podinfo\") pod \"e2e-worker-w3-cefb27-openshift-default-577b649d97-9cjs5\" (UID: \"34603daa-226c-4698-9bb5-774255e781a6\") " pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-9cjs5"
Oct 30 00:50:20.685116 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:20.685123    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34603daa-226c-4698-9bb5-774255e781a6-istio-token\") pod \"e2e-worker-w3-cefb27-openshift-default-577b649d97-9cjs5\" (UID: \"34603daa-226c-4698-9bb5-774255e781a6\") " pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-9cjs5"
Oct 30 00:50:20.685469 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:20.685151    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34603daa-226c-4698-9bb5-774255e781a6-istiod-ca-cert\") pod \"e2e-worker-w3-cefb27-openshift-default-577b649d97-9cjs5\" (UID: \"34603daa-226c-4698-9bb5-774255e781a6\") " pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-9cjs5"
Oct 30 00:50:20.685469 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:20.685179    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34603daa-226c-4698-9bb5-774255e781a6-workload-socket\") pod \"e2e-worker-w3-cefb27-openshift-default-577b649d97-9cjs5\" (UID: \"34603daa-226c-4698-9bb5-774255e781a6\") " pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-9cjs5"
Oct 30 00:50:20.685469 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:20.685194    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34603daa-226c-4698-9bb5-774255e781a6-istio-envoy\") pod \"e2e-worker-w3-cefb27-openshift-default-577b649d97-9cjs5\" (UID: \"34603daa-226c-4698-9bb5-774255e781a6\") " pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-9cjs5"
Oct 30 00:50:20.685469 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:20.685227    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ksfpr\" (UniqueName: \"kubernetes.io/projected/34603daa-226c-4698-9bb5-774255e781a6-kube-api-access-ksfpr\") pod \"e2e-worker-w3-cefb27-openshift-default-577b649d97-9cjs5\" (UID: \"34603daa-226c-4698-9bb5-774255e781a6\") " pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-9cjs5"
Oct 30 00:50:20.767841 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:20.767803    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7d8z4"]
Oct 30 00:50:20.786537 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:20.786498    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34603daa-226c-4698-9bb5-774255e781a6-istio-data\") pod \"e2e-worker-w3-cefb27-openshift-default-577b649d97-9cjs5\" (UID: \"34603daa-226c-4698-9bb5-774255e781a6\") " pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-9cjs5"
Oct 30 00:50:20.786704 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:20.786636    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34603daa-226c-4698-9bb5-774255e781a6-credential-socket\") pod \"e2e-worker-w3-cefb27-openshift-default-577b649d97-9cjs5\" (UID: \"34603daa-226c-4698-9bb5-774255e781a6\") " pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-9cjs5"
Oct 30 00:50:20.786704 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:20.786692    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34603daa-226c-4698-9bb5-774255e781a6-istio-podinfo\") pod \"e2e-worker-w3-cefb27-openshift-default-577b649d97-9cjs5\" (UID: \"34603daa-226c-4698-9bb5-774255e781a6\") " pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-9cjs5"
Oct 30 00:50:20.786844 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:20.786818    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34603daa-226c-4698-9bb5-774255e781a6-istio-token\") pod \"e2e-worker-w3-cefb27-openshift-default-577b649d97-9cjs5\" (UID: \"34603daa-226c-4698-9bb5-774255e781a6\") " pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-9cjs5"
Oct 30 00:50:20.786903 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:20.786863    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34603daa-226c-4698-9bb5-774255e781a6-istio-data\") pod \"e2e-worker-w3-cefb27-openshift-default-577b649d97-9cjs5\" (UID: \"34603daa-226c-4698-9bb5-774255e781a6\") " pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-9cjs5"
Oct 30 00:50:20.786903 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:20.786893    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34603daa-226c-4698-9bb5-774255e781a6-istiod-ca-cert\") pod \"e2e-worker-w3-cefb27-openshift-default-577b649d97-9cjs5\" (UID: \"34603daa-226c-4698-9bb5-774255e781a6\") " pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-9cjs5"
Oct 30 00:50:20.787010 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:20.786941    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34603daa-226c-4698-9bb5-774255e781a6-workload-socket\") pod \"e2e-worker-w3-cefb27-openshift-default-577b649d97-9cjs5\" (UID: \"34603daa-226c-4698-9bb5-774255e781a6\") " pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-9cjs5"
Oct 30 00:50:20.787010 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:20.786962    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34603daa-226c-4698-9bb5-774255e781a6-istio-envoy\") pod \"e2e-worker-w3-cefb27-openshift-default-577b649d97-9cjs5\" (UID: \"34603daa-226c-4698-9bb5-774255e781a6\") " pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-9cjs5"
Oct 30 00:50:20.787113 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:20.787014    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ksfpr\" (UniqueName: \"kubernetes.io/projected/34603daa-226c-4698-9bb5-774255e781a6-kube-api-access-ksfpr\") pod \"e2e-worker-w3-cefb27-openshift-default-577b649d97-9cjs5\" (UID: \"34603daa-226c-4698-9bb5-774255e781a6\") " pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-9cjs5"
Oct 30 00:50:20.787113 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:20.787018    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34603daa-226c-4698-9bb5-774255e781a6-credential-socket\") pod \"e2e-worker-w3-cefb27-openshift-default-577b649d97-9cjs5\" (UID: \"34603daa-226c-4698-9bb5-774255e781a6\") " pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-9cjs5"
Oct 30 00:50:20.787209 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:20.787170    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34603daa-226c-4698-9bb5-774255e781a6-workload-certs\") pod \"e2e-worker-w3-cefb27-openshift-default-577b649d97-9cjs5\" (UID: \"34603daa-226c-4698-9bb5-774255e781a6\") " pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-9cjs5"
Oct 30 00:50:20.787333 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:20.787312    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34603daa-226c-4698-9bb5-774255e781a6-workload-socket\") pod \"e2e-worker-w3-cefb27-openshift-default-577b649d97-9cjs5\" (UID: \"34603daa-226c-4698-9bb5-774255e781a6\") " pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-9cjs5"
Oct 30 00:50:20.787596 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:20.787575    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34603daa-226c-4698-9bb5-774255e781a6-workload-certs\") pod \"e2e-worker-w3-cefb27-openshift-default-577b649d97-9cjs5\" (UID: \"34603daa-226c-4698-9bb5-774255e781a6\") " pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-9cjs5"
Oct 30 00:50:20.788008 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:20.787985    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34603daa-226c-4698-9bb5-774255e781a6-istiod-ca-cert\") pod \"e2e-worker-w3-cefb27-openshift-default-577b649d97-9cjs5\" (UID: \"34603daa-226c-4698-9bb5-774255e781a6\") " pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-9cjs5"
Oct 30 00:50:20.789697 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:20.789668    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34603daa-226c-4698-9bb5-774255e781a6-istio-envoy\") pod \"e2e-worker-w3-cefb27-openshift-default-577b649d97-9cjs5\" (UID: \"34603daa-226c-4698-9bb5-774255e781a6\") " pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-9cjs5"
Oct 30 00:50:20.789923 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:20.789906    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34603daa-226c-4698-9bb5-774255e781a6-istio-podinfo\") pod \"e2e-worker-w3-cefb27-openshift-default-577b649d97-9cjs5\" (UID: \"34603daa-226c-4698-9bb5-774255e781a6\") " pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-9cjs5"
Oct 30 00:50:20.829414 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:20.826734    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34603daa-226c-4698-9bb5-774255e781a6-istio-token\") pod \"e2e-worker-w3-cefb27-openshift-default-577b649d97-9cjs5\" (UID: \"34603daa-226c-4698-9bb5-774255e781a6\") " pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-9cjs5"
Oct 30 00:50:20.830250 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:20.829598    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ksfpr\" (UniqueName: \"kubernetes.io/projected/34603daa-226c-4698-9bb5-774255e781a6-kube-api-access-ksfpr\") pod \"e2e-worker-w3-cefb27-openshift-default-577b649d97-9cjs5\" (UID: \"34603daa-226c-4698-9bb5-774255e781a6\") " pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-9cjs5"
Oct 30 00:50:20.861027 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:20.860942    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-c85z8"
Oct 30 00:50:20.930066 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:20.930033    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-9cjs5"
Oct 30 00:50:20.989463 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:20.989327    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/792cb412-63fa-4eab-b70a-b6969a02897e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/792cb412-63fa-4eab-b70a-b6969a02897e-istio-envoy\") pod \"792cb412-63fa-4eab-b70a-b6969a02897e\" (UID: \"792cb412-63fa-4eab-b70a-b6969a02897e\") "
Oct 30 00:50:20.989630 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:20.989490    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/792cb412-63fa-4eab-b70a-b6969a02897e-istio-token\" (UniqueName: \"kubernetes.io/projected/792cb412-63fa-4eab-b70a-b6969a02897e-istio-token\") pod \"792cb412-63fa-4eab-b70a-b6969a02897e\" (UID: \"792cb412-63fa-4eab-b70a-b6969a02897e\") "
Oct 30 00:50:20.989630 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:20.989539    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/792cb412-63fa-4eab-b70a-b6969a02897e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/792cb412-63fa-4eab-b70a-b6969a02897e-workload-socket\") pod \"792cb412-63fa-4eab-b70a-b6969a02897e\" (UID: \"792cb412-63fa-4eab-b70a-b6969a02897e\") "
Oct 30 00:50:20.989630 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:20.989593    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/792cb412-63fa-4eab-b70a-b6969a02897e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/792cb412-63fa-4eab-b70a-b6969a02897e-istio-data\") pod \"792cb412-63fa-4eab-b70a-b6969a02897e\" (UID: \"792cb412-63fa-4eab-b70a-b6969a02897e\") "
Oct 30 00:50:20.989812 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:20.989630    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/792cb412-63fa-4eab-b70a-b6969a02897e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/792cb412-63fa-4eab-b70a-b6969a02897e-workload-certs\") pod \"792cb412-63fa-4eab-b70a-b6969a02897e\" (UID: \"792cb412-63fa-4eab-b70a-b6969a02897e\") "
Oct 30 00:50:20.989812 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:20.989687    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/792cb412-63fa-4eab-b70a-b6969a02897e-kube-api-access-k2png\" (UniqueName: \"kubernetes.io/projected/792cb412-63fa-4eab-b70a-b6969a02897e-kube-api-access-k2png\") pod \"792cb412-63fa-4eab-b70a-b6969a02897e\" (UID: \"792cb412-63fa-4eab-b70a-b6969a02897e\") "
Oct 30 00:50:20.989812 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:20.989765    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/792cb412-63fa-4eab-b70a-b6969a02897e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/792cb412-63fa-4eab-b70a-b6969a02897e-credential-socket\") pod \"792cb412-63fa-4eab-b70a-b6969a02897e\" (UID: \"792cb412-63fa-4eab-b70a-b6969a02897e\") "
Oct 30 00:50:20.989959 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:20.989845    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/792cb412-63fa-4eab-b70a-b6969a02897e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/792cb412-63fa-4eab-b70a-b6969a02897e-istiod-ca-cert\") pod \"792cb412-63fa-4eab-b70a-b6969a02897e\" (UID: \"792cb412-63fa-4eab-b70a-b6969a02897e\") "
Oct 30 00:50:20.989959 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:20.989902    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/792cb412-63fa-4eab-b70a-b6969a02897e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/792cb412-63fa-4eab-b70a-b6969a02897e-istio-podinfo\") pod \"792cb412-63fa-4eab-b70a-b6969a02897e\" (UID: \"792cb412-63fa-4eab-b70a-b6969a02897e\") "
Oct 30 00:50:20.989959 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:20.989912    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/792cb412-63fa-4eab-b70a-b6969a02897e-workload-socket" pod "792cb412-63fa-4eab-b70a-b6969a02897e" (UID: "792cb412-63fa-4eab-b70a-b6969a02897e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:50:20.990777 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:20.990272    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/792cb412-63fa-4eab-b70a-b6969a02897e-istio-data" pod "792cb412-63fa-4eab-b70a-b6969a02897e" (UID: "792cb412-63fa-4eab-b70a-b6969a02897e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:50:20.990777 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:20.990649    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/792cb412-63fa-4eab-b70a-b6969a02897e-workload-certs" pod "792cb412-63fa-4eab-b70a-b6969a02897e" (UID: "792cb412-63fa-4eab-b70a-b6969a02897e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:50:20.990777 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:20.990706    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/792cb412-63fa-4eab-b70a-b6969a02897e-credential-socket" pod "792cb412-63fa-4eab-b70a-b6969a02897e" (UID: "792cb412-63fa-4eab-b70a-b6969a02897e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:50:20.990777 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:20.990707    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/792cb412-63fa-4eab-b70a-b6969a02897e-istiod-ca-cert" pod "792cb412-63fa-4eab-b70a-b6969a02897e" (UID: "792cb412-63fa-4eab-b70a-b6969a02897e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:50:20.990777 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:20.990733    5002 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/792cb412-63fa-4eab-b70a-b6969a02897e-workload-socket\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:50:20.990777 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:20.990751    5002 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/792cb412-63fa-4eab-b70a-b6969a02897e-istio-data\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:50:20.993703 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:20.993084    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/792cb412-63fa-4eab-b70a-b6969a02897e-istio-podinfo" pod "792cb412-63fa-4eab-b70a-b6969a02897e" (UID: "792cb412-63fa-4eab-b70a-b6969a02897e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 00:50:20.999859 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:20.996752    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/792cb412-63fa-4eab-b70a-b6969a02897e-kube-api-access-k2png" pod "792cb412-63fa-4eab-b70a-b6969a02897e" (UID: "792cb412-63fa-4eab-b70a-b6969a02897e"). InnerVolumeSpecName "kube-api-access-k2png". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:50:21.001529 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:21.001503    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/792cb412-63fa-4eab-b70a-b6969a02897e-istio-token" pod "792cb412-63fa-4eab-b70a-b6969a02897e" (UID: "792cb412-63fa-4eab-b70a-b6969a02897e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:50:21.001529 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:21.001507    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/792cb412-63fa-4eab-b70a-b6969a02897e-istio-envoy" pod "792cb412-63fa-4eab-b70a-b6969a02897e" (UID: "792cb412-63fa-4eab-b70a-b6969a02897e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:50:21.092364 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:21.092329    5002 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/792cb412-63fa-4eab-b70a-b6969a02897e-istiod-ca-cert\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:50:21.092364 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:21.092362    5002 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/792cb412-63fa-4eab-b70a-b6969a02897e-istio-podinfo\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:50:21.092364 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:21.092371    5002 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/792cb412-63fa-4eab-b70a-b6969a02897e-istio-envoy\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:50:21.092364 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:21.092394    5002 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/792cb412-63fa-4eab-b70a-b6969a02897e-istio-token\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:50:21.092645 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:21.092408    5002 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/792cb412-63fa-4eab-b70a-b6969a02897e-workload-certs\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:50:21.092645 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:21.092422    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k2png\" (UniqueName: \"kubernetes.io/projected/792cb412-63fa-4eab-b70a-b6969a02897e-kube-api-access-k2png\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:50:21.092645 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:21.092432    5002 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/792cb412-63fa-4eab-b70a-b6969a02897e-credential-socket\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:50:21.264668 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:50:21.263803    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34603daa_226c_4698_9bb5_774255e781a6.slice/crio-8f8af9b5c86b01211e93c20eb1a4ecc1794787267f59ab68393e3a4cfea38f32 WatchSource:0}: Error finding container 8f8af9b5c86b01211e93c20eb1a4ecc1794787267f59ab68393e3a4cfea38f32: Status 404 returned error can't find the container with id 8f8af9b5c86b01211e93c20eb1a4ecc1794787267f59ab68393e3a4cfea38f32
Oct 30 00:50:21.267083 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:21.267041    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 00:50:21.267229 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:21.267121    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 00:50:21.267229 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:21.267157    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 00:50:21.267357 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:21.267328    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-9cjs5"]
Oct 30 00:50:21.322751 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:21.322716    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-9cjs5" event={"ID":"34603daa-226c-4698-9bb5-774255e781a6","Type":"ContainerStarted","Data":"8f8af9b5c86b01211e93c20eb1a4ecc1794787267f59ab68393e3a4cfea38f32"}
Oct 30 00:50:21.324339 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:21.324318    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-c85z8"
Oct 30 00:50:21.324475 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:21.324327    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-c85z8" event={"ID":"792cb412-63fa-4eab-b70a-b6969a02897e","Type":"ContainerDied","Data":"f170a2cd29bb93de9bb7cba86067b7dbcefadec217b0ca9bb820fb6b62d69ca0"}
Oct 30 00:50:21.324475 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:21.324414    5002 scope.go:122] "RemoveContainer" containerID="36e6c2f97204a877e63e47faf015d2016acf0464d823a74500d8a90be5afae3c"
Oct 30 00:50:21.466804 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:21.466771    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-c85z8"]
Oct 30 00:50:21.479036 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:21.479005    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-c85z8"]
Oct 30 00:50:22.389411 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:22.380987    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-9cjs5" event={"ID":"34603daa-226c-4698-9bb5-774255e781a6","Type":"ContainerStarted","Data":"a5698f3ca1de1d0c16a72b38175949d2427581fb55270e1bd94653fe36131191"}
Oct 30 00:50:22.468441 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:22.467949    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-9cjs5" podStartSLOduration=2.467922179 podStartE2EDuration="2.467922179s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:50:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:50:22.448190141 +0000 UTC m=+1118.064623032" watchObservedRunningTime="2026-10-30 00:50:22.467922179 +0000 UTC m=+1118.084355058"
Oct 30 00:50:22.535439 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:22.531336    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="792cb412-63fa-4eab-b70a-b6969a02897e" path="/var/lib/kubelet/pods/792cb412-63fa-4eab-b70a-b6969a02897e/volumes"
Oct 30 00:50:22.935401 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:22.933649    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-9cjs5"
Oct 30 00:50:22.935401 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:22.933781    5002 patch_prober.go:29] interesting pod/e2e-worker-w3-cefb27-openshift-default-577b649d97-9cjs5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Oct 30 00:50:22.935401 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:22.933827    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-9cjs5" podUID="34603daa-226c-4698-9bb5-774255e781a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Oct 30 00:50:23.936402 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:23.935811    5002 patch_prober.go:29] interesting pod/e2e-worker-w3-cefb27-openshift-default-577b649d97-9cjs5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Oct 30 00:50:23.936402 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:23.935901    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-9cjs5" podUID="34603daa-226c-4698-9bb5-774255e781a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Oct 30 00:50:24.317462 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:24.313549    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1a11cf/distinct-2-w2-1a11cf-kserve-758bb6dd57-j5f2k" podUID="f2d66dea-9f0b-43f3-a774-bb065e88f378" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:50:24.931091 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:24.931052    5002 patch_prober.go:29] interesting pod/e2e-worker-w3-cefb27-openshift-default-577b649d97-9cjs5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Oct 30 00:50:24.931319 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:24.931107    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-9cjs5" podUID="34603daa-226c-4698-9bb5-774255e781a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Oct 30 00:50:25.931258 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:25.931167    5002 patch_prober.go:29] interesting pod/e2e-worker-w3-cefb27-openshift-default-577b649d97-9cjs5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Oct 30 00:50:25.931258 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:25.931226    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-9cjs5" podUID="34603daa-226c-4698-9bb5-774255e781a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Oct 30 00:50:26.930933 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:26.930897    5002 patch_prober.go:29] interesting pod/e2e-worker-w3-cefb27-openshift-default-577b649d97-9cjs5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Oct 30 00:50:26.931113 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:26.930950    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-9cjs5" podUID="34603daa-226c-4698-9bb5-774255e781a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Oct 30 00:50:27.930617 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:27.930575    5002 patch_prober.go:29] interesting pod/e2e-worker-w3-cefb27-openshift-default-577b649d97-9cjs5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Oct 30 00:50:27.930990 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:27.930646    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-9cjs5" podUID="34603daa-226c-4698-9bb5-774255e781a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Oct 30 00:50:29.031062 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:29.031031    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-9cjs5"
Oct 30 00:50:29.032151 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:29.032122    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-9cjs5"
Oct 30 00:50:29.033120 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:29.033100    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-9cjs5"
Oct 30 00:50:35.730300 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:35.730261    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-thfqc"]
Oct 30 00:50:35.731276 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:35.730494    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-thfqc" podUID="40800c99-ad9a-4219-8e6e-f9aa7fd77f76" containerName="istio-proxy" containerID="cri-o://2a0a4d1434b7cd0d24d70350b5d1ad6d6529cec0c484fbc73c215b8fb04b8fac" gracePeriod=30
Oct 30 00:50:37.029238 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:37.029202    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0aed46c7-xvzsh"]
Oct 30 00:50:37.043469 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:37.043421    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0aed46c7-xvzsh"]
Oct 30 00:50:38.496695 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:38.496662    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7c497d9-dbad-44f0-8e38-8f275d3f0410" path="/var/lib/kubelet/pods/c7c497d9-dbad-44f0-8e38-8f275d3f0410/volumes"
Oct 30 00:50:41.023826 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:41.023800    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-thfqc"
Oct 30 00:50:41.097915 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:41.097831    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40800c99-ad9a-4219-8e6e-f9aa7fd77f76-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40800c99-ad9a-4219-8e6e-f9aa7fd77f76-istiod-ca-cert\") pod \"40800c99-ad9a-4219-8e6e-f9aa7fd77f76\" (UID: \"40800c99-ad9a-4219-8e6e-f9aa7fd77f76\") "
Oct 30 00:50:41.098102 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:41.097945    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/40800c99-ad9a-4219-8e6e-f9aa7fd77f76-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40800c99-ad9a-4219-8e6e-f9aa7fd77f76-istio-podinfo\") pod \"40800c99-ad9a-4219-8e6e-f9aa7fd77f76\" (UID: \"40800c99-ad9a-4219-8e6e-f9aa7fd77f76\") "
Oct 30 00:50:41.098102 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:41.097996    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40800c99-ad9a-4219-8e6e-f9aa7fd77f76-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40800c99-ad9a-4219-8e6e-f9aa7fd77f76-workload-socket\") pod \"40800c99-ad9a-4219-8e6e-f9aa7fd77f76\" (UID: \"40800c99-ad9a-4219-8e6e-f9aa7fd77f76\") "
Oct 30 00:50:41.098238 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:41.098150    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40800c99-ad9a-4219-8e6e-f9aa7fd77f76-kube-api-access-kdmnf\" (UniqueName: \"kubernetes.io/projected/40800c99-ad9a-4219-8e6e-f9aa7fd77f76-kube-api-access-kdmnf\") pod \"40800c99-ad9a-4219-8e6e-f9aa7fd77f76\" (UID: \"40800c99-ad9a-4219-8e6e-f9aa7fd77f76\") "
Oct 30 00:50:41.098238 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:41.098181    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40800c99-ad9a-4219-8e6e-f9aa7fd77f76-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40800c99-ad9a-4219-8e6e-f9aa7fd77f76-istio-envoy\") pod \"40800c99-ad9a-4219-8e6e-f9aa7fd77f76\" (UID: \"40800c99-ad9a-4219-8e6e-f9aa7fd77f76\") "
Oct 30 00:50:41.098238 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:41.098205    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40800c99-ad9a-4219-8e6e-f9aa7fd77f76-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40800c99-ad9a-4219-8e6e-f9aa7fd77f76-istio-data\") pod \"40800c99-ad9a-4219-8e6e-f9aa7fd77f76\" (UID: \"40800c99-ad9a-4219-8e6e-f9aa7fd77f76\") "
Oct 30 00:50:41.098238 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:41.098223    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40800c99-ad9a-4219-8e6e-f9aa7fd77f76-istio-token\" (UniqueName: \"kubernetes.io/projected/40800c99-ad9a-4219-8e6e-f9aa7fd77f76-istio-token\") pod \"40800c99-ad9a-4219-8e6e-f9aa7fd77f76\" (UID: \"40800c99-ad9a-4219-8e6e-f9aa7fd77f76\") "
Oct 30 00:50:41.098497 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:41.098258    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40800c99-ad9a-4219-8e6e-f9aa7fd77f76-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40800c99-ad9a-4219-8e6e-f9aa7fd77f76-credential-socket\") pod \"40800c99-ad9a-4219-8e6e-f9aa7fd77f76\" (UID: \"40800c99-ad9a-4219-8e6e-f9aa7fd77f76\") "
Oct 30 00:50:41.098497 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:41.098282    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40800c99-ad9a-4219-8e6e-f9aa7fd77f76-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40800c99-ad9a-4219-8e6e-f9aa7fd77f76-workload-certs\") pod \"40800c99-ad9a-4219-8e6e-f9aa7fd77f76\" (UID: \"40800c99-ad9a-4219-8e6e-f9aa7fd77f76\") "
Oct 30 00:50:41.098497 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:41.098304    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40800c99-ad9a-4219-8e6e-f9aa7fd77f76-istiod-ca-cert" pod "40800c99-ad9a-4219-8e6e-f9aa7fd77f76" (UID: "40800c99-ad9a-4219-8e6e-f9aa7fd77f76"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:50:41.098864 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:41.098834    5002 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40800c99-ad9a-4219-8e6e-f9aa7fd77f76-istiod-ca-cert\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:50:41.099057 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:41.098914    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40800c99-ad9a-4219-8e6e-f9aa7fd77f76-credential-socket" pod "40800c99-ad9a-4219-8e6e-f9aa7fd77f76" (UID: "40800c99-ad9a-4219-8e6e-f9aa7fd77f76"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:50:41.099271 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:41.099245    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40800c99-ad9a-4219-8e6e-f9aa7fd77f76-istio-data" pod "40800c99-ad9a-4219-8e6e-f9aa7fd77f76" (UID: "40800c99-ad9a-4219-8e6e-f9aa7fd77f76"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:50:41.099425 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:41.099360    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40800c99-ad9a-4219-8e6e-f9aa7fd77f76-workload-socket" pod "40800c99-ad9a-4219-8e6e-f9aa7fd77f76" (UID: "40800c99-ad9a-4219-8e6e-f9aa7fd77f76"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:50:41.099716 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:41.099683    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40800c99-ad9a-4219-8e6e-f9aa7fd77f76-workload-certs" pod "40800c99-ad9a-4219-8e6e-f9aa7fd77f76" (UID: "40800c99-ad9a-4219-8e6e-f9aa7fd77f76"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:50:41.101200 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:41.101172    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40800c99-ad9a-4219-8e6e-f9aa7fd77f76-istio-token" pod "40800c99-ad9a-4219-8e6e-f9aa7fd77f76" (UID: "40800c99-ad9a-4219-8e6e-f9aa7fd77f76"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:50:41.102494 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:41.101271    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40800c99-ad9a-4219-8e6e-f9aa7fd77f76-istio-envoy" pod "40800c99-ad9a-4219-8e6e-f9aa7fd77f76" (UID: "40800c99-ad9a-4219-8e6e-f9aa7fd77f76"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:50:41.102494 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:41.101942    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40800c99-ad9a-4219-8e6e-f9aa7fd77f76-kube-api-access-kdmnf" pod "40800c99-ad9a-4219-8e6e-f9aa7fd77f76" (UID: "40800c99-ad9a-4219-8e6e-f9aa7fd77f76"). InnerVolumeSpecName "kube-api-access-kdmnf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:50:41.103540 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:41.103510    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/40800c99-ad9a-4219-8e6e-f9aa7fd77f76-istio-podinfo" pod "40800c99-ad9a-4219-8e6e-f9aa7fd77f76" (UID: "40800c99-ad9a-4219-8e6e-f9aa7fd77f76"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 00:50:41.201897 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:41.201738    5002 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40800c99-ad9a-4219-8e6e-f9aa7fd77f76-istio-podinfo\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:50:41.201897 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:41.201772    5002 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40800c99-ad9a-4219-8e6e-f9aa7fd77f76-workload-socket\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:50:41.201897 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:41.201786    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kdmnf\" (UniqueName: \"kubernetes.io/projected/40800c99-ad9a-4219-8e6e-f9aa7fd77f76-kube-api-access-kdmnf\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:50:41.201897 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:41.201800    5002 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40800c99-ad9a-4219-8e6e-f9aa7fd77f76-istio-envoy\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:50:41.201897 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:41.201816    5002 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40800c99-ad9a-4219-8e6e-f9aa7fd77f76-istio-data\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:50:41.201897 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:41.201831    5002 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40800c99-ad9a-4219-8e6e-f9aa7fd77f76-istio-token\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:50:41.201897 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:41.201845    5002 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40800c99-ad9a-4219-8e6e-f9aa7fd77f76-credential-socket\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:50:41.201897 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:41.201859    5002 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40800c99-ad9a-4219-8e6e-f9aa7fd77f76-workload-certs\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:50:41.548713 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:41.548682    5002 generic.go:356] "Generic (PLEG): container finished" podID="40800c99-ad9a-4219-8e6e-f9aa7fd77f76" containerID="2a0a4d1434b7cd0d24d70350b5d1ad6d6529cec0c484fbc73c215b8fb04b8fac" exitCode=0
Oct 30 00:50:41.548890 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:41.548727    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-thfqc"
Oct 30 00:50:41.548890 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:41.548728    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-thfqc" event={"ID":"40800c99-ad9a-4219-8e6e-f9aa7fd77f76","Type":"ContainerDied","Data":"2a0a4d1434b7cd0d24d70350b5d1ad6d6529cec0c484fbc73c215b8fb04b8fac"}
Oct 30 00:50:41.548890 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:41.548850    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-thfqc" event={"ID":"40800c99-ad9a-4219-8e6e-f9aa7fd77f76","Type":"ContainerDied","Data":"7d9d2e9078a95b347ccfa21d588532640246d97b649694fc03fb5f339522b15e"}
Oct 30 00:50:41.548890 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:41.548878    5002 scope.go:122] "RemoveContainer" containerID="2a0a4d1434b7cd0d24d70350b5d1ad6d6529cec0c484fbc73c215b8fb04b8fac"
Oct 30 00:50:41.560962 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:41.560942    5002 scope.go:122] "RemoveContainer" containerID="2a0a4d1434b7cd0d24d70350b5d1ad6d6529cec0c484fbc73c215b8fb04b8fac"
Oct 30 00:50:41.561259 ip-10-0-128-7 kubenswrapper[5002]: E1030 00:50:41.561240    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a0a4d1434b7cd0d24d70350b5d1ad6d6529cec0c484fbc73c215b8fb04b8fac\": container with ID starting with 2a0a4d1434b7cd0d24d70350b5d1ad6d6529cec0c484fbc73c215b8fb04b8fac not found: ID does not exist" containerID="2a0a4d1434b7cd0d24d70350b5d1ad6d6529cec0c484fbc73c215b8fb04b8fac"
Oct 30 00:50:41.561323 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:41.561269    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a0a4d1434b7cd0d24d70350b5d1ad6d6529cec0c484fbc73c215b8fb04b8fac"} err="failed to get container status \"2a0a4d1434b7cd0d24d70350b5d1ad6d6529cec0c484fbc73c215b8fb04b8fac\": rpc error: code = NotFound desc = could not find container \"2a0a4d1434b7cd0d24d70350b5d1ad6d6529cec0c484fbc73c215b8fb04b8fac\": container with ID starting with 2a0a4d1434b7cd0d24d70350b5d1ad6d6529cec0c484fbc73c215b8fb04b8fac not found: ID does not exist"
Oct 30 00:50:41.621576 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:41.621543    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-thfqc"]
Oct 30 00:50:41.634750 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:41.634722    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-thfqc"]
Oct 30 00:50:42.495601 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:42.495565    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40800c99-ad9a-4219-8e6e-f9aa7fd77f76" path="/var/lib/kubelet/pods/40800c99-ad9a-4219-8e6e-f9aa7fd77f76/volumes"
Oct 30 00:50:42.866428 ip-10-0-128-7 kubenswrapper[5002]: E1030 00:50:42.866321    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbjdq" containerName="extractor"
Oct 30 00:50:48.737962 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:48.736929    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7d8z4"]
Oct 30 00:50:55.713067 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:55.713028    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f92e816-s28n5"]
Oct 30 00:50:55.714681 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:55.714655    5002 cpu_manager.go:414] "containerMap: removing container" podUID="40800c99-ad9a-4219-8e6e-f9aa7fd77f76" containerName="istio-proxy"
Oct 30 00:50:55.714844 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:55.714829    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40800c99-ad9a-4219-8e6e-f9aa7fd77f76" containerName="istio-proxy"
Oct 30 00:50:55.715315 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:55.715287    5002 cpu_manager.go:414] "containerMap: removing container" podUID="792cb412-63fa-4eab-b70a-b6969a02897e" containerName="istio-proxy"
Oct 30 00:50:55.715315 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:55.715312    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="792cb412-63fa-4eab-b70a-b6969a02897e" containerName="istio-proxy"
Oct 30 00:50:55.715533 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:55.715521    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="40800c99-ad9a-4219-8e6e-f9aa7fd77f76" containerName="istio-proxy"
Oct 30 00:50:55.715604 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:55.715537    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="792cb412-63fa-4eab-b70a-b6969a02897e" containerName="istio-proxy"
Oct 30 00:50:55.767675 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:55.767624    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f92e816-s28n5"]
Oct 30 00:50:55.767872 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:55.767780    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f92e816-s28n5"
Oct 30 00:50:55.793352 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:55.791665    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 00:50:55.802003 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:55.801887    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 00:50:55.807700 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:55.802276    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 00:50:55.850127 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:55.815641    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5phsh\""
Oct 30 00:50:55.851506 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:55.851394    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mt2f2\" (UniqueName: \"kubernetes.io/projected/87b20090-dc73-4f5b-a8bb-082d2a4365ec-kube-api-access-mt2f2\") pod \"maas-api-revoke-sub-6f92e816-s28n5\" (UID: \"87b20090-dc73-4f5b-a8bb-082d2a4365ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f92e816-s28n5"
Oct 30 00:50:55.851783 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:55.851762    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87b20090-dc73-4f5b-a8bb-082d2a4365ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6f92e816-s28n5\" (UID: \"87b20090-dc73-4f5b-a8bb-082d2a4365ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f92e816-s28n5"
Oct 30 00:50:55.851960 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:55.851943    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87b20090-dc73-4f5b-a8bb-082d2a4365ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6f92e816-s28n5\" (UID: \"87b20090-dc73-4f5b-a8bb-082d2a4365ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f92e816-s28n5"
Oct 30 00:50:55.954356 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:55.954319    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87b20090-dc73-4f5b-a8bb-082d2a4365ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6f92e816-s28n5\" (UID: \"87b20090-dc73-4f5b-a8bb-082d2a4365ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f92e816-s28n5"
Oct 30 00:50:55.954576 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:55.954433    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87b20090-dc73-4f5b-a8bb-082d2a4365ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6f92e816-s28n5\" (UID: \"87b20090-dc73-4f5b-a8bb-082d2a4365ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f92e816-s28n5"
Oct 30 00:50:55.954688 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:55.954669    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mt2f2\" (UniqueName: \"kubernetes.io/projected/87b20090-dc73-4f5b-a8bb-082d2a4365ec-kube-api-access-mt2f2\") pod \"maas-api-revoke-sub-6f92e816-s28n5\" (UID: \"87b20090-dc73-4f5b-a8bb-082d2a4365ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f92e816-s28n5"
Oct 30 00:50:55.955150 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:55.955099    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87b20090-dc73-4f5b-a8bb-082d2a4365ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6f92e816-s28n5\" (UID: \"87b20090-dc73-4f5b-a8bb-082d2a4365ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f92e816-s28n5"
Oct 30 00:50:55.957669 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:55.957642    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87b20090-dc73-4f5b-a8bb-082d2a4365ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6f92e816-s28n5\" (UID: \"87b20090-dc73-4f5b-a8bb-082d2a4365ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f92e816-s28n5"
Oct 30 00:50:55.983440 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:55.983319    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mt2f2\" (UniqueName: \"kubernetes.io/projected/87b20090-dc73-4f5b-a8bb-082d2a4365ec-kube-api-access-mt2f2\") pod \"maas-api-revoke-sub-6f92e816-s28n5\" (UID: \"87b20090-dc73-4f5b-a8bb-082d2a4365ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f92e816-s28n5"
Oct 30 00:50:56.098348 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:56.098302    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f92e816-s28n5"
Oct 30 00:50:56.419451 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:50:56.419407    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87b20090_dc73_4f5b_a8bb_082d2a4365ec.slice/crio-703833da37abeeb47b70da374dabdec10f1a8d3b9d4859e45e69980bd1ea4db3 WatchSource:0}: Error finding container 703833da37abeeb47b70da374dabdec10f1a8d3b9d4859e45e69980bd1ea4db3: Status 404 returned error can't find the container with id 703833da37abeeb47b70da374dabdec10f1a8d3b9d4859e45e69980bd1ea4db3
Oct 30 00:50:56.421115 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:56.421089    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f92e816-s28n5"]
Oct 30 00:50:56.735608 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:56.735570    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f92e816-s28n5" event={"ID":"87b20090-dc73-4f5b-a8bb-082d2a4365ec","Type":"ContainerStarted","Data":"703833da37abeeb47b70da374dabdec10f1a8d3b9d4859e45e69980bd1ea4db3"}
Oct 30 00:50:57.743774 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:57.743679    5002 generic.go:356] "Generic (PLEG): container finished" podID="87b20090-dc73-4f5b-a8bb-082d2a4365ec" containerID="b29ab1d27affa01ab294bcedff8fc6dcbdae7c06be30fd3267f99cd921c51f4d" exitCode=0
Oct 30 00:50:57.744172 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:57.743766    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f92e816-s28n5" event={"ID":"87b20090-dc73-4f5b-a8bb-082d2a4365ec","Type":"ContainerDied","Data":"b29ab1d27affa01ab294bcedff8fc6dcbdae7c06be30fd3267f99cd921c51f4d"}
Oct 30 00:50:58.942284 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:58.942259    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f92e816-s28n5"
Oct 30 00:50:59.104845 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:59.104761    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/87b20090-dc73-4f5b-a8bb-082d2a4365ec-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87b20090-dc73-4f5b-a8bb-082d2a4365ec-maas-api-xks-ca\") pod \"87b20090-dc73-4f5b-a8bb-082d2a4365ec\" (UID: \"87b20090-dc73-4f5b-a8bb-082d2a4365ec\") "
Oct 30 00:50:59.104990 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:59.104855    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87b20090-dc73-4f5b-a8bb-082d2a4365ec-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87b20090-dc73-4f5b-a8bb-082d2a4365ec-maas-api-openshift-ca\") pod \"87b20090-dc73-4f5b-a8bb-082d2a4365ec\" (UID: \"87b20090-dc73-4f5b-a8bb-082d2a4365ec\") "
Oct 30 00:50:59.105036 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:59.105021    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87b20090-dc73-4f5b-a8bb-082d2a4365ec-kube-api-access-mt2f2\" (UniqueName: \"kubernetes.io/projected/87b20090-dc73-4f5b-a8bb-082d2a4365ec-kube-api-access-mt2f2\") pod \"87b20090-dc73-4f5b-a8bb-082d2a4365ec\" (UID: \"87b20090-dc73-4f5b-a8bb-082d2a4365ec\") "
Oct 30 00:50:59.105249 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:59.105222    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87b20090-dc73-4f5b-a8bb-082d2a4365ec-maas-api-openshift-ca" pod "87b20090-dc73-4f5b-a8bb-082d2a4365ec" (UID: "87b20090-dc73-4f5b-a8bb-082d2a4365ec"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:50:59.105515 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:59.105492    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87b20090-dc73-4f5b-a8bb-082d2a4365ec-maas-api-openshift-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:50:59.106998 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:59.106965    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87b20090-dc73-4f5b-a8bb-082d2a4365ec-maas-api-xks-ca" pod "87b20090-dc73-4f5b-a8bb-082d2a4365ec" (UID: "87b20090-dc73-4f5b-a8bb-082d2a4365ec"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:50:59.107090 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:59.107033    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87b20090-dc73-4f5b-a8bb-082d2a4365ec-kube-api-access-mt2f2" pod "87b20090-dc73-4f5b-a8bb-082d2a4365ec" (UID: "87b20090-dc73-4f5b-a8bb-082d2a4365ec"). InnerVolumeSpecName "kube-api-access-mt2f2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:50:59.207024 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:59.206987    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mt2f2\" (UniqueName: \"kubernetes.io/projected/87b20090-dc73-4f5b-a8bb-082d2a4365ec-kube-api-access-mt2f2\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:50:59.207024 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:59.207016    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87b20090-dc73-4f5b-a8bb-082d2a4365ec-maas-api-xks-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:50:59.760168 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:59.760141    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f92e816-s28n5"
Oct 30 00:50:59.760373 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:59.760146    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f92e816-s28n5" event={"ID":"87b20090-dc73-4f5b-a8bb-082d2a4365ec","Type":"ContainerDied","Data":"703833da37abeeb47b70da374dabdec10f1a8d3b9d4859e45e69980bd1ea4db3"}
Oct 30 00:50:59.760373 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:50:59.760217    5002 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="703833da37abeeb47b70da374dabdec10f1a8d3b9d4859e45e69980bd1ea4db3"
Oct 30 00:51:01.398594 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:01.398557    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-be862c05-openshift-default-546c6fcf65-kfvq9"]
Oct 30 00:51:01.399664 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:01.399636    5002 cpu_manager.go:414] "containerMap: removing container" podUID="87b20090-dc73-4f5b-a8bb-082d2a4365ec" containerName="revoke-keys"
Oct 30 00:51:01.399664 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:01.399664    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87b20090-dc73-4f5b-a8bb-082d2a4365ec" containerName="revoke-keys"
Oct 30 00:51:01.399883 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:01.399823    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="87b20090-dc73-4f5b-a8bb-082d2a4365ec" containerName="revoke-keys"
Oct 30 00:51:01.403733 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:01.403713    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-be862c05-openshift-default-546c6fcf65-kfvq9"
Oct 30 00:51:01.420848 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:01.420826    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-be862c05-openshift-default-dockercfg-5t5dk\""
Oct 30 00:51:01.440986 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:01.440947    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-be862c05-openshift-default-546c6fcf65-kfvq9"]
Oct 30 00:51:01.536964 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:01.536923    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed3bd09f-a9e9-4da0-bd17-337a41c81f40-credential-socket\") pod \"e2e-derive-be862c05-openshift-default-546c6fcf65-kfvq9\" (UID: \"ed3bd09f-a9e9-4da0-bd17-337a41c81f40\") " pod="openshift-ingress/e2e-derive-be862c05-openshift-default-546c6fcf65-kfvq9"
Oct 30 00:51:01.537139 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:01.536982    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed3bd09f-a9e9-4da0-bd17-337a41c81f40-istio-token\") pod \"e2e-derive-be862c05-openshift-default-546c6fcf65-kfvq9\" (UID: \"ed3bd09f-a9e9-4da0-bd17-337a41c81f40\") " pod="openshift-ingress/e2e-derive-be862c05-openshift-default-546c6fcf65-kfvq9"
Oct 30 00:51:01.537189 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:01.537153    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed3bd09f-a9e9-4da0-bd17-337a41c81f40-istio-podinfo\") pod \"e2e-derive-be862c05-openshift-default-546c6fcf65-kfvq9\" (UID: \"ed3bd09f-a9e9-4da0-bd17-337a41c81f40\") " pod="openshift-ingress/e2e-derive-be862c05-openshift-default-546c6fcf65-kfvq9"
Oct 30 00:51:01.537239 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:01.537191    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed3bd09f-a9e9-4da0-bd17-337a41c81f40-istio-envoy\") pod \"e2e-derive-be862c05-openshift-default-546c6fcf65-kfvq9\" (UID: \"ed3bd09f-a9e9-4da0-bd17-337a41c81f40\") " pod="openshift-ingress/e2e-derive-be862c05-openshift-default-546c6fcf65-kfvq9"
Oct 30 00:51:01.537281 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:01.537253    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvsf5\" (UniqueName: \"kubernetes.io/projected/ed3bd09f-a9e9-4da0-bd17-337a41c81f40-kube-api-access-nvsf5\") pod \"e2e-derive-be862c05-openshift-default-546c6fcf65-kfvq9\" (UID: \"ed3bd09f-a9e9-4da0-bd17-337a41c81f40\") " pod="openshift-ingress/e2e-derive-be862c05-openshift-default-546c6fcf65-kfvq9"
Oct 30 00:51:01.537281 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:01.537276    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed3bd09f-a9e9-4da0-bd17-337a41c81f40-workload-certs\") pod \"e2e-derive-be862c05-openshift-default-546c6fcf65-kfvq9\" (UID: \"ed3bd09f-a9e9-4da0-bd17-337a41c81f40\") " pod="openshift-ingress/e2e-derive-be862c05-openshift-default-546c6fcf65-kfvq9"
Oct 30 00:51:01.537358 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:01.537292    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed3bd09f-a9e9-4da0-bd17-337a41c81f40-istiod-ca-cert\") pod \"e2e-derive-be862c05-openshift-default-546c6fcf65-kfvq9\" (UID: \"ed3bd09f-a9e9-4da0-bd17-337a41c81f40\") " pod="openshift-ingress/e2e-derive-be862c05-openshift-default-546c6fcf65-kfvq9"
Oct 30 00:51:01.537430 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:01.537413    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed3bd09f-a9e9-4da0-bd17-337a41c81f40-workload-socket\") pod \"e2e-derive-be862c05-openshift-default-546c6fcf65-kfvq9\" (UID: \"ed3bd09f-a9e9-4da0-bd17-337a41c81f40\") " pod="openshift-ingress/e2e-derive-be862c05-openshift-default-546c6fcf65-kfvq9"
Oct 30 00:51:01.537467 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:01.537433    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed3bd09f-a9e9-4da0-bd17-337a41c81f40-istio-data\") pod \"e2e-derive-be862c05-openshift-default-546c6fcf65-kfvq9\" (UID: \"ed3bd09f-a9e9-4da0-bd17-337a41c81f40\") " pod="openshift-ingress/e2e-derive-be862c05-openshift-default-546c6fcf65-kfvq9"
Oct 30 00:51:01.639089 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:01.639052    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed3bd09f-a9e9-4da0-bd17-337a41c81f40-istio-podinfo\") pod \"e2e-derive-be862c05-openshift-default-546c6fcf65-kfvq9\" (UID: \"ed3bd09f-a9e9-4da0-bd17-337a41c81f40\") " pod="openshift-ingress/e2e-derive-be862c05-openshift-default-546c6fcf65-kfvq9"
Oct 30 00:51:01.639274 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:01.639097    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed3bd09f-a9e9-4da0-bd17-337a41c81f40-istio-envoy\") pod \"e2e-derive-be862c05-openshift-default-546c6fcf65-kfvq9\" (UID: \"ed3bd09f-a9e9-4da0-bd17-337a41c81f40\") " pod="openshift-ingress/e2e-derive-be862c05-openshift-default-546c6fcf65-kfvq9"
Oct 30 00:51:01.639274 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:01.639148    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvsf5\" (UniqueName: \"kubernetes.io/projected/ed3bd09f-a9e9-4da0-bd17-337a41c81f40-kube-api-access-nvsf5\") pod \"e2e-derive-be862c05-openshift-default-546c6fcf65-kfvq9\" (UID: \"ed3bd09f-a9e9-4da0-bd17-337a41c81f40\") " pod="openshift-ingress/e2e-derive-be862c05-openshift-default-546c6fcf65-kfvq9"
Oct 30 00:51:01.639274 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:01.639171    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed3bd09f-a9e9-4da0-bd17-337a41c81f40-workload-certs\") pod \"e2e-derive-be862c05-openshift-default-546c6fcf65-kfvq9\" (UID: \"ed3bd09f-a9e9-4da0-bd17-337a41c81f40\") " pod="openshift-ingress/e2e-derive-be862c05-openshift-default-546c6fcf65-kfvq9"
Oct 30 00:51:01.639274 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:01.639187    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed3bd09f-a9e9-4da0-bd17-337a41c81f40-istiod-ca-cert\") pod \"e2e-derive-be862c05-openshift-default-546c6fcf65-kfvq9\" (UID: \"ed3bd09f-a9e9-4da0-bd17-337a41c81f40\") " pod="openshift-ingress/e2e-derive-be862c05-openshift-default-546c6fcf65-kfvq9"
Oct 30 00:51:01.639518 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:01.639321    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed3bd09f-a9e9-4da0-bd17-337a41c81f40-workload-socket\") pod \"e2e-derive-be862c05-openshift-default-546c6fcf65-kfvq9\" (UID: \"ed3bd09f-a9e9-4da0-bd17-337a41c81f40\") " pod="openshift-ingress/e2e-derive-be862c05-openshift-default-546c6fcf65-kfvq9"
Oct 30 00:51:01.639518 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:01.639393    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed3bd09f-a9e9-4da0-bd17-337a41c81f40-istio-data\") pod \"e2e-derive-be862c05-openshift-default-546c6fcf65-kfvq9\" (UID: \"ed3bd09f-a9e9-4da0-bd17-337a41c81f40\") " pod="openshift-ingress/e2e-derive-be862c05-openshift-default-546c6fcf65-kfvq9"
Oct 30 00:51:01.639751 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:01.639668    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed3bd09f-a9e9-4da0-bd17-337a41c81f40-workload-certs\") pod \"e2e-derive-be862c05-openshift-default-546c6fcf65-kfvq9\" (UID: \"ed3bd09f-a9e9-4da0-bd17-337a41c81f40\") " pod="openshift-ingress/e2e-derive-be862c05-openshift-default-546c6fcf65-kfvq9"
Oct 30 00:51:01.640069 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:01.640041    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed3bd09f-a9e9-4da0-bd17-337a41c81f40-istiod-ca-cert\") pod \"e2e-derive-be862c05-openshift-default-546c6fcf65-kfvq9\" (UID: \"ed3bd09f-a9e9-4da0-bd17-337a41c81f40\") " pod="openshift-ingress/e2e-derive-be862c05-openshift-default-546c6fcf65-kfvq9"
Oct 30 00:51:01.640175 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:01.640147    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed3bd09f-a9e9-4da0-bd17-337a41c81f40-credential-socket\") pod \"e2e-derive-be862c05-openshift-default-546c6fcf65-kfvq9\" (UID: \"ed3bd09f-a9e9-4da0-bd17-337a41c81f40\") " pod="openshift-ingress/e2e-derive-be862c05-openshift-default-546c6fcf65-kfvq9"
Oct 30 00:51:01.640369 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:01.640328    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed3bd09f-a9e9-4da0-bd17-337a41c81f40-istio-data\") pod \"e2e-derive-be862c05-openshift-default-546c6fcf65-kfvq9\" (UID: \"ed3bd09f-a9e9-4da0-bd17-337a41c81f40\") " pod="openshift-ingress/e2e-derive-be862c05-openshift-default-546c6fcf65-kfvq9"
Oct 30 00:51:01.640450 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:01.640367    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed3bd09f-a9e9-4da0-bd17-337a41c81f40-istio-token\") pod \"e2e-derive-be862c05-openshift-default-546c6fcf65-kfvq9\" (UID: \"ed3bd09f-a9e9-4da0-bd17-337a41c81f40\") " pod="openshift-ingress/e2e-derive-be862c05-openshift-default-546c6fcf65-kfvq9"
Oct 30 00:51:01.640513 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:01.640487    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed3bd09f-a9e9-4da0-bd17-337a41c81f40-credential-socket\") pod \"e2e-derive-be862c05-openshift-default-546c6fcf65-kfvq9\" (UID: \"ed3bd09f-a9e9-4da0-bd17-337a41c81f40\") " pod="openshift-ingress/e2e-derive-be862c05-openshift-default-546c6fcf65-kfvq9"
Oct 30 00:51:01.640776 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:01.640754    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed3bd09f-a9e9-4da0-bd17-337a41c81f40-workload-socket\") pod \"e2e-derive-be862c05-openshift-default-546c6fcf65-kfvq9\" (UID: \"ed3bd09f-a9e9-4da0-bd17-337a41c81f40\") " pod="openshift-ingress/e2e-derive-be862c05-openshift-default-546c6fcf65-kfvq9"
Oct 30 00:51:01.641924 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:01.641896    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed3bd09f-a9e9-4da0-bd17-337a41c81f40-istio-envoy\") pod \"e2e-derive-be862c05-openshift-default-546c6fcf65-kfvq9\" (UID: \"ed3bd09f-a9e9-4da0-bd17-337a41c81f40\") " pod="openshift-ingress/e2e-derive-be862c05-openshift-default-546c6fcf65-kfvq9"
Oct 30 00:51:01.642070 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:01.642048    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed3bd09f-a9e9-4da0-bd17-337a41c81f40-istio-podinfo\") pod \"e2e-derive-be862c05-openshift-default-546c6fcf65-kfvq9\" (UID: \"ed3bd09f-a9e9-4da0-bd17-337a41c81f40\") " pod="openshift-ingress/e2e-derive-be862c05-openshift-default-546c6fcf65-kfvq9"
Oct 30 00:51:01.664126 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:01.664064    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvsf5\" (UniqueName: \"kubernetes.io/projected/ed3bd09f-a9e9-4da0-bd17-337a41c81f40-kube-api-access-nvsf5\") pod \"e2e-derive-be862c05-openshift-default-546c6fcf65-kfvq9\" (UID: \"ed3bd09f-a9e9-4da0-bd17-337a41c81f40\") " pod="openshift-ingress/e2e-derive-be862c05-openshift-default-546c6fcf65-kfvq9"
Oct 30 00:51:01.675727 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:01.675695    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed3bd09f-a9e9-4da0-bd17-337a41c81f40-istio-token\") pod \"e2e-derive-be862c05-openshift-default-546c6fcf65-kfvq9\" (UID: \"ed3bd09f-a9e9-4da0-bd17-337a41c81f40\") " pod="openshift-ingress/e2e-derive-be862c05-openshift-default-546c6fcf65-kfvq9"
Oct 30 00:51:01.718301 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:01.718267    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-be862c05-openshift-default-546c6fcf65-kfvq9"
Oct 30 00:51:02.042105 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:02.042074    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-be862c05-openshift-default-546c6fcf65-kfvq9"]
Oct 30 00:51:02.043076 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:51:02.043006    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded3bd09f_a9e9_4da0_bd17_337a41c81f40.slice/crio-0b0d9c997a3947b27bf6820fdd02b0a9ad628851bba36747caaa1b1ddffbbb57 WatchSource:0}: Error finding container 0b0d9c997a3947b27bf6820fdd02b0a9ad628851bba36747caaa1b1ddffbbb57: Status 404 returned error can't find the container with id 0b0d9c997a3947b27bf6820fdd02b0a9ad628851bba36747caaa1b1ddffbbb57
Oct 30 00:51:02.046126 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:02.046072    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 00:51:02.046219 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:02.046173    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 00:51:02.046219 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:02.046213    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 00:51:02.786905 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:02.786865    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-be862c05-openshift-default-546c6fcf65-kfvq9" event={"ID":"ed3bd09f-a9e9-4da0-bd17-337a41c81f40","Type":"ContainerStarted","Data":"bc18c1e759758447f042aea25b0c8822476e63797558396094f1fce92a7a45a8"}
Oct 30 00:51:02.787370 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:02.786922    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-be862c05-openshift-default-546c6fcf65-kfvq9" event={"ID":"ed3bd09f-a9e9-4da0-bd17-337a41c81f40","Type":"ContainerStarted","Data":"0b0d9c997a3947b27bf6820fdd02b0a9ad628851bba36747caaa1b1ddffbbb57"}
Oct 30 00:51:02.859403 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:02.850968    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-be862c05-openshift-default-546c6fcf65-kfvq9" podStartSLOduration=1.850947658 podStartE2EDuration="1.850947658s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:51:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:51:02.84913852 +0000 UTC m=+1158.465571400" watchObservedRunningTime="2026-10-30 00:51:02.850947658 +0000 UTC m=+1158.467380540"
Oct 30 00:51:03.719064 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:03.719030    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-be862c05-openshift-default-546c6fcf65-kfvq9"
Oct 30 00:51:03.720356 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:03.720327    5002 patch_prober.go:29] interesting pod/e2e-derive-be862c05-openshift-default-546c6fcf65-kfvq9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Oct 30 00:51:03.720501 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:03.720400    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-be862c05-openshift-default-546c6fcf65-kfvq9" podUID="ed3bd09f-a9e9-4da0-bd17-337a41c81f40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Oct 30 00:51:04.719754 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:04.719717    5002 patch_prober.go:29] interesting pod/e2e-derive-be862c05-openshift-default-546c6fcf65-kfvq9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Oct 30 00:51:04.720146 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:04.719783    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-be862c05-openshift-default-546c6fcf65-kfvq9" podUID="ed3bd09f-a9e9-4da0-bd17-337a41c81f40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Oct 30 00:51:05.718828 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:05.718790    5002 patch_prober.go:29] interesting pod/e2e-derive-be862c05-openshift-default-546c6fcf65-kfvq9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Oct 30 00:51:05.719005 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:05.718872    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-be862c05-openshift-default-546c6fcf65-kfvq9" podUID="ed3bd09f-a9e9-4da0-bd17-337a41c81f40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Oct 30 00:51:05.838864 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:05.838828    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-t4s54"]
Oct 30 00:51:05.843250 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:05.843227    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-t4s54"
Oct 30 00:51:05.879776 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:05.879747    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-6f4fba-openshift-default-dockercfg-l52jt\""
Oct 30 00:51:05.885218 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:05.884589    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-t4s54"]
Oct 30 00:51:05.931642 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:05.931608    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a107188-58d7-4c62-a2e0-e81eb181d540-workload-socket\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-t4s54\" (UID: \"3a107188-58d7-4c62-a2e0-e81eb181d540\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-t4s54"
Oct 30 00:51:05.931811 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:05.931663    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a107188-58d7-4c62-a2e0-e81eb181d540-istio-envoy\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-t4s54\" (UID: \"3a107188-58d7-4c62-a2e0-e81eb181d540\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-t4s54"
Oct 30 00:51:05.931811 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:05.931792    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a107188-58d7-4c62-a2e0-e81eb181d540-istio-podinfo\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-t4s54\" (UID: \"3a107188-58d7-4c62-a2e0-e81eb181d540\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-t4s54"
Oct 30 00:51:05.931922 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:05.931859    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cl67m\" (UniqueName: \"kubernetes.io/projected/3a107188-58d7-4c62-a2e0-e81eb181d540-kube-api-access-cl67m\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-t4s54\" (UID: \"3a107188-58d7-4c62-a2e0-e81eb181d540\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-t4s54"
Oct 30 00:51:05.931922 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:05.931909    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a107188-58d7-4c62-a2e0-e81eb181d540-istiod-ca-cert\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-t4s54\" (UID: \"3a107188-58d7-4c62-a2e0-e81eb181d540\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-t4s54"
Oct 30 00:51:05.931999 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:05.931956    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a107188-58d7-4c62-a2e0-e81eb181d540-istio-token\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-t4s54\" (UID: \"3a107188-58d7-4c62-a2e0-e81eb181d540\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-t4s54"
Oct 30 00:51:05.932047 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:05.932006    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a107188-58d7-4c62-a2e0-e81eb181d540-istio-data\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-t4s54\" (UID: \"3a107188-58d7-4c62-a2e0-e81eb181d540\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-t4s54"
Oct 30 00:51:05.932144 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:05.932127    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a107188-58d7-4c62-a2e0-e81eb181d540-credential-socket\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-t4s54\" (UID: \"3a107188-58d7-4c62-a2e0-e81eb181d540\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-t4s54"
Oct 30 00:51:05.932195 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:05.932171    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a107188-58d7-4c62-a2e0-e81eb181d540-workload-certs\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-t4s54\" (UID: \"3a107188-58d7-4c62-a2e0-e81eb181d540\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-t4s54"
Oct 30 00:51:06.054452 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:06.033524    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a107188-58d7-4c62-a2e0-e81eb181d540-istio-data\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-t4s54\" (UID: \"3a107188-58d7-4c62-a2e0-e81eb181d540\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-t4s54"
Oct 30 00:51:06.054452 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:06.033681    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a107188-58d7-4c62-a2e0-e81eb181d540-credential-socket\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-t4s54\" (UID: \"3a107188-58d7-4c62-a2e0-e81eb181d540\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-t4s54"
Oct 30 00:51:06.054452 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:06.033716    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a107188-58d7-4c62-a2e0-e81eb181d540-workload-certs\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-t4s54\" (UID: \"3a107188-58d7-4c62-a2e0-e81eb181d540\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-t4s54"
Oct 30 00:51:06.054452 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:06.033811    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a107188-58d7-4c62-a2e0-e81eb181d540-workload-socket\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-t4s54\" (UID: \"3a107188-58d7-4c62-a2e0-e81eb181d540\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-t4s54"
Oct 30 00:51:06.054452 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:06.033886    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a107188-58d7-4c62-a2e0-e81eb181d540-istio-envoy\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-t4s54\" (UID: \"3a107188-58d7-4c62-a2e0-e81eb181d540\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-t4s54"
Oct 30 00:51:06.054452 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:06.033968    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a107188-58d7-4c62-a2e0-e81eb181d540-istio-podinfo\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-t4s54\" (UID: \"3a107188-58d7-4c62-a2e0-e81eb181d540\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-t4s54"
Oct 30 00:51:06.054452 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:06.034020    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cl67m\" (UniqueName: \"kubernetes.io/projected/3a107188-58d7-4c62-a2e0-e81eb181d540-kube-api-access-cl67m\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-t4s54\" (UID: \"3a107188-58d7-4c62-a2e0-e81eb181d540\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-t4s54"
Oct 30 00:51:06.054452 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:06.034077    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a107188-58d7-4c62-a2e0-e81eb181d540-istiod-ca-cert\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-t4s54\" (UID: \"3a107188-58d7-4c62-a2e0-e81eb181d540\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-t4s54"
Oct 30 00:51:06.054452 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:06.034106    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a107188-58d7-4c62-a2e0-e81eb181d540-istio-token\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-t4s54\" (UID: \"3a107188-58d7-4c62-a2e0-e81eb181d540\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-t4s54"
Oct 30 00:51:06.054452 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:06.034942    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a107188-58d7-4c62-a2e0-e81eb181d540-istio-data\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-t4s54\" (UID: \"3a107188-58d7-4c62-a2e0-e81eb181d540\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-t4s54"
Oct 30 00:51:06.054452 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:06.035203    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a107188-58d7-4c62-a2e0-e81eb181d540-credential-socket\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-t4s54\" (UID: \"3a107188-58d7-4c62-a2e0-e81eb181d540\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-t4s54"
Oct 30 00:51:06.054452 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:06.035475    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a107188-58d7-4c62-a2e0-e81eb181d540-workload-certs\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-t4s54\" (UID: \"3a107188-58d7-4c62-a2e0-e81eb181d540\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-t4s54"
Oct 30 00:51:06.054452 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:06.036113    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a107188-58d7-4c62-a2e0-e81eb181d540-workload-socket\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-t4s54\" (UID: \"3a107188-58d7-4c62-a2e0-e81eb181d540\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-t4s54"
Oct 30 00:51:06.054452 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:06.036427    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a107188-58d7-4c62-a2e0-e81eb181d540-istiod-ca-cert\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-t4s54\" (UID: \"3a107188-58d7-4c62-a2e0-e81eb181d540\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-t4s54"
Oct 30 00:51:06.054452 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:06.039175    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a107188-58d7-4c62-a2e0-e81eb181d540-istio-podinfo\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-t4s54\" (UID: \"3a107188-58d7-4c62-a2e0-e81eb181d540\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-t4s54"
Oct 30 00:51:06.054452 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:06.045518    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a107188-58d7-4c62-a2e0-e81eb181d540-istio-envoy\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-t4s54\" (UID: \"3a107188-58d7-4c62-a2e0-e81eb181d540\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-t4s54"
Oct 30 00:51:06.096411 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:06.096009    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a107188-58d7-4c62-a2e0-e81eb181d540-istio-token\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-t4s54\" (UID: \"3a107188-58d7-4c62-a2e0-e81eb181d540\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-t4s54"
Oct 30 00:51:06.117431 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:06.101511    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cl67m\" (UniqueName: \"kubernetes.io/projected/3a107188-58d7-4c62-a2e0-e81eb181d540-kube-api-access-cl67m\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-t4s54\" (UID: \"3a107188-58d7-4c62-a2e0-e81eb181d540\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-t4s54"
Oct 30 00:51:06.188129 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:06.188091    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-t4s54"
Oct 30 00:51:06.722481 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:06.718725    5002 patch_prober.go:29] interesting pod/e2e-derive-be862c05-openshift-default-546c6fcf65-kfvq9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Oct 30 00:51:06.722481 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:06.718794    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-be862c05-openshift-default-546c6fcf65-kfvq9" podUID="ed3bd09f-a9e9-4da0-bd17-337a41c81f40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Oct 30 00:51:07.014514 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:07.013346    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-t4s54"]
Oct 30 00:51:07.018297 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:51:07.017423    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a107188_58d7_4c62_a2e0_e81eb181d540.slice/crio-e25fd4f1ced7f5c66f4fded538e156709c7bd25391a20e183518522213516415 WatchSource:0}: Error finding container e25fd4f1ced7f5c66f4fded538e156709c7bd25391a20e183518522213516415: Status 404 returned error can't find the container with id e25fd4f1ced7f5c66f4fded538e156709c7bd25391a20e183518522213516415
Oct 30 00:51:07.020874 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:07.020842    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 00:51:07.020977 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:07.020909    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 00:51:07.020977 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:07.020937    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 00:51:07.720522 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:07.720483    5002 patch_prober.go:29] interesting pod/e2e-derive-be862c05-openshift-default-546c6fcf65-kfvq9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Oct 30 00:51:07.720676 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:07.720569    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-be862c05-openshift-default-546c6fcf65-kfvq9" podUID="ed3bd09f-a9e9-4da0-bd17-337a41c81f40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Oct 30 00:51:07.919404 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:07.917361    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-t4s54" event={"ID":"3a107188-58d7-4c62-a2e0-e81eb181d540","Type":"ContainerStarted","Data":"cf7d7f440c1f4124242028ef09533f0122a4dc6177bbc75d06fc326e5c288fe1"}
Oct 30 00:51:07.919404 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:07.917499    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-t4s54" event={"ID":"3a107188-58d7-4c62-a2e0-e81eb181d540","Type":"ContainerStarted","Data":"e25fd4f1ced7f5c66f4fded538e156709c7bd25391a20e183518522213516415"}
Oct 30 00:51:08.015281 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:08.015159    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-t4s54" podStartSLOduration=3.015139766 podStartE2EDuration="3.015139766s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:51:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:51:07.983093256 +0000 UTC m=+1163.599526144" watchObservedRunningTime="2026-10-30 00:51:08.015139766 +0000 UTC m=+1163.631572644"
Oct 30 00:51:08.201403 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:08.197489    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-t4s54"
Oct 30 00:51:08.208972 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:08.208935    5002 patch_prober.go:29] interesting pod/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-t4s54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Oct 30 00:51:08.209279 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:08.209247    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-t4s54" podUID="3a107188-58d7-4c62-a2e0-e81eb181d540" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Oct 30 00:51:08.971403 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:08.968677    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-be862c05-openshift-default-546c6fcf65-kfvq9"
Oct 30 00:51:08.971403 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:08.970225    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-be862c05-openshift-default-546c6fcf65-kfvq9"
Oct 30 00:51:08.994405 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:08.993166    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-be862c05-openshift-default-546c6fcf65-kfvq9"
Oct 30 00:51:09.188936 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:09.188901    5002 patch_prober.go:29] interesting pod/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-t4s54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Oct 30 00:51:09.189321 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:09.188967    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-t4s54" podUID="3a107188-58d7-4c62-a2e0-e81eb181d540" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Oct 30 00:51:10.188891 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:10.188851    5002 patch_prober.go:29] interesting pod/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-t4s54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Oct 30 00:51:10.189049 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:10.188934    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-t4s54" podUID="3a107188-58d7-4c62-a2e0-e81eb181d540" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Oct 30 00:51:11.194401 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:11.193899    5002 patch_prober.go:29] interesting pod/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-t4s54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Oct 30 00:51:11.194401 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:11.193988    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-t4s54" podUID="3a107188-58d7-4c62-a2e0-e81eb181d540" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Oct 30 00:51:12.190720 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:12.190682    5002 patch_prober.go:29] interesting pod/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-t4s54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Oct 30 00:51:12.191030 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:12.190734    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-t4s54" podUID="3a107188-58d7-4c62-a2e0-e81eb181d540" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Oct 30 00:51:12.982405 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:12.969861    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-52650f/distinct-w0-52650f-kserve-d85899b5-vlhps" podUID="ea1850d9-abba-4977-a540-f6e0eaefb17d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:51:13.113006 ip-10-0-128-7 kubenswrapper[5002]: E1030 00:51:13.112969    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbjdq" containerName="extractor"
Oct 30 00:51:13.191784 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:13.188168    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-be862c05-openshift-default-546c6fcf65-kfvq9"]
Oct 30 00:51:13.192013 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:13.191926    5002 patch_prober.go:29] interesting pod/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-t4s54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Oct 30 00:51:13.192083 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:13.192012    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-t4s54" podUID="3a107188-58d7-4c62-a2e0-e81eb181d540" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Oct 30 00:51:13.227439 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:13.193885    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-be862c05-openshift-default-546c6fcf65-kfvq9" podUID="ed3bd09f-a9e9-4da0-bd17-337a41c81f40" containerName="istio-proxy" containerID="cri-o://bc18c1e759758447f042aea25b0c8822476e63797558396094f1fce92a7a45a8" gracePeriod=30
Oct 30 00:51:14.189235 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:14.189194    5002 patch_prober.go:29] interesting pod/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-t4s54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Oct 30 00:51:14.189643 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:14.189253    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-t4s54" podUID="3a107188-58d7-4c62-a2e0-e81eb181d540" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Oct 30 00:51:15.193415 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:15.193369    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-t4s54"
Oct 30 00:51:15.194104 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:15.194075    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-t4s54"
Oct 30 00:51:15.194657 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:15.194639    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-t4s54"
Oct 30 00:51:18.459481 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:18.459456    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-be862c05-openshift-default-546c6fcf65-kfvq9"
Oct 30 00:51:18.526564 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:18.526481    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed3bd09f-a9e9-4da0-bd17-337a41c81f40-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed3bd09f-a9e9-4da0-bd17-337a41c81f40-credential-socket\") pod \"ed3bd09f-a9e9-4da0-bd17-337a41c81f40\" (UID: \"ed3bd09f-a9e9-4da0-bd17-337a41c81f40\") "
Oct 30 00:51:18.526564 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:18.526540    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed3bd09f-a9e9-4da0-bd17-337a41c81f40-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed3bd09f-a9e9-4da0-bd17-337a41c81f40-istio-data\") pod \"ed3bd09f-a9e9-4da0-bd17-337a41c81f40\" (UID: \"ed3bd09f-a9e9-4da0-bd17-337a41c81f40\") "
Oct 30 00:51:18.526795 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:18.526578    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed3bd09f-a9e9-4da0-bd17-337a41c81f40-istio-token\" (UniqueName: \"kubernetes.io/projected/ed3bd09f-a9e9-4da0-bd17-337a41c81f40-istio-token\") pod \"ed3bd09f-a9e9-4da0-bd17-337a41c81f40\" (UID: \"ed3bd09f-a9e9-4da0-bd17-337a41c81f40\") "
Oct 30 00:51:18.526795 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:18.526621    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed3bd09f-a9e9-4da0-bd17-337a41c81f40-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed3bd09f-a9e9-4da0-bd17-337a41c81f40-workload-certs\") pod \"ed3bd09f-a9e9-4da0-bd17-337a41c81f40\" (UID: \"ed3bd09f-a9e9-4da0-bd17-337a41c81f40\") "
Oct 30 00:51:18.526795 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:18.526643    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed3bd09f-a9e9-4da0-bd17-337a41c81f40-kube-api-access-nvsf5\" (UniqueName: \"kubernetes.io/projected/ed3bd09f-a9e9-4da0-bd17-337a41c81f40-kube-api-access-nvsf5\") pod \"ed3bd09f-a9e9-4da0-bd17-337a41c81f40\" (UID: \"ed3bd09f-a9e9-4da0-bd17-337a41c81f40\") "
Oct 30 00:51:18.526795 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:18.526673    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed3bd09f-a9e9-4da0-bd17-337a41c81f40-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed3bd09f-a9e9-4da0-bd17-337a41c81f40-workload-socket\") pod \"ed3bd09f-a9e9-4da0-bd17-337a41c81f40\" (UID: \"ed3bd09f-a9e9-4da0-bd17-337a41c81f40\") "
Oct 30 00:51:18.526795 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:18.526691    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed3bd09f-a9e9-4da0-bd17-337a41c81f40-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed3bd09f-a9e9-4da0-bd17-337a41c81f40-istio-envoy\") pod \"ed3bd09f-a9e9-4da0-bd17-337a41c81f40\" (UID: \"ed3bd09f-a9e9-4da0-bd17-337a41c81f40\") "
Oct 30 00:51:18.526795 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:18.526764    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed3bd09f-a9e9-4da0-bd17-337a41c81f40-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed3bd09f-a9e9-4da0-bd17-337a41c81f40-istiod-ca-cert\") pod \"ed3bd09f-a9e9-4da0-bd17-337a41c81f40\" (UID: \"ed3bd09f-a9e9-4da0-bd17-337a41c81f40\") "
Oct 30 00:51:18.527154 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:18.526814    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ed3bd09f-a9e9-4da0-bd17-337a41c81f40-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed3bd09f-a9e9-4da0-bd17-337a41c81f40-istio-podinfo\") pod \"ed3bd09f-a9e9-4da0-bd17-337a41c81f40\" (UID: \"ed3bd09f-a9e9-4da0-bd17-337a41c81f40\") "
Oct 30 00:51:18.527154 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:18.526807    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed3bd09f-a9e9-4da0-bd17-337a41c81f40-credential-socket" pod "ed3bd09f-a9e9-4da0-bd17-337a41c81f40" (UID: "ed3bd09f-a9e9-4da0-bd17-337a41c81f40"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:51:18.527304 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:18.527285    5002 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed3bd09f-a9e9-4da0-bd17-337a41c81f40-credential-socket\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:51:18.528221 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:18.527750    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed3bd09f-a9e9-4da0-bd17-337a41c81f40-istiod-ca-cert" pod "ed3bd09f-a9e9-4da0-bd17-337a41c81f40" (UID: "ed3bd09f-a9e9-4da0-bd17-337a41c81f40"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:51:18.528221 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:18.527970    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed3bd09f-a9e9-4da0-bd17-337a41c81f40-istio-data" pod "ed3bd09f-a9e9-4da0-bd17-337a41c81f40" (UID: "ed3bd09f-a9e9-4da0-bd17-337a41c81f40"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:51:18.528221 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:18.528098    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed3bd09f-a9e9-4da0-bd17-337a41c81f40-workload-certs" pod "ed3bd09f-a9e9-4da0-bd17-337a41c81f40" (UID: "ed3bd09f-a9e9-4da0-bd17-337a41c81f40"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:51:18.528221 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:18.528194    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed3bd09f-a9e9-4da0-bd17-337a41c81f40-workload-socket" pod "ed3bd09f-a9e9-4da0-bd17-337a41c81f40" (UID: "ed3bd09f-a9e9-4da0-bd17-337a41c81f40"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:51:18.530966 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:18.530937    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ed3bd09f-a9e9-4da0-bd17-337a41c81f40-istio-podinfo" pod "ed3bd09f-a9e9-4da0-bd17-337a41c81f40" (UID: "ed3bd09f-a9e9-4da0-bd17-337a41c81f40"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 00:51:18.531116 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:18.531077    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed3bd09f-a9e9-4da0-bd17-337a41c81f40-istio-envoy" pod "ed3bd09f-a9e9-4da0-bd17-337a41c81f40" (UID: "ed3bd09f-a9e9-4da0-bd17-337a41c81f40"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:51:18.531660 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:18.531638    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed3bd09f-a9e9-4da0-bd17-337a41c81f40-kube-api-access-nvsf5" pod "ed3bd09f-a9e9-4da0-bd17-337a41c81f40" (UID: "ed3bd09f-a9e9-4da0-bd17-337a41c81f40"). InnerVolumeSpecName "kube-api-access-nvsf5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:51:18.533535 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:18.533505    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed3bd09f-a9e9-4da0-bd17-337a41c81f40-istio-token" pod "ed3bd09f-a9e9-4da0-bd17-337a41c81f40" (UID: "ed3bd09f-a9e9-4da0-bd17-337a41c81f40"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:51:18.633931 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:18.633891    5002 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed3bd09f-a9e9-4da0-bd17-337a41c81f40-istio-data\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:51:18.633931 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:18.633930    5002 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed3bd09f-a9e9-4da0-bd17-337a41c81f40-istio-token\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:51:18.634201 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:18.633940    5002 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed3bd09f-a9e9-4da0-bd17-337a41c81f40-workload-certs\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:51:18.634201 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:18.633953    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nvsf5\" (UniqueName: \"kubernetes.io/projected/ed3bd09f-a9e9-4da0-bd17-337a41c81f40-kube-api-access-nvsf5\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:51:18.634201 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:18.633969    5002 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed3bd09f-a9e9-4da0-bd17-337a41c81f40-workload-socket\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:51:18.634201 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:18.633982    5002 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed3bd09f-a9e9-4da0-bd17-337a41c81f40-istio-envoy\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:51:18.634201 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:18.633992    5002 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed3bd09f-a9e9-4da0-bd17-337a41c81f40-istiod-ca-cert\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:51:18.634201 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:18.634000    5002 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed3bd09f-a9e9-4da0-bd17-337a41c81f40-istio-podinfo\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:51:19.148991 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:19.148912    5002 generic.go:356] "Generic (PLEG): container finished" podID="ed3bd09f-a9e9-4da0-bd17-337a41c81f40" containerID="bc18c1e759758447f042aea25b0c8822476e63797558396094f1fce92a7a45a8" exitCode=0
Oct 30 00:51:19.148991 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:19.148971    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-be862c05-openshift-default-546c6fcf65-kfvq9"
Oct 30 00:51:19.149310 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:19.149010    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-be862c05-openshift-default-546c6fcf65-kfvq9" event={"ID":"ed3bd09f-a9e9-4da0-bd17-337a41c81f40","Type":"ContainerDied","Data":"bc18c1e759758447f042aea25b0c8822476e63797558396094f1fce92a7a45a8"}
Oct 30 00:51:19.149310 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:19.149080    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-be862c05-openshift-default-546c6fcf65-kfvq9" event={"ID":"ed3bd09f-a9e9-4da0-bd17-337a41c81f40","Type":"ContainerDied","Data":"0b0d9c997a3947b27bf6820fdd02b0a9ad628851bba36747caaa1b1ddffbbb57"}
Oct 30 00:51:19.149310 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:19.149114    5002 scope.go:122] "RemoveContainer" containerID="bc18c1e759758447f042aea25b0c8822476e63797558396094f1fce92a7a45a8"
Oct 30 00:51:19.163578 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:19.163013    5002 scope.go:122] "RemoveContainer" containerID="bc18c1e759758447f042aea25b0c8822476e63797558396094f1fce92a7a45a8"
Oct 30 00:51:19.163661 ip-10-0-128-7 kubenswrapper[5002]: E1030 00:51:19.163549    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc18c1e759758447f042aea25b0c8822476e63797558396094f1fce92a7a45a8\": container with ID starting with bc18c1e759758447f042aea25b0c8822476e63797558396094f1fce92a7a45a8 not found: ID does not exist" containerID="bc18c1e759758447f042aea25b0c8822476e63797558396094f1fce92a7a45a8"
Oct 30 00:51:19.163661 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:19.163602    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc18c1e759758447f042aea25b0c8822476e63797558396094f1fce92a7a45a8"} err="failed to get container status \"bc18c1e759758447f042aea25b0c8822476e63797558396094f1fce92a7a45a8\": rpc error: code = NotFound desc = could not find container \"bc18c1e759758447f042aea25b0c8822476e63797558396094f1fce92a7a45a8\": container with ID starting with bc18c1e759758447f042aea25b0c8822476e63797558396094f1fce92a7a45a8 not found: ID does not exist"
Oct 30 00:51:19.207815 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:19.207773    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-be862c05-openshift-default-546c6fcf65-kfvq9"]
Oct 30 00:51:19.221025 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:19.220989    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-be862c05-openshift-default-546c6fcf65-kfvq9"]
Oct 30 00:51:20.498084 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:20.498041    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed3bd09f-a9e9-4da0-bd17-337a41c81f40" path="/var/lib/kubelet/pods/ed3bd09f-a9e9-4da0-bd17-337a41c81f40/volumes"
Oct 30 00:51:20.889978 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:20.889892    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-cbs5z"]
Oct 30 00:51:20.890758 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:20.890736    5002 cpu_manager.go:414] "containerMap: removing container" podUID="ed3bd09f-a9e9-4da0-bd17-337a41c81f40" containerName="istio-proxy"
Oct 30 00:51:20.890863 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:20.890759    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed3bd09f-a9e9-4da0-bd17-337a41c81f40" containerName="istio-proxy"
Oct 30 00:51:20.890969 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:20.890957    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed3bd09f-a9e9-4da0-bd17-337a41c81f40" containerName="istio-proxy"
Oct 30 00:51:20.896705 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:20.896683    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-cbs5z"
Oct 30 00:51:20.923741 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:20.923703    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-cbs5z"]
Oct 30 00:51:20.967365 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:20.967325    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f86c006d-5087-4443-b907-c2a30815b206-istio-envoy\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-cbs5z\" (UID: \"f86c006d-5087-4443-b907-c2a30815b206\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-cbs5z"
Oct 30 00:51:20.967687 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:20.967659    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f86c006d-5087-4443-b907-c2a30815b206-istio-podinfo\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-cbs5z\" (UID: \"f86c006d-5087-4443-b907-c2a30815b206\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-cbs5z"
Oct 30 00:51:20.967821 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:20.967780    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lmhz\" (UniqueName: \"kubernetes.io/projected/f86c006d-5087-4443-b907-c2a30815b206-kube-api-access-8lmhz\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-cbs5z\" (UID: \"f86c006d-5087-4443-b907-c2a30815b206\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-cbs5z"
Oct 30 00:51:20.967918 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:20.967902    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f86c006d-5087-4443-b907-c2a30815b206-credential-socket\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-cbs5z\" (UID: \"f86c006d-5087-4443-b907-c2a30815b206\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-cbs5z"
Oct 30 00:51:20.968001 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:20.967971    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f86c006d-5087-4443-b907-c2a30815b206-workload-certs\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-cbs5z\" (UID: \"f86c006d-5087-4443-b907-c2a30815b206\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-cbs5z"
Oct 30 00:51:20.968055 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:20.967999    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f86c006d-5087-4443-b907-c2a30815b206-istio-token\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-cbs5z\" (UID: \"f86c006d-5087-4443-b907-c2a30815b206\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-cbs5z"
Oct 30 00:51:20.968110 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:20.968073    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f86c006d-5087-4443-b907-c2a30815b206-istio-data\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-cbs5z\" (UID: \"f86c006d-5087-4443-b907-c2a30815b206\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-cbs5z"
Oct 30 00:51:20.968110 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:20.968103    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f86c006d-5087-4443-b907-c2a30815b206-workload-socket\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-cbs5z\" (UID: \"f86c006d-5087-4443-b907-c2a30815b206\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-cbs5z"
Oct 30 00:51:20.968282 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:20.968265    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f86c006d-5087-4443-b907-c2a30815b206-istiod-ca-cert\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-cbs5z\" (UID: \"f86c006d-5087-4443-b907-c2a30815b206\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-cbs5z"
Oct 30 00:51:21.069666 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:21.069627    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f86c006d-5087-4443-b907-c2a30815b206-credential-socket\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-cbs5z\" (UID: \"f86c006d-5087-4443-b907-c2a30815b206\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-cbs5z"
Oct 30 00:51:21.069841 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:21.069703    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f86c006d-5087-4443-b907-c2a30815b206-workload-certs\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-cbs5z\" (UID: \"f86c006d-5087-4443-b907-c2a30815b206\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-cbs5z"
Oct 30 00:51:21.069841 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:21.069726    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f86c006d-5087-4443-b907-c2a30815b206-istio-token\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-cbs5z\" (UID: \"f86c006d-5087-4443-b907-c2a30815b206\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-cbs5z"
Oct 30 00:51:21.069841 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:21.069786    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f86c006d-5087-4443-b907-c2a30815b206-istio-data\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-cbs5z\" (UID: \"f86c006d-5087-4443-b907-c2a30815b206\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-cbs5z"
Oct 30 00:51:21.069841 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:21.069818    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f86c006d-5087-4443-b907-c2a30815b206-workload-socket\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-cbs5z\" (UID: \"f86c006d-5087-4443-b907-c2a30815b206\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-cbs5z"
Oct 30 00:51:21.070076 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:21.069937    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f86c006d-5087-4443-b907-c2a30815b206-istiod-ca-cert\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-cbs5z\" (UID: \"f86c006d-5087-4443-b907-c2a30815b206\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-cbs5z"
Oct 30 00:51:21.070076 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:21.069968    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f86c006d-5087-4443-b907-c2a30815b206-istio-envoy\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-cbs5z\" (UID: \"f86c006d-5087-4443-b907-c2a30815b206\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-cbs5z"
Oct 30 00:51:21.070076 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:21.070007    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f86c006d-5087-4443-b907-c2a30815b206-istio-podinfo\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-cbs5z\" (UID: \"f86c006d-5087-4443-b907-c2a30815b206\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-cbs5z"
Oct 30 00:51:21.070250 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:21.070081    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lmhz\" (UniqueName: \"kubernetes.io/projected/f86c006d-5087-4443-b907-c2a30815b206-kube-api-access-8lmhz\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-cbs5z\" (UID: \"f86c006d-5087-4443-b907-c2a30815b206\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-cbs5z"
Oct 30 00:51:21.070250 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:21.070106    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f86c006d-5087-4443-b907-c2a30815b206-credential-socket\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-cbs5z\" (UID: \"f86c006d-5087-4443-b907-c2a30815b206\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-cbs5z"
Oct 30 00:51:21.070250 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:21.070104    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f86c006d-5087-4443-b907-c2a30815b206-workload-certs\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-cbs5z\" (UID: \"f86c006d-5087-4443-b907-c2a30815b206\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-cbs5z"
Oct 30 00:51:21.070250 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:21.070169    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f86c006d-5087-4443-b907-c2a30815b206-istio-data\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-cbs5z\" (UID: \"f86c006d-5087-4443-b907-c2a30815b206\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-cbs5z"
Oct 30 00:51:21.070513 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:21.070422    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f86c006d-5087-4443-b907-c2a30815b206-workload-socket\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-cbs5z\" (UID: \"f86c006d-5087-4443-b907-c2a30815b206\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-cbs5z"
Oct 30 00:51:21.070754 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:21.070733    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f86c006d-5087-4443-b907-c2a30815b206-istiod-ca-cert\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-cbs5z\" (UID: \"f86c006d-5087-4443-b907-c2a30815b206\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-cbs5z"
Oct 30 00:51:21.072182 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:21.072159    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f86c006d-5087-4443-b907-c2a30815b206-istio-envoy\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-cbs5z\" (UID: \"f86c006d-5087-4443-b907-c2a30815b206\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-cbs5z"
Oct 30 00:51:21.072519 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:21.072501    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f86c006d-5087-4443-b907-c2a30815b206-istio-podinfo\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-cbs5z\" (UID: \"f86c006d-5087-4443-b907-c2a30815b206\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-cbs5z"
Oct 30 00:51:21.094715 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:21.094684    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f86c006d-5087-4443-b907-c2a30815b206-istio-token\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-cbs5z\" (UID: \"f86c006d-5087-4443-b907-c2a30815b206\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-cbs5z"
Oct 30 00:51:21.094875 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:21.094824    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lmhz\" (UniqueName: \"kubernetes.io/projected/f86c006d-5087-4443-b907-c2a30815b206-kube-api-access-8lmhz\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-cbs5z\" (UID: \"f86c006d-5087-4443-b907-c2a30815b206\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-cbs5z"
Oct 30 00:51:21.201406 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:21.193944    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7d8z4"]
Oct 30 00:51:21.211685 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:21.211649    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-cbs5z"
Oct 30 00:51:21.671563 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:51:21.671528    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf86c006d_5087_4443_b907_c2a30815b206.slice/crio-429886fee08c7d8302aa2f9c2f0a2c22cc4e6cbfd9d5924cb3a83d09a0fae90a WatchSource:0}: Error finding container 429886fee08c7d8302aa2f9c2f0a2c22cc4e6cbfd9d5924cb3a83d09a0fae90a: Status 404 returned error can't find the container with id 429886fee08c7d8302aa2f9c2f0a2c22cc4e6cbfd9d5924cb3a83d09a0fae90a
Oct 30 00:51:21.674727 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:21.674683    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 00:51:21.674870 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:21.674787    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 00:51:21.674870 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:21.674838    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 00:51:21.674955 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:21.674909    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-cbs5z"]
Oct 30 00:51:22.256251 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:22.256214    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-cbs5z" event={"ID":"f86c006d-5087-4443-b907-c2a30815b206","Type":"ContainerStarted","Data":"3f464cc547370328ffeb0384981e8e49111bd27743b52f2b77406696aa09d5c5"}
Oct 30 00:51:22.256442 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:22.256281    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-cbs5z" event={"ID":"f86c006d-5087-4443-b907-c2a30815b206","Type":"ContainerStarted","Data":"429886fee08c7d8302aa2f9c2f0a2c22cc4e6cbfd9d5924cb3a83d09a0fae90a"}
Oct 30 00:51:22.305531 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:22.305481    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-cbs5z" podStartSLOduration=2.305467422 podStartE2EDuration="2.305467422s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:51:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:51:22.305115416 +0000 UTC m=+1177.921548295" watchObservedRunningTime="2026-10-30 00:51:22.305467422 +0000 UTC m=+1177.921900301"
Oct 30 00:51:23.215892 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:23.215848    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-cbs5z"
Oct 30 00:51:23.217281 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:23.217249    5002 patch_prober.go:29] interesting pod/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-cbs5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body=
Oct 30 00:51:23.217433 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:23.217315    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-cbs5z" podUID="f86c006d-5087-4443-b907-c2a30815b206" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused"
Oct 30 00:51:24.227404 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:24.223543    5002 patch_prober.go:29] interesting pod/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-cbs5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body=
Oct 30 00:51:24.227404 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:24.223608    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-cbs5z" podUID="f86c006d-5087-4443-b907-c2a30815b206" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused"
Oct 30 00:51:24.409579 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:24.409532    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6f4fba/simulator-w5-6f4fba-kserve-5bf8dfddd5-wcwzh" podUID="3a2c2f8f-649c-4ab6-af98-5d87df0e511f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:51:24.862434 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:24.843484    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca5363/distinct-2-w4-ca5363-kserve-576576f9b5-xkkqc" podUID="5de8bebf-cfad-49da-9a98-08aedde4442c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:51:25.228844 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:25.228804    5002 patch_prober.go:29] interesting pod/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-cbs5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body=
Oct 30 00:51:25.229418 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:25.229363    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-cbs5z" podUID="f86c006d-5087-4443-b907-c2a30815b206" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused"
Oct 30 00:51:26.216051 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:26.216015    5002 patch_prober.go:29] interesting pod/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-cbs5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body=
Oct 30 00:51:26.216256 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:26.216070    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-cbs5z" podUID="f86c006d-5087-4443-b907-c2a30815b206" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused"
Oct 30 00:51:27.216335 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:27.216296    5002 patch_prober.go:29] interesting pod/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-cbs5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body=
Oct 30 00:51:27.216750 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:27.216353    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-cbs5z" podUID="f86c006d-5087-4443-b907-c2a30815b206" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused"
Oct 30 00:51:28.216302 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:28.216264    5002 patch_prober.go:29] interesting pod/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-cbs5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body=
Oct 30 00:51:28.216522 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:28.216332    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-cbs5z" podUID="f86c006d-5087-4443-b907-c2a30815b206" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused"
Oct 30 00:51:29.539370 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:29.539335    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-cbs5z"
Oct 30 00:51:29.539921 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:29.539781    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-cbs5z"
Oct 30 00:51:29.540492 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:29.540472    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-cbs5z"
Oct 30 00:51:42.873765 ip-10-0-128-7 kubenswrapper[5002]: E1030 00:51:42.873719    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbjdq" containerName="extractor"
Oct 30 00:51:44.636800 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:44.636768    5002 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-g7c2w_5dc56e7e-f3c1-43eb-a21b-8be6bb4706cf/console-operator/1.log"
Oct 30 00:51:44.645869 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:44.645843    5002 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-g7c2w_5dc56e7e-f3c1-43eb-a21b-8be6bb4706cf/console-operator/1.log"
Oct 30 00:51:46.089407 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:46.088615    5002 scope.go:122] "RemoveContainer" containerID="50ff46e8a04b45d0ba9be413e88b04df2509ff644b741adb24bd447c5d17a344"
Oct 30 00:51:46.128077 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:46.128039    5002 scope.go:122] "RemoveContainer" containerID="9c2f81cea168f013fcefe1836fbcc2ea77a48bd01c278fe3f9d8aee78e7dc972"
Oct 30 00:51:50.098403 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:50.088226    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7d8z4"]
Oct 30 00:51:52.977406 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:52.973679    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-52650f/distinct-w0-52650f-kserve-d85899b5-vlhps" podUID="ea1850d9-abba-4977-a540-f6e0eaefb17d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:51:53.024413 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:53.023693    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-29be73/simulator-w1-29be73-kserve-9958d76db-fks2t" podUID="fbb3c806-8967-4d39-84c6-9dc022fc5cae" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/ready\": context deadline exceeded"
Oct 30 00:51:53.287464 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:53.286448    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1a11cf/distinct-w2-1a11cf-kserve-854cf4cdf8-cddw6" podUID="32a03478-46eb-4363-97b6-51a32c4099c7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded"
Oct 30 00:51:53.368603 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:51:53.368539    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6f4fba/simulator-w5-6f4fba-kserve-5bf8dfddd5-wcwzh" podUID="3a2c2f8f-649c-4ab6-af98-5d87df0e511f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:52:12.861904 ip-10-0-128-7 kubenswrapper[5002]: E1030 00:52:12.861828    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbjdq" containerName="extractor"
Oct 30 00:52:15.031495 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:52:15.031455    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-jwx9q"]
Oct 30 00:52:15.054260 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:52:15.054225    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-jwx9q"
Oct 30 00:52:15.064296 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:52:15.064271    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-jwx9q"]
Oct 30 00:52:15.201005 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:52:15.200969    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4f0d6bc-42a1-4f97-8713-daad565c8d7f-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-jwx9q\" (UID: \"a4f0d6bc-42a1-4f97-8713-daad565c8d7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-jwx9q"
Oct 30 00:52:15.201173 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:52:15.201033    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4f0d6bc-42a1-4f97-8713-daad565c8d7f-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-jwx9q\" (UID: \"a4f0d6bc-42a1-4f97-8713-daad565c8d7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-jwx9q"
Oct 30 00:52:15.201173 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:52:15.201148    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4f0d6bc-42a1-4f97-8713-daad565c8d7f-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-jwx9q\" (UID: \"a4f0d6bc-42a1-4f97-8713-daad565c8d7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-jwx9q"
Oct 30 00:52:15.201252 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:52:15.201195    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4f0d6bc-42a1-4f97-8713-daad565c8d7f-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-jwx9q\" (UID: \"a4f0d6bc-42a1-4f97-8713-daad565c8d7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-jwx9q"
Oct 30 00:52:15.201287 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:52:15.201258    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4f0d6bc-42a1-4f97-8713-daad565c8d7f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-jwx9q\" (UID: \"a4f0d6bc-42a1-4f97-8713-daad565c8d7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-jwx9q"
Oct 30 00:52:15.201321 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:52:15.201303    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4f0d6bc-42a1-4f97-8713-daad565c8d7f-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-jwx9q\" (UID: \"a4f0d6bc-42a1-4f97-8713-daad565c8d7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-jwx9q"
Oct 30 00:52:15.201357 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:52:15.201332    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4f0d6bc-42a1-4f97-8713-daad565c8d7f-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-jwx9q\" (UID: \"a4f0d6bc-42a1-4f97-8713-daad565c8d7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-jwx9q"
Oct 30 00:52:15.201402 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:52:15.201354    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4f0d6bc-42a1-4f97-8713-daad565c8d7f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-jwx9q\" (UID: \"a4f0d6bc-42a1-4f97-8713-daad565c8d7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-jwx9q"
Oct 30 00:52:15.201458 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:52:15.201444    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2jnl\" (UniqueName: \"kubernetes.io/projected/a4f0d6bc-42a1-4f97-8713-daad565c8d7f-kube-api-access-f2jnl\") pod \"maas-default-gateway-openshift-default-7d9897b54c-jwx9q\" (UID: \"a4f0d6bc-42a1-4f97-8713-daad565c8d7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-jwx9q"
Oct 30 00:52:15.302514 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:52:15.302424    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4f0d6bc-42a1-4f97-8713-daad565c8d7f-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-jwx9q\" (UID: \"a4f0d6bc-42a1-4f97-8713-daad565c8d7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-jwx9q"
Oct 30 00:52:15.302514 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:52:15.302499    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4f0d6bc-42a1-4f97-8713-daad565c8d7f-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-jwx9q\" (UID: \"a4f0d6bc-42a1-4f97-8713-daad565c8d7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-jwx9q"
Oct 30 00:52:15.302753 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:52:15.302555    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4f0d6bc-42a1-4f97-8713-daad565c8d7f-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-jwx9q\" (UID: \"a4f0d6bc-42a1-4f97-8713-daad565c8d7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-jwx9q"
Oct 30 00:52:15.302753 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:52:15.302585    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4f0d6bc-42a1-4f97-8713-daad565c8d7f-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-jwx9q\" (UID: \"a4f0d6bc-42a1-4f97-8713-daad565c8d7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-jwx9q"
Oct 30 00:52:15.302753 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:52:15.302610    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4f0d6bc-42a1-4f97-8713-daad565c8d7f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-jwx9q\" (UID: \"a4f0d6bc-42a1-4f97-8713-daad565c8d7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-jwx9q"
Oct 30 00:52:15.302753 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:52:15.302631    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4f0d6bc-42a1-4f97-8713-daad565c8d7f-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-jwx9q\" (UID: \"a4f0d6bc-42a1-4f97-8713-daad565c8d7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-jwx9q"
Oct 30 00:52:15.302753 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:52:15.302674    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4f0d6bc-42a1-4f97-8713-daad565c8d7f-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-jwx9q\" (UID: \"a4f0d6bc-42a1-4f97-8713-daad565c8d7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-jwx9q"
Oct 30 00:52:15.302753 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:52:15.302701    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4f0d6bc-42a1-4f97-8713-daad565c8d7f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-jwx9q\" (UID: \"a4f0d6bc-42a1-4f97-8713-daad565c8d7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-jwx9q"
Oct 30 00:52:15.302753 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:52:15.302754    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2jnl\" (UniqueName: \"kubernetes.io/projected/a4f0d6bc-42a1-4f97-8713-daad565c8d7f-kube-api-access-f2jnl\") pod \"maas-default-gateway-openshift-default-7d9897b54c-jwx9q\" (UID: \"a4f0d6bc-42a1-4f97-8713-daad565c8d7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-jwx9q"
Oct 30 00:52:15.303121 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:52:15.302970    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4f0d6bc-42a1-4f97-8713-daad565c8d7f-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-jwx9q\" (UID: \"a4f0d6bc-42a1-4f97-8713-daad565c8d7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-jwx9q"
Oct 30 00:52:15.303121 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:52:15.302982    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4f0d6bc-42a1-4f97-8713-daad565c8d7f-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-jwx9q\" (UID: \"a4f0d6bc-42a1-4f97-8713-daad565c8d7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-jwx9q"
Oct 30 00:52:15.303231 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:52:15.303112    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4f0d6bc-42a1-4f97-8713-daad565c8d7f-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-jwx9q\" (UID: \"a4f0d6bc-42a1-4f97-8713-daad565c8d7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-jwx9q"
Oct 30 00:52:15.303231 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:52:15.303175    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4f0d6bc-42a1-4f97-8713-daad565c8d7f-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-jwx9q\" (UID: \"a4f0d6bc-42a1-4f97-8713-daad565c8d7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-jwx9q"
Oct 30 00:52:15.303590 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:52:15.303567    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4f0d6bc-42a1-4f97-8713-daad565c8d7f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-jwx9q\" (UID: \"a4f0d6bc-42a1-4f97-8713-daad565c8d7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-jwx9q"
Oct 30 00:52:15.305128 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:52:15.305103    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4f0d6bc-42a1-4f97-8713-daad565c8d7f-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-jwx9q\" (UID: \"a4f0d6bc-42a1-4f97-8713-daad565c8d7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-jwx9q"
Oct 30 00:52:15.305262 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:52:15.305185    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4f0d6bc-42a1-4f97-8713-daad565c8d7f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-jwx9q\" (UID: \"a4f0d6bc-42a1-4f97-8713-daad565c8d7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-jwx9q"
Oct 30 00:52:15.327349 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:52:15.327318    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4f0d6bc-42a1-4f97-8713-daad565c8d7f-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-jwx9q\" (UID: \"a4f0d6bc-42a1-4f97-8713-daad565c8d7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-jwx9q"
Oct 30 00:52:15.327538 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:52:15.327520    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2jnl\" (UniqueName: \"kubernetes.io/projected/a4f0d6bc-42a1-4f97-8713-daad565c8d7f-kube-api-access-f2jnl\") pod \"maas-default-gateway-openshift-default-7d9897b54c-jwx9q\" (UID: \"a4f0d6bc-42a1-4f97-8713-daad565c8d7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-jwx9q"
Oct 30 00:52:15.376966 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:52:15.376932    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-jwx9q"
Oct 30 00:52:15.701505 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:52:15.701474    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4f0d6bc_42a1_4f97_8713_daad565c8d7f.slice/crio-c6beb31e285dfd4cd6a48b9e82d4a6c997aafdb6cc11333d74fe758bcae99ec2 WatchSource:0}: Error finding container c6beb31e285dfd4cd6a48b9e82d4a6c997aafdb6cc11333d74fe758bcae99ec2: Status 404 returned error can't find the container with id c6beb31e285dfd4cd6a48b9e82d4a6c997aafdb6cc11333d74fe758bcae99ec2
Oct 30 00:52:15.703875 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:52:15.703851    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-jwx9q"]
Oct 30 00:52:15.704422 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:52:15.704376    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 00:52:15.704474 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:52:15.704457    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 00:52:15.704509 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:52:15.704486    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 00:52:15.941255 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:52:15.941216    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-jwx9q" event={"ID":"a4f0d6bc-42a1-4f97-8713-daad565c8d7f","Type":"ContainerStarted","Data":"603aaf5acff7be23073e0383dd80cc2cf4e78f778ca57de14f8771ba8b16790f"}
Oct 30 00:52:15.941485 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:52:15.941275    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-jwx9q" event={"ID":"a4f0d6bc-42a1-4f97-8713-daad565c8d7f","Type":"ContainerStarted","Data":"c6beb31e285dfd4cd6a48b9e82d4a6c997aafdb6cc11333d74fe758bcae99ec2"}
Oct 30 00:52:15.999633 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:52:15.999510    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-jwx9q" podStartSLOduration=1.999494262 podStartE2EDuration="1.999494262s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:52:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:52:15.993782181 +0000 UTC m=+1231.610215082" watchObservedRunningTime="2026-10-30 00:52:15.999494262 +0000 UTC m=+1231.615927141"
Oct 30 00:52:16.377414 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:52:16.377287    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-jwx9q"
Oct 30 00:52:16.379861 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:52:16.379823    5002 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-jwx9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body=
Oct 30 00:52:16.379994 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:52:16.379897    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-jwx9q" podUID="a4f0d6bc-42a1-4f97-8713-daad565c8d7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused"
Oct 30 00:52:17.304443 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:52:17.304406    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-d7a6db25/e2e-lifecycle-model-d7a6db25-kserve-556c6db9f7-nsdhk"]
Oct 30 00:52:17.309543 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:52:17.309518    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-d7a6db25/e2e-lifecycle-model-d7a6db25-kserve-556c6db9f7-nsdhk"
Oct 30 00:52:17.329207 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:52:17.329173    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-d7a6db25/e2e-lifecycle-model-d7a6db25-kserve-556c6db9f7-nsdhk"]
Oct 30 00:52:17.338649 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:52:17.338620    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-d7a6db25\"/\"default-dockercfg-b696n\""
Oct 30 00:52:17.338971 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:52:17.338926    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-d7a6db25\"/\"e2e-lifecycle-model-d7a6db25-kserve-self-signed-certs\""
Oct 30 00:52:17.398404 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:52:17.393891    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-d7a6db25\"/\"openshift-service-ca.crt\""
Oct 30 00:52:17.398404 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:52:17.394023    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-d7a6db25\"/\"kube-root-ca.crt\""
Oct 30 00:52:17.398404 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:52:17.397266    5002 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-jwx9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body=
Oct 30 00:52:17.398404 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:52:17.397325    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-jwx9q" podUID="a4f0d6bc-42a1-4f97-8713-daad565c8d7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused"
Oct 30 00:52:17.443024 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:52:17.442987    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c783e213-5fa1-4e80-9f2d-66b362015d39-tls-certs\") pod \"e2e-lifecycle-model-d7a6db25-kserve-556c6db9f7-nsdhk\" (UID: \"c783e213-5fa1-4e80-9f2d-66b362015d39\") " pod="ai-tenant-e2e-mt-d7a6db25/e2e-lifecycle-model-d7a6db25-kserve-556c6db9f7-nsdhk"
Oct 30 00:52:17.443225 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:52:17.443116    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c783e213-5fa1-4e80-9f2d-66b362015d39-home\") pod \"e2e-lifecycle-model-d7a6db25-kserve-556c6db9f7-nsdhk\" (UID: \"c783e213-5fa1-4e80-9f2d-66b362015d39\") " pod="ai-tenant-e2e-mt-d7a6db25/e2e-lifecycle-model-d7a6db25-kserve-556c6db9f7-nsdhk"
Oct 30 00:52:17.443225 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:52:17.443144    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59kx9\" (UniqueName: \"kubernetes.io/projected/c783e213-5fa1-4e80-9f2d-66b362015d39-kube-api-access-59kx9\") pod \"e2e-lifecycle-model-d7a6db25-kserve-556c6db9f7-nsdhk\" (UID: \"c783e213-5fa1-4e80-9f2d-66b362015d39\") " pod="ai-tenant-e2e-mt-d7a6db25/e2e-lifecycle-model-d7a6db25-kserve-556c6db9f7-nsdhk"
Oct 30 00:52:17.443354 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:52:17.443288    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c783e213-5fa1-4e80-9f2d-66b362015d39-tmp-dir\") pod \"e2e-lifecycle-model-d7a6db25-kserve-556c6db9f7-nsdhk\" (UID: \"c783e213-5fa1-4e80-9f2d-66b362015d39\") " pod="ai-tenant-e2e-mt-d7a6db25/e2e-lifecycle-model-d7a6db25-kserve-556c6db9f7-nsdhk"
Oct 30 00:52:17.443459 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:52:17.443425    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c783e213-5fa1-4e80-9f2d-66b362015d39-dshm\") pod \"e2e-lifecycle-model-d7a6db25-kserve-556c6db9f7-nsdhk\" (UID: \"c783e213-5fa1-4e80-9f2d-66b362015d39\") " pod="ai-tenant-e2e-mt-d7a6db25/e2e-lifecycle-model-d7a6db25-kserve-556c6db9f7-nsdhk"
Oct 30 00:52:17.443532 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:52:17.443485    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c783e213-5fa1-4e80-9f2d-66b362015d39-model-cache\") pod \"e2e-lifecycle-model-d7a6db25-kserve-556c6db9f7-nsdhk\" (UID: \"c783e213-5fa1-4e80-9f2d-66b362015d39\") " pod="ai-tenant-e2e-mt-d7a6db25/e2e-lifecycle-model-d7a6db25-kserve-556c6db9f7-nsdhk"
Oct 30 00:52:17.545134 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:52:17.545093    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c783e213-5fa1-4e80-9f2d-66b362015d39-tls-certs\") pod \"e2e-lifecycle-model-d7a6db25-kserve-556c6db9f7-nsdhk\" (UID: \"c783e213-5fa1-4e80-9f2d-66b362015d39\") " pod="ai-tenant-e2e-mt-d7a6db25/e2e-lifecycle-model-d7a6db25-kserve-556c6db9f7-nsdhk"
Oct 30 00:52:17.545317 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:52:17.545173    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c783e213-5fa1-4e80-9f2d-66b362015d39-home\") pod \"e2e-lifecycle-model-d7a6db25-kserve-556c6db9f7-nsdhk\" (UID: \"c783e213-5fa1-4e80-9f2d-66b362015d39\") " pod="ai-tenant-e2e-mt-d7a6db25/e2e-lifecycle-model-d7a6db25-kserve-556c6db9f7-nsdhk"
Oct 30 00:52:17.545317 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:52:17.545191    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59kx9\" (UniqueName: \"kubernetes.io/projected/c783e213-5fa1-4e80-9f2d-66b362015d39-kube-api-access-59kx9\") pod \"e2e-lifecycle-model-d7a6db25-kserve-556c6db9f7-nsdhk\" (UID: \"c783e213-5fa1-4e80-9f2d-66b362015d39\") " pod="ai-tenant-e2e-mt-d7a6db25/e2e-lifecycle-model-d7a6db25-kserve-556c6db9f7-nsdhk"
Oct 30 00:52:17.545317 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:52:17.545280    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c783e213-5fa1-4e80-9f2d-66b362015d39-tmp-dir\") pod \"e2e-lifecycle-model-d7a6db25-kserve-556c6db9f7-nsdhk\" (UID: \"c783e213-5fa1-4e80-9f2d-66b362015d39\") " pod="ai-tenant-e2e-mt-d7a6db25/e2e-lifecycle-model-d7a6db25-kserve-556c6db9f7-nsdhk"
Oct 30 00:52:17.545460 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:52:17.545351    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c783e213-5fa1-4e80-9f2d-66b362015d39-dshm\") pod \"e2e-lifecycle-model-d7a6db25-kserve-556c6db9f7-nsdhk\" (UID: \"c783e213-5fa1-4e80-9f2d-66b362015d39\") " pod="ai-tenant-e2e-mt-d7a6db25/e2e-lifecycle-model-d7a6db25-kserve-556c6db9f7-nsdhk"
Oct 30 00:52:17.545460 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:52:17.545375    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c783e213-5fa1-4e80-9f2d-66b362015d39-model-cache\") pod \"e2e-lifecycle-model-d7a6db25-kserve-556c6db9f7-nsdhk\" (UID: \"c783e213-5fa1-4e80-9f2d-66b362015d39\") " pod="ai-tenant-e2e-mt-d7a6db25/e2e-lifecycle-model-d7a6db25-kserve-556c6db9f7-nsdhk"
Oct 30 00:52:17.546571 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:52:17.546546    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c783e213-5fa1-4e80-9f2d-66b362015d39-home\") pod \"e2e-lifecycle-model-d7a6db25-kserve-556c6db9f7-nsdhk\" (UID: \"c783e213-5fa1-4e80-9f2d-66b362015d39\") " pod="ai-tenant-e2e-mt-d7a6db25/e2e-lifecycle-model-d7a6db25-kserve-556c6db9f7-nsdhk"
Oct 30 00:52:17.546857 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:52:17.546830    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c783e213-5fa1-4e80-9f2d-66b362015d39-model-cache\") pod \"e2e-lifecycle-model-d7a6db25-kserve-556c6db9f7-nsdhk\" (UID: \"c783e213-5fa1-4e80-9f2d-66b362015d39\") " pod="ai-tenant-e2e-mt-d7a6db25/e2e-lifecycle-model-d7a6db25-kserve-556c6db9f7-nsdhk"
Oct 30 00:52:17.547110 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:52:17.547082    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c783e213-5fa1-4e80-9f2d-66b362015d39-tmp-dir\") pod \"e2e-lifecycle-model-d7a6db25-kserve-556c6db9f7-nsdhk\" (UID: \"c783e213-5fa1-4e80-9f2d-66b362015d39\") " pod="ai-tenant-e2e-mt-d7a6db25/e2e-lifecycle-model-d7a6db25-kserve-556c6db9f7-nsdhk"
Oct 30 00:52:17.548650 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:52:17.548627    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c783e213-5fa1-4e80-9f2d-66b362015d39-tls-certs\") pod \"e2e-lifecycle-model-d7a6db25-kserve-556c6db9f7-nsdhk\" (UID: \"c783e213-5fa1-4e80-9f2d-66b362015d39\") " pod="ai-tenant-e2e-mt-d7a6db25/e2e-lifecycle-model-d7a6db25-kserve-556c6db9f7-nsdhk"
Oct 30 00:52:17.549260 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:52:17.549237    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c783e213-5fa1-4e80-9f2d-66b362015d39-dshm\") pod \"e2e-lifecycle-model-d7a6db25-kserve-556c6db9f7-nsdhk\" (UID: \"c783e213-5fa1-4e80-9f2d-66b362015d39\") " pod="ai-tenant-e2e-mt-d7a6db25/e2e-lifecycle-model-d7a6db25-kserve-556c6db9f7-nsdhk"
Oct 30 00:52:17.586411 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:52:17.586319    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-59kx9\" (UniqueName: \"kubernetes.io/projected/c783e213-5fa1-4e80-9f2d-66b362015d39-kube-api-access-59kx9\") pod \"e2e-lifecycle-model-d7a6db25-kserve-556c6db9f7-nsdhk\" (UID: \"c783e213-5fa1-4e80-9f2d-66b362015d39\") " pod="ai-tenant-e2e-mt-d7a6db25/e2e-lifecycle-model-d7a6db25-kserve-556c6db9f7-nsdhk"
Oct 30 00:52:17.654417 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:52:17.644725    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-d7a6db25/e2e-lifecycle-model-d7a6db25-kserve-556c6db9f7-nsdhk"
Oct 30 00:52:18.187994 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:52:18.187958    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-d7a6db25/e2e-lifecycle-model-d7a6db25-kserve-556c6db9f7-nsdhk"]
Oct 30 00:52:18.189166 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:52:18.189137    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc783e213_5fa1_4e80_9f2d_66b362015d39.slice/crio-42fa9d5d6590de0094b07bb763e61e5da7abc5c06ce0cf9df874f9d0d5dcdf0e WatchSource:0}: Error finding container 42fa9d5d6590de0094b07bb763e61e5da7abc5c06ce0cf9df874f9d0d5dcdf0e: Status 404 returned error can't find the container with id 42fa9d5d6590de0094b07bb763e61e5da7abc5c06ce0cf9df874f9d0d5dcdf0e
Oct 30 00:52:18.381427 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:52:18.381275    5002 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-jwx9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body=
Oct 30 00:52:18.381427 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:52:18.381336    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-jwx9q" podUID="a4f0d6bc-42a1-4f97-8713-daad565c8d7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused"
Oct 30 00:52:19.027164 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:52:19.027127    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-d7a6db25/e2e-lifecycle-model-d7a6db25-kserve-556c6db9f7-nsdhk" event={"ID":"c783e213-5fa1-4e80-9f2d-66b362015d39","Type":"ContainerStarted","Data":"8bd15bc70e4c41517f36d790600910b34ea8e3b13e0c55ba5ab961cf78099d69"}
Oct 30 00:52:19.027164 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:52:19.027170    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-d7a6db25/e2e-lifecycle-model-d7a6db25-kserve-556c6db9f7-nsdhk" event={"ID":"c783e213-5fa1-4e80-9f2d-66b362015d39","Type":"ContainerStarted","Data":"42fa9d5d6590de0094b07bb763e61e5da7abc5c06ce0cf9df874f9d0d5dcdf0e"}
Oct 30 00:52:19.391037 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:52:19.390943    5002 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-jwx9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body=
Oct 30 00:52:19.391037 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:52:19.391017    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-jwx9q" podUID="a4f0d6bc-42a1-4f97-8713-daad565c8d7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused"
Oct 30 00:52:19.647288 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:52:19.647207    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-d7a6db25/e2e-lifecycle-model-d7a6db25-kserve-556c6db9f7-nsdhk"
Oct 30 00:52:20.381419 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:52:20.381359    5002 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-jwx9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body=
Oct 30 00:52:20.381929 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:52:20.381445    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-jwx9q" podUID="a4f0d6bc-42a1-4f97-8713-daad565c8d7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused"
Oct 30 00:52:21.030827 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:52:21.028532    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-mt-d7a6db25/e2e-lifecycle-model-d7a6db25-kserve-556c6db9f7-nsdhk" podStartSLOduration=4.028510916 podStartE2EDuration="4.028510916s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:52:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:52:19.079018614 +0000 UTC m=+1234.695451518" watchObservedRunningTime="2026-10-30 00:52:21.028510916 +0000 UTC m=+1236.644943796"
Oct 30 00:52:21.034560 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:52:21.034527    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eabf2670-bg57b"]
Oct 30 00:52:21.045534 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:52:21.045498    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eabf2670-bg57b"]
Oct 30 00:52:21.388465 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:52:21.382411    5002 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-jwx9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body=
Oct 30 00:52:21.388465 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:52:21.382475    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-jwx9q" podUID="a4f0d6bc-42a1-4f97-8713-daad565c8d7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused"
Oct 30 00:52:22.394918 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:52:22.394493    5002 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-jwx9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body=
Oct 30 00:52:22.394918 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:52:22.394566    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-jwx9q" podUID="a4f0d6bc-42a1-4f97-8713-daad565c8d7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused"
Oct 30 00:52:22.498063 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:52:22.498019    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf85f00d-1e7a-458c-8cc3-bc8eebc361c0" path="/var/lib/kubelet/pods/bf85f00d-1e7a-458c-8cc3-bc8eebc361c0/volumes"
Oct 30 00:52:23.377719 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:52:23.377676    5002 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-jwx9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body=
Oct 30 00:52:23.377928 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:52:23.377739    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-jwx9q" podUID="a4f0d6bc-42a1-4f97-8713-daad565c8d7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused"
Oct 30 00:52:23.784962 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:52:23.784909    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-d7a6db25-openshift-default-68896bcdc7-pgwjg"]
Oct 30 00:52:23.789553 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:52:23.789532    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d7a6db25-openshift-default-68896bcdc7-pgwjg"
Oct 30 00:52:23.807372 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:52:23.807344    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-d7a6db25-openshift-default-dockercfg-qnm52\""
Oct 30 00:52:23.831207 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:52:23.831176    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-d7a6db25-openshift-default-68896bcdc7-pgwjg"]
Oct 30 00:52:23.854261 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:52:23.854219    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fabdd87d-54a8-4fae-adf9-f2e23ff299cf-credential-socket\") pod \"e2e-mt-d7a6db25-openshift-default-68896bcdc7-pgwjg\" (UID: \"fabdd87d-54a8-4fae-adf9-f2e23ff299cf\") " pod="openshift-ingress/e2e-mt-d7a6db25-openshift-default-68896bcdc7-pgwjg"
Oct 30 00:52:23.854478 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:52:23.854295    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fabdd87d-54a8-4fae-adf9-f2e23ff299cf-istio-envoy\") pod \"e2e-mt-d7a6db25-openshift-default-68896bcdc7-pgwjg\" (UID: \"fabdd87d-54a8-4fae-adf9-f2e23ff299cf\") " pod="openshift-ingress/e2e-mt-d7a6db25-openshift-default-68896bcdc7-pgwjg"
Oct 30 00:52:23.854478 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:52:23.854440    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fabdd87d-54a8-4fae-adf9-f2e23ff299cf-workload-certs\") pod \"e2e-mt-d7a6db25-openshift-default-68896bcdc7-pgwjg\" (UID: \"fabdd87d-54a8-4fae-adf9-f2e23ff299cf\") " pod="openshift-ingress/e2e-mt-d7a6db25-openshift-default-68896bcdc7-pgwjg"
Oct 30 00:52:23.854640 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:52:23.854611    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fabdd87d-54a8-4fae-adf9-f2e23ff299cf-istio-token\") pod \"e2e-mt-d7a6db25-openshift-default-68896bcdc7-pgwjg\" (UID: \"fabdd87d-54a8-4fae-adf9-f2e23ff299cf\") " pod="openshift-ingress/e2e-mt-d7a6db25-openshift-default-68896bcdc7-pgwjg"
Oct 30 00:52:23.854720 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:52:23.854700    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkvqh\" (UniqueName: \"kubernetes.io/projected/fabdd87d-54a8-4fae-adf9-f2e23ff299cf-kube-api-access-kkvqh\") pod \"e2e-mt-d7a6db25-openshift-default-68896bcdc7-pgwjg\" (UID: \"fabdd87d-54a8-4fae-adf9-f2e23ff299cf\") " pod="openshift-ingress/e2e-mt-d7a6db25-openshift-default-68896bcdc7-pgwjg"
Oct 30 00:52:23.854769 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:52:23.854741    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fabdd87d-54a8-4fae-adf9-f2e23ff299cf-istio-podinfo\") pod \"e2e-mt-d7a6db25-openshift-default-68896bcdc7-pgwjg\" (UID: \"fabdd87d-54a8-4fae-adf9-f2e23ff299cf\") " pod="openshift-ingress/e2e-mt-d7a6db25-openshift-default-68896bcdc7-pgwjg"
Oct 30 00:52:23.854835 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:52:23.854818    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fabdd87d-54a8-4fae-adf9-f2e23ff299cf-istiod-ca-cert\") pod \"e2e-mt-d7a6db25-openshift-default-68896bcdc7-pgwjg\" (UID: \"fabdd87d-54a8-4fae-adf9-f2e23ff299cf\") " pod="openshift-ingress/e2e-mt-d7a6db25-openshift-default-68896bcdc7-pgwjg"
Oct 30 00:52:23.854889 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:52:23.854868    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fabdd87d-54a8-4fae-adf9-f2e23ff299cf-istio-data\") pod \"e2e-mt-d7a6db25-openshift-default-68896bcdc7-pgwjg\" (UID: \"fabdd87d-54a8-4fae-adf9-f2e23ff299cf\") " pod="openshift-ingress/e2e-mt-d7a6db25-openshift-default-68896bcdc7-pgwjg"
Oct 30 00:52:23.854944 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:52:23.854906    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fabdd87d-54a8-4fae-adf9-f2e23ff299cf-workload-socket\") pod \"e2e-mt-d7a6db25-openshift-default-68896bcdc7-pgwjg\" (UID: \"fabdd87d-54a8-4fae-adf9-f2e23ff299cf\") " pod="openshift-ingress/e2e-mt-d7a6db25-openshift-default-68896bcdc7-pgwjg"
Oct 30 00:52:23.956820 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:52:23.956777    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fabdd87d-54a8-4fae-adf9-f2e23ff299cf-credential-socket\") pod \"e2e-mt-d7a6db25-openshift-default-68896bcdc7-pgwjg\" (UID: \"fabdd87d-54a8-4fae-adf9-f2e23ff299cf\") " pod="openshift-ingress/e2e-mt-d7a6db25-openshift-default-68896bcdc7-pgwjg"
Oct 30 00:52:23.957016 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:52:23.956864    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fabdd87d-54a8-4fae-adf9-f2e23ff299cf-istio-envoy\") pod \"e2e-mt-d7a6db25-openshift-default-68896bcdc7-pgwjg\" (UID: \"fabdd87d-54a8-4fae-adf9-f2e23ff299cf\") " pod="openshift-ingress/e2e-mt-d7a6db25-openshift-default-68896bcdc7-pgwjg"
Oct 30 00:52:23.957016 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:52:23.956947    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fabdd87d-54a8-4fae-adf9-f2e23ff299cf-workload-certs\") pod \"e2e-mt-d7a6db25-openshift-default-68896bcdc7-pgwjg\" (UID: \"fabdd87d-54a8-4fae-adf9-f2e23ff299cf\") " pod="openshift-ingress/e2e-mt-d7a6db25-openshift-default-68896bcdc7-pgwjg"
Oct 30 00:52:23.957148 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:52:23.957086    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fabdd87d-54a8-4fae-adf9-f2e23ff299cf-istio-token\") pod \"e2e-mt-d7a6db25-openshift-default-68896bcdc7-pgwjg\" (UID: \"fabdd87d-54a8-4fae-adf9-f2e23ff299cf\") " pod="openshift-ingress/e2e-mt-d7a6db25-openshift-default-68896bcdc7-pgwjg"
Oct 30 00:52:23.957207 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:52:23.957174    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkvqh\" (UniqueName: \"kubernetes.io/projected/fabdd87d-54a8-4fae-adf9-f2e23ff299cf-kube-api-access-kkvqh\") pod \"e2e-mt-d7a6db25-openshift-default-68896bcdc7-pgwjg\" (UID: \"fabdd87d-54a8-4fae-adf9-f2e23ff299cf\") " pod="openshift-ingress/e2e-mt-d7a6db25-openshift-default-68896bcdc7-pgwjg"
Oct 30 00:52:23.957262 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:52:23.957212    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fabdd87d-54a8-4fae-adf9-f2e23ff299cf-istio-podinfo\") pod \"e2e-mt-d7a6db25-openshift-default-68896bcdc7-pgwjg\" (UID: \"fabdd87d-54a8-4fae-adf9-f2e23ff299cf\") " pod="openshift-ingress/e2e-mt-d7a6db25-openshift-default-68896bcdc7-pgwjg"
Oct 30 00:52:23.957262 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:52:23.957236    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fabdd87d-54a8-4fae-adf9-f2e23ff299cf-credential-socket\") pod \"e2e-mt-d7a6db25-openshift-default-68896bcdc7-pgwjg\" (UID: \"fabdd87d-54a8-4fae-adf9-f2e23ff299cf\") " pod="openshift-ingress/e2e-mt-d7a6db25-openshift-default-68896bcdc7-pgwjg"
Oct 30 00:52:23.957365 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:52:23.957309    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fabdd87d-54a8-4fae-adf9-f2e23ff299cf-istiod-ca-cert\") pod \"e2e-mt-d7a6db25-openshift-default-68896bcdc7-pgwjg\" (UID: \"fabdd87d-54a8-4fae-adf9-f2e23ff299cf\") " pod="openshift-ingress/e2e-mt-d7a6db25-openshift-default-68896bcdc7-pgwjg"
Oct 30 00:52:23.957365 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:52:23.957327    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fabdd87d-54a8-4fae-adf9-f2e23ff299cf-workload-certs\") pod \"e2e-mt-d7a6db25-openshift-default-68896bcdc7-pgwjg\" (UID: \"fabdd87d-54a8-4fae-adf9-f2e23ff299cf\") " pod="openshift-ingress/e2e-mt-d7a6db25-openshift-default-68896bcdc7-pgwjg"
Oct 30 00:52:23.957505 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:52:23.957361    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fabdd87d-54a8-4fae-adf9-f2e23ff299cf-istio-data\") pod \"e2e-mt-d7a6db25-openshift-default-68896bcdc7-pgwjg\" (UID: \"fabdd87d-54a8-4fae-adf9-f2e23ff299cf\") " pod="openshift-ingress/e2e-mt-d7a6db25-openshift-default-68896bcdc7-pgwjg"
Oct 30 00:52:23.957505 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:52:23.957437    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fabdd87d-54a8-4fae-adf9-f2e23ff299cf-workload-socket\") pod \"e2e-mt-d7a6db25-openshift-default-68896bcdc7-pgwjg\" (UID: \"fabdd87d-54a8-4fae-adf9-f2e23ff299cf\") " pod="openshift-ingress/e2e-mt-d7a6db25-openshift-default-68896bcdc7-pgwjg"
Oct 30 00:52:23.957854 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:52:23.957828    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fabdd87d-54a8-4fae-adf9-f2e23ff299cf-istio-data\") pod \"e2e-mt-d7a6db25-openshift-default-68896bcdc7-pgwjg\" (UID: \"fabdd87d-54a8-4fae-adf9-f2e23ff299cf\") " pod="openshift-ingress/e2e-mt-d7a6db25-openshift-default-68896bcdc7-pgwjg"
Oct 30 00:52:23.958192 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:52:23.957901    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fabdd87d-54a8-4fae-adf9-f2e23ff299cf-workload-socket\") pod \"e2e-mt-d7a6db25-openshift-default-68896bcdc7-pgwjg\" (UID: \"fabdd87d-54a8-4fae-adf9-f2e23ff299cf\") " pod="openshift-ingress/e2e-mt-d7a6db25-openshift-default-68896bcdc7-pgwjg"
Oct 30 00:52:23.958192 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:52:23.958101    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fabdd87d-54a8-4fae-adf9-f2e23ff299cf-istiod-ca-cert\") pod \"e2e-mt-d7a6db25-openshift-default-68896bcdc7-pgwjg\" (UID: \"fabdd87d-54a8-4fae-adf9-f2e23ff299cf\") " pod="openshift-ingress/e2e-mt-d7a6db25-openshift-default-68896bcdc7-pgwjg"
Oct 30 00:52:23.960128 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:52:23.959962    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fabdd87d-54a8-4fae-adf9-f2e23ff299cf-istio-envoy\") pod \"e2e-mt-d7a6db25-openshift-default-68896bcdc7-pgwjg\" (UID: \"fabdd87d-54a8-4fae-adf9-f2e23ff299cf\") " pod="openshift-ingress/e2e-mt-d7a6db25-openshift-default-68896bcdc7-pgwjg"
Oct 30 00:52:23.960233 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:52:23.960173    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fabdd87d-54a8-4fae-adf9-f2e23ff299cf-istio-podinfo\") pod \"e2e-mt-d7a6db25-openshift-default-68896bcdc7-pgwjg\" (UID: \"fabdd87d-54a8-4fae-adf9-f2e23ff299cf\") " pod="openshift-ingress/e2e-mt-d7a6db25-openshift-default-68896bcdc7-pgwjg"
Oct 30 00:52:23.982734 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:52:23.982703    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fabdd87d-54a8-4fae-adf9-f2e23ff299cf-istio-token\") pod \"e2e-mt-d7a6db25-openshift-default-68896bcdc7-pgwjg\" (UID: \"fabdd87d-54a8-4fae-adf9-f2e23ff299cf\") " pod="openshift-ingress/e2e-mt-d7a6db25-openshift-default-68896bcdc7-pgwjg"
Oct 30 00:52:23.983015 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:52:23.982995    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkvqh\" (UniqueName: \"kubernetes.io/projected/fabdd87d-54a8-4fae-adf9-f2e23ff299cf-kube-api-access-kkvqh\") pod \"e2e-mt-d7a6db25-openshift-default-68896bcdc7-pgwjg\" (UID: \"fabdd87d-54a8-4fae-adf9-f2e23ff299cf\") " pod="openshift-ingress/e2e-mt-d7a6db25-openshift-default-68896bcdc7-pgwjg"
Oct 30 00:52:24.108352 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:52:24.108255    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d7a6db25-openshift-default-68896bcdc7-pgwjg"
Oct 30 00:52:24.452341 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:52:24.452314    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-d7a6db25-openshift-default-68896bcdc7-pgwjg"]
Oct 30 00:52:24.452522 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:52:24.452503    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfabdd87d_54a8_4fae_adf9_f2e23ff299cf.slice/crio-6fc710c6235483b913ee37bfbfe64cd4cd07adb1c6d42987681a061fae666151 WatchSource:0}: Error finding container 6fc710c6235483b913ee37bfbfe64cd4cd07adb1c6d42987681a061fae666151: Status 404 returned error can't find the container with id 6fc710c6235483b913ee37bfbfe64cd4cd07adb1c6d42987681a061fae666151
Oct 30 00:52:24.455046 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:52:24.455016    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 00:52:24.455159 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:52:24.455077    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 00:52:24.455159 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:52:24.455111    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 00:52:25.117345 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:52:25.117308    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d7a6db25-openshift-default-68896bcdc7-pgwjg" event={"ID":"fabdd87d-54a8-4fae-adf9-f2e23ff299cf","Type":"ContainerStarted","Data":"b58568d8c76e3febbab0fa823c357c99c840fbb59378f57b8f45eeb3a827274b"}
Oct 30 00:52:25.117825 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:52:25.117803    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d7a6db25-openshift-default-68896bcdc7-pgwjg" event={"ID":"fabdd87d-54a8-4fae-adf9-f2e23ff299cf","Type":"ContainerStarted","Data":"6fc710c6235483b913ee37bfbfe64cd4cd07adb1c6d42987681a061fae666151"}
Oct 30 00:52:25.180394 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:52:25.180346    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-jwx9q"
Oct 30 00:52:25.221409 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:52:25.220189    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-jwx9q"
Oct 30 00:52:25.221409 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:52:25.221017    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-d7a6db25-openshift-default-68896bcdc7-pgwjg" podStartSLOduration=2.220829855 podStartE2EDuration="2.220829855s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:52:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:52:25.179903291 +0000 UTC m=+1240.796336198" watchObservedRunningTime="2026-10-30 00:52:25.220829855 +0000 UTC m=+1240.837262735"
Oct 30 00:52:25.241008 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:52:25.240966    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-jwx9q"
Oct 30 00:52:26.118404 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:52:26.114765    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-d7a6db25-openshift-default-68896bcdc7-pgwjg"
Oct 30 00:52:26.118404 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:52:26.118232    5002 patch_prober.go:29] interesting pod/e2e-mt-d7a6db25-openshift-default-68896bcdc7-pgwjg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Oct 30 00:52:26.118404 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:52:26.118293    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d7a6db25-openshift-default-68896bcdc7-pgwjg" podUID="fabdd87d-54a8-4fae-adf9-f2e23ff299cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Oct 30 00:52:27.127698 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:52:27.127660    5002 patch_prober.go:29] interesting pod/e2e-mt-d7a6db25-openshift-default-68896bcdc7-pgwjg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Oct 30 00:52:27.128327 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:52:27.128285    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d7a6db25-openshift-default-68896bcdc7-pgwjg" podUID="fabdd87d-54a8-4fae-adf9-f2e23ff299cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Oct 30 00:52:27.646302 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:52:27.646266    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-d7a6db25/e2e-lifecycle-model-d7a6db25-kserve-556c6db9f7-nsdhk"
Oct 30 00:52:27.697727 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:52:27.697681    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-d7a6db25/e2e-lifecycle-model-d7a6db25-kserve-556c6db9f7-nsdhk"
Oct 30 00:52:28.110289 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:52:28.110042    5002 patch_prober.go:29] interesting pod/e2e-mt-d7a6db25-openshift-default-68896bcdc7-pgwjg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Oct 30 00:52:28.110289 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:52:28.110096    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d7a6db25-openshift-default-68896bcdc7-pgwjg" podUID="fabdd87d-54a8-4fae-adf9-f2e23ff299cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Oct 30 00:52:28.208120 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:52:28.208085    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-d7a6db25/e2e-lifecycle-model-d7a6db25-kserve-556c6db9f7-nsdhk"
Oct 30 00:52:29.108866 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:52:29.108829    5002 patch_prober.go:29] interesting pod/e2e-mt-d7a6db25-openshift-default-68896bcdc7-pgwjg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Oct 30 00:52:29.109052 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:52:29.108898    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d7a6db25-openshift-default-68896bcdc7-pgwjg" podUID="fabdd87d-54a8-4fae-adf9-f2e23ff299cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Oct 30 00:52:30.109020 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:52:30.108982    5002 patch_prober.go:29] interesting pod/e2e-mt-d7a6db25-openshift-default-68896bcdc7-pgwjg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Oct 30 00:52:30.109457 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:52:30.109040    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d7a6db25-openshift-default-68896bcdc7-pgwjg" podUID="fabdd87d-54a8-4fae-adf9-f2e23ff299cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Oct 30 00:52:31.113783 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:52:31.113749    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-d7a6db25-openshift-default-68896bcdc7-pgwjg"
Oct 30 00:52:31.114256 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:52:31.114156    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-d7a6db25-openshift-default-68896bcdc7-pgwjg"
Oct 30 00:52:31.114791 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:52:31.114773    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-d7a6db25-openshift-default-68896bcdc7-pgwjg"
Oct 30 00:52:41.030524 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:52:41.030490    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16f76fb3-jmbfm"]
Oct 30 00:52:41.044170 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:52:41.044144    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16f76fb3-jmbfm"]
Oct 30 00:52:42.496259 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:52:42.496223    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1acd774-f759-4d08-8b45-7f07976478fe" path="/var/lib/kubelet/pods/a1acd774-f759-4d08-8b45-7f07976478fe/volumes"
Oct 30 00:52:42.863406 ip-10-0-128-7 kubenswrapper[5002]: E1030 00:52:42.863289    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbjdq" containerName="extractor"
Oct 30 00:52:43.928287 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:52:43.928238    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7d8z4"]
Oct 30 00:52:56.034801 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:52:56.034767    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b586d6ef-6mxlr"]
Oct 30 00:52:56.048724 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:52:56.048694    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b586d6ef-6mxlr"]
Oct 30 00:52:56.497259 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:52:56.497219    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8cb5fca0-dafe-4619-8341-1ad59e427c64" path="/var/lib/kubelet/pods/8cb5fca0-dafe-4619-8341-1ad59e427c64/volumes"
Oct 30 00:53:12.933457 ip-10-0-128-7 kubenswrapper[5002]: E1030 00:53:12.933407    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbjdq" containerName="extractor"
Oct 30 00:53:14.105176 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:53:14.105131    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e026345b-r64w7"]
Oct 30 00:53:14.150419 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:53:14.138664    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e026345b-r64w7"]
Oct 30 00:53:14.150419 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:53:14.138790    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e026345b-r64w7"
Oct 30 00:53:14.250460 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:53:14.248844    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 00:53:14.250460 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:53:14.248984    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5phsh\""
Oct 30 00:53:14.261445 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:53:14.261406    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/717cdfa3-84f9-47b2-bc60-0aee1b494a82-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e026345b-r64w7\" (UID: \"717cdfa3-84f9-47b2-bc60-0aee1b494a82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e026345b-r64w7"
Oct 30 00:53:14.261703 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:53:14.261683    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/717cdfa3-84f9-47b2-bc60-0aee1b494a82-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e026345b-r64w7\" (UID: \"717cdfa3-84f9-47b2-bc60-0aee1b494a82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e026345b-r64w7"
Oct 30 00:53:14.270404 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:53:14.261968    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hds7\" (UniqueName: \"kubernetes.io/projected/717cdfa3-84f9-47b2-bc60-0aee1b494a82-kube-api-access-7hds7\") pod \"maas-api-revoke-sub-e026345b-r64w7\" (UID: \"717cdfa3-84f9-47b2-bc60-0aee1b494a82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e026345b-r64w7"
Oct 30 00:53:14.271543 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:53:14.271515    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 00:53:14.274534 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:53:14.271943    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 00:53:14.372165 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:53:14.372070    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hds7\" (UniqueName: \"kubernetes.io/projected/717cdfa3-84f9-47b2-bc60-0aee1b494a82-kube-api-access-7hds7\") pod \"maas-api-revoke-sub-e026345b-r64w7\" (UID: \"717cdfa3-84f9-47b2-bc60-0aee1b494a82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e026345b-r64w7"
Oct 30 00:53:14.372337 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:53:14.372291    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/717cdfa3-84f9-47b2-bc60-0aee1b494a82-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e026345b-r64w7\" (UID: \"717cdfa3-84f9-47b2-bc60-0aee1b494a82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e026345b-r64w7"
Oct 30 00:53:14.372463 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:53:14.372337    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/717cdfa3-84f9-47b2-bc60-0aee1b494a82-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e026345b-r64w7\" (UID: \"717cdfa3-84f9-47b2-bc60-0aee1b494a82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e026345b-r64w7"
Oct 30 00:53:14.373028 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:53:14.372998    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/717cdfa3-84f9-47b2-bc60-0aee1b494a82-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e026345b-r64w7\" (UID: \"717cdfa3-84f9-47b2-bc60-0aee1b494a82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e026345b-r64w7"
Oct 30 00:53:14.374970 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:53:14.374944    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/717cdfa3-84f9-47b2-bc60-0aee1b494a82-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e026345b-r64w7\" (UID: \"717cdfa3-84f9-47b2-bc60-0aee1b494a82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e026345b-r64w7"
Oct 30 00:53:14.398121 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:53:14.398089    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hds7\" (UniqueName: \"kubernetes.io/projected/717cdfa3-84f9-47b2-bc60-0aee1b494a82-kube-api-access-7hds7\") pod \"maas-api-revoke-sub-e026345b-r64w7\" (UID: \"717cdfa3-84f9-47b2-bc60-0aee1b494a82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e026345b-r64w7"
Oct 30 00:53:14.629507 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:53:14.629182    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e026345b-r64w7"
Oct 30 00:53:15.065265 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:53:15.065205    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod717cdfa3_84f9_47b2_bc60_0aee1b494a82.slice/crio-64ebf87ca2beaebdf36dcd078b77506ea28d9fca4edc3d5b03db67073d514b67 WatchSource:0}: Error finding container 64ebf87ca2beaebdf36dcd078b77506ea28d9fca4edc3d5b03db67073d514b67: Status 404 returned error can't find the container with id 64ebf87ca2beaebdf36dcd078b77506ea28d9fca4edc3d5b03db67073d514b67
Oct 30 00:53:15.069915 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:53:15.069889    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e026345b-r64w7"]
Oct 30 00:53:15.939130 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:53:15.939087    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e026345b-r64w7" event={"ID":"717cdfa3-84f9-47b2-bc60-0aee1b494a82","Type":"ContainerStarted","Data":"381641d2380b8b0643d458eaafbfabc023a82fd006f21e5abc952fdb4296ba77"}
Oct 30 00:53:15.939557 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:53:15.939149    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e026345b-r64w7" event={"ID":"717cdfa3-84f9-47b2-bc60-0aee1b494a82","Type":"ContainerStarted","Data":"64ebf87ca2beaebdf36dcd078b77506ea28d9fca4edc3d5b03db67073d514b67"}
Oct 30 00:53:15.986847 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:53:15.986787    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e026345b-r64w7" podStartSLOduration=1.986772339 podStartE2EDuration="1.986772339s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:53:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:53:15.983796641 +0000 UTC m=+1291.600229560" watchObservedRunningTime="2026-10-30 00:53:15.986772339 +0000 UTC m=+1291.603205220"
Oct 30 00:53:16.947255 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:53:16.947214    5002 generic.go:356] "Generic (PLEG): container finished" podID="717cdfa3-84f9-47b2-bc60-0aee1b494a82" containerID="381641d2380b8b0643d458eaafbfabc023a82fd006f21e5abc952fdb4296ba77" exitCode=0
Oct 30 00:53:16.947702 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:53:16.947291    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e026345b-r64w7" event={"ID":"717cdfa3-84f9-47b2-bc60-0aee1b494a82","Type":"ContainerDied","Data":"381641d2380b8b0643d458eaafbfabc023a82fd006f21e5abc952fdb4296ba77"}
Oct 30 00:53:18.134468 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:53:18.134438    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e026345b-r64w7"
Oct 30 00:53:18.251652 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:53:18.251567    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/717cdfa3-84f9-47b2-bc60-0aee1b494a82-kube-api-access-7hds7\" (UniqueName: \"kubernetes.io/projected/717cdfa3-84f9-47b2-bc60-0aee1b494a82-kube-api-access-7hds7\") pod \"717cdfa3-84f9-47b2-bc60-0aee1b494a82\" (UID: \"717cdfa3-84f9-47b2-bc60-0aee1b494a82\") "
Oct 30 00:53:18.251831 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:53:18.251752    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/717cdfa3-84f9-47b2-bc60-0aee1b494a82-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/717cdfa3-84f9-47b2-bc60-0aee1b494a82-maas-api-xks-ca\") pod \"717cdfa3-84f9-47b2-bc60-0aee1b494a82\" (UID: \"717cdfa3-84f9-47b2-bc60-0aee1b494a82\") "
Oct 30 00:53:18.251831 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:53:18.251782    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/717cdfa3-84f9-47b2-bc60-0aee1b494a82-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/717cdfa3-84f9-47b2-bc60-0aee1b494a82-maas-api-openshift-ca\") pod \"717cdfa3-84f9-47b2-bc60-0aee1b494a82\" (UID: \"717cdfa3-84f9-47b2-bc60-0aee1b494a82\") "
Oct 30 00:53:18.252268 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:53:18.252235    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/717cdfa3-84f9-47b2-bc60-0aee1b494a82-maas-api-openshift-ca" pod "717cdfa3-84f9-47b2-bc60-0aee1b494a82" (UID: "717cdfa3-84f9-47b2-bc60-0aee1b494a82"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:53:18.254015 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:53:18.253986    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/717cdfa3-84f9-47b2-bc60-0aee1b494a82-kube-api-access-7hds7" pod "717cdfa3-84f9-47b2-bc60-0aee1b494a82" (UID: "717cdfa3-84f9-47b2-bc60-0aee1b494a82"). InnerVolumeSpecName "kube-api-access-7hds7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:53:18.254300 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:53:18.254279    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/717cdfa3-84f9-47b2-bc60-0aee1b494a82-maas-api-xks-ca" pod "717cdfa3-84f9-47b2-bc60-0aee1b494a82" (UID: "717cdfa3-84f9-47b2-bc60-0aee1b494a82"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:53:18.353778 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:53:18.353737    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/717cdfa3-84f9-47b2-bc60-0aee1b494a82-maas-api-xks-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:53:18.353778 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:53:18.353776    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/717cdfa3-84f9-47b2-bc60-0aee1b494a82-maas-api-openshift-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:53:18.354037 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:53:18.353793    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7hds7\" (UniqueName: \"kubernetes.io/projected/717cdfa3-84f9-47b2-bc60-0aee1b494a82-kube-api-access-7hds7\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:53:18.965764 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:53:18.965729    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e026345b-r64w7"
Oct 30 00:53:18.965764 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:53:18.965742    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e026345b-r64w7" event={"ID":"717cdfa3-84f9-47b2-bc60-0aee1b494a82","Type":"ContainerDied","Data":"64ebf87ca2beaebdf36dcd078b77506ea28d9fca4edc3d5b03db67073d514b67"}
Oct 30 00:53:18.966016 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:53:18.965794    5002 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="64ebf87ca2beaebdf36dcd078b77506ea28d9fca4edc3d5b03db67073d514b67"
Oct 30 00:53:24.250527 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:53:24.250488    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-bdc256-openshift-default-5cf7d97495-76wgr"]
Oct 30 00:53:24.251979 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:53:24.251947    5002 cpu_manager.go:414] "containerMap: removing container" podUID="717cdfa3-84f9-47b2-bc60-0aee1b494a82" containerName="revoke-keys"
Oct 30 00:53:24.251979 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:53:24.251975    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="717cdfa3-84f9-47b2-bc60-0aee1b494a82" containerName="revoke-keys"
Oct 30 00:53:24.252161 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:53:24.252142    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="717cdfa3-84f9-47b2-bc60-0aee1b494a82" containerName="revoke-keys"
Oct 30 00:53:24.256618 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:53:24.256565    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-bdc256-openshift-default-5cf7d97495-76wgr"
Oct 30 00:53:24.286556 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:53:24.286529    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-bdc256-openshift-default-dockercfg-j9nfz\""
Oct 30 00:53:24.305400 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:53:24.299347    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-bdc256-openshift-default-5cf7d97495-76wgr"]
Oct 30 00:53:24.352830 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:53:24.352786    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bcdd7ce4-a94f-4e6d-aebc-e7ca25d36b4a-istio-envoy\") pod \"e2e-api-b-bdc256-openshift-default-5cf7d97495-76wgr\" (UID: \"bcdd7ce4-a94f-4e6d-aebc-e7ca25d36b4a\") " pod="openshift-ingress/e2e-api-b-bdc256-openshift-default-5cf7d97495-76wgr"
Oct 30 00:53:24.353031 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:53:24.352857    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcdd7ce4-a94f-4e6d-aebc-e7ca25d36b4a-credential-socket\") pod \"e2e-api-b-bdc256-openshift-default-5cf7d97495-76wgr\" (UID: \"bcdd7ce4-a94f-4e6d-aebc-e7ca25d36b4a\") " pod="openshift-ingress/e2e-api-b-bdc256-openshift-default-5cf7d97495-76wgr"
Oct 30 00:53:24.353031 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:53:24.352972    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z485d\" (UniqueName: \"kubernetes.io/projected/bcdd7ce4-a94f-4e6d-aebc-e7ca25d36b4a-kube-api-access-z485d\") pod \"e2e-api-b-bdc256-openshift-default-5cf7d97495-76wgr\" (UID: \"bcdd7ce4-a94f-4e6d-aebc-e7ca25d36b4a\") " pod="openshift-ingress/e2e-api-b-bdc256-openshift-default-5cf7d97495-76wgr"
Oct 30 00:53:24.353155 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:53:24.353036    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcdd7ce4-a94f-4e6d-aebc-e7ca25d36b4a-workload-socket\") pod \"e2e-api-b-bdc256-openshift-default-5cf7d97495-76wgr\" (UID: \"bcdd7ce4-a94f-4e6d-aebc-e7ca25d36b4a\") " pod="openshift-ingress/e2e-api-b-bdc256-openshift-default-5cf7d97495-76wgr"
Oct 30 00:53:24.353206 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:53:24.353147    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bcdd7ce4-a94f-4e6d-aebc-e7ca25d36b4a-istio-data\") pod \"e2e-api-b-bdc256-openshift-default-5cf7d97495-76wgr\" (UID: \"bcdd7ce4-a94f-4e6d-aebc-e7ca25d36b4a\") " pod="openshift-ingress/e2e-api-b-bdc256-openshift-default-5cf7d97495-76wgr"
Oct 30 00:53:24.353347 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:53:24.353329    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bcdd7ce4-a94f-4e6d-aebc-e7ca25d36b4a-workload-certs\") pod \"e2e-api-b-bdc256-openshift-default-5cf7d97495-76wgr\" (UID: \"bcdd7ce4-a94f-4e6d-aebc-e7ca25d36b4a\") " pod="openshift-ingress/e2e-api-b-bdc256-openshift-default-5cf7d97495-76wgr"
Oct 30 00:53:24.353426 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:53:24.353373    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bcdd7ce4-a94f-4e6d-aebc-e7ca25d36b4a-istio-podinfo\") pod \"e2e-api-b-bdc256-openshift-default-5cf7d97495-76wgr\" (UID: \"bcdd7ce4-a94f-4e6d-aebc-e7ca25d36b4a\") " pod="openshift-ingress/e2e-api-b-bdc256-openshift-default-5cf7d97495-76wgr"
Oct 30 00:53:24.353493 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:53:24.353449    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bcdd7ce4-a94f-4e6d-aebc-e7ca25d36b4a-istiod-ca-cert\") pod \"e2e-api-b-bdc256-openshift-default-5cf7d97495-76wgr\" (UID: \"bcdd7ce4-a94f-4e6d-aebc-e7ca25d36b4a\") " pod="openshift-ingress/e2e-api-b-bdc256-openshift-default-5cf7d97495-76wgr"
Oct 30 00:53:24.353544 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:53:24.353532    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bcdd7ce4-a94f-4e6d-aebc-e7ca25d36b4a-istio-token\") pod \"e2e-api-b-bdc256-openshift-default-5cf7d97495-76wgr\" (UID: \"bcdd7ce4-a94f-4e6d-aebc-e7ca25d36b4a\") " pod="openshift-ingress/e2e-api-b-bdc256-openshift-default-5cf7d97495-76wgr"
Oct 30 00:53:24.458519 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:53:24.458464    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bcdd7ce4-a94f-4e6d-aebc-e7ca25d36b4a-workload-certs\") pod \"e2e-api-b-bdc256-openshift-default-5cf7d97495-76wgr\" (UID: \"bcdd7ce4-a94f-4e6d-aebc-e7ca25d36b4a\") " pod="openshift-ingress/e2e-api-b-bdc256-openshift-default-5cf7d97495-76wgr"
Oct 30 00:53:24.458799 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:53:24.458774    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bcdd7ce4-a94f-4e6d-aebc-e7ca25d36b4a-istio-podinfo\") pod \"e2e-api-b-bdc256-openshift-default-5cf7d97495-76wgr\" (UID: \"bcdd7ce4-a94f-4e6d-aebc-e7ca25d36b4a\") " pod="openshift-ingress/e2e-api-b-bdc256-openshift-default-5cf7d97495-76wgr"
Oct 30 00:53:24.458956 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:53:24.458938    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bcdd7ce4-a94f-4e6d-aebc-e7ca25d36b4a-istiod-ca-cert\") pod \"e2e-api-b-bdc256-openshift-default-5cf7d97495-76wgr\" (UID: \"bcdd7ce4-a94f-4e6d-aebc-e7ca25d36b4a\") " pod="openshift-ingress/e2e-api-b-bdc256-openshift-default-5cf7d97495-76wgr"
Oct 30 00:53:24.459087 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:53:24.459073    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bcdd7ce4-a94f-4e6d-aebc-e7ca25d36b4a-istio-token\") pod \"e2e-api-b-bdc256-openshift-default-5cf7d97495-76wgr\" (UID: \"bcdd7ce4-a94f-4e6d-aebc-e7ca25d36b4a\") " pod="openshift-ingress/e2e-api-b-bdc256-openshift-default-5cf7d97495-76wgr"
Oct 30 00:53:24.459265 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:53:24.459248    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bcdd7ce4-a94f-4e6d-aebc-e7ca25d36b4a-istio-envoy\") pod \"e2e-api-b-bdc256-openshift-default-5cf7d97495-76wgr\" (UID: \"bcdd7ce4-a94f-4e6d-aebc-e7ca25d36b4a\") " pod="openshift-ingress/e2e-api-b-bdc256-openshift-default-5cf7d97495-76wgr"
Oct 30 00:53:24.459890 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:53:24.459864    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcdd7ce4-a94f-4e6d-aebc-e7ca25d36b4a-credential-socket\") pod \"e2e-api-b-bdc256-openshift-default-5cf7d97495-76wgr\" (UID: \"bcdd7ce4-a94f-4e6d-aebc-e7ca25d36b4a\") " pod="openshift-ingress/e2e-api-b-bdc256-openshift-default-5cf7d97495-76wgr"
Oct 30 00:53:24.460072 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:53:24.460046    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z485d\" (UniqueName: \"kubernetes.io/projected/bcdd7ce4-a94f-4e6d-aebc-e7ca25d36b4a-kube-api-access-z485d\") pod \"e2e-api-b-bdc256-openshift-default-5cf7d97495-76wgr\" (UID: \"bcdd7ce4-a94f-4e6d-aebc-e7ca25d36b4a\") " pod="openshift-ingress/e2e-api-b-bdc256-openshift-default-5cf7d97495-76wgr"
Oct 30 00:53:24.463992 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:53:24.460240    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcdd7ce4-a94f-4e6d-aebc-e7ca25d36b4a-workload-socket\") pod \"e2e-api-b-bdc256-openshift-default-5cf7d97495-76wgr\" (UID: \"bcdd7ce4-a94f-4e6d-aebc-e7ca25d36b4a\") " pod="openshift-ingress/e2e-api-b-bdc256-openshift-default-5cf7d97495-76wgr"
Oct 30 00:53:24.478403 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:53:24.473182    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bcdd7ce4-a94f-4e6d-aebc-e7ca25d36b4a-istio-data\") pod \"e2e-api-b-bdc256-openshift-default-5cf7d97495-76wgr\" (UID: \"bcdd7ce4-a94f-4e6d-aebc-e7ca25d36b4a\") " pod="openshift-ingress/e2e-api-b-bdc256-openshift-default-5cf7d97495-76wgr"
Oct 30 00:53:24.489668 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:53:24.464569    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcdd7ce4-a94f-4e6d-aebc-e7ca25d36b4a-workload-socket\") pod \"e2e-api-b-bdc256-openshift-default-5cf7d97495-76wgr\" (UID: \"bcdd7ce4-a94f-4e6d-aebc-e7ca25d36b4a\") " pod="openshift-ingress/e2e-api-b-bdc256-openshift-default-5cf7d97495-76wgr"
Oct 30 00:53:24.489906 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:53:24.464815    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcdd7ce4-a94f-4e6d-aebc-e7ca25d36b4a-credential-socket\") pod \"e2e-api-b-bdc256-openshift-default-5cf7d97495-76wgr\" (UID: \"bcdd7ce4-a94f-4e6d-aebc-e7ca25d36b4a\") " pod="openshift-ingress/e2e-api-b-bdc256-openshift-default-5cf7d97495-76wgr"
Oct 30 00:53:24.490035 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:53:24.465190    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bcdd7ce4-a94f-4e6d-aebc-e7ca25d36b4a-istiod-ca-cert\") pod \"e2e-api-b-bdc256-openshift-default-5cf7d97495-76wgr\" (UID: \"bcdd7ce4-a94f-4e6d-aebc-e7ca25d36b4a\") " pod="openshift-ingress/e2e-api-b-bdc256-openshift-default-5cf7d97495-76wgr"
Oct 30 00:53:24.490124 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:53:24.464513    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bcdd7ce4-a94f-4e6d-aebc-e7ca25d36b4a-workload-certs\") pod \"e2e-api-b-bdc256-openshift-default-5cf7d97495-76wgr\" (UID: \"bcdd7ce4-a94f-4e6d-aebc-e7ca25d36b4a\") " pod="openshift-ingress/e2e-api-b-bdc256-openshift-default-5cf7d97495-76wgr"
Oct 30 00:53:24.490209 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:53:24.479038    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bcdd7ce4-a94f-4e6d-aebc-e7ca25d36b4a-istio-data\") pod \"e2e-api-b-bdc256-openshift-default-5cf7d97495-76wgr\" (UID: \"bcdd7ce4-a94f-4e6d-aebc-e7ca25d36b4a\") " pod="openshift-ingress/e2e-api-b-bdc256-openshift-default-5cf7d97495-76wgr"
Oct 30 00:53:24.512740 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:53:24.512665    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bcdd7ce4-a94f-4e6d-aebc-e7ca25d36b4a-istio-envoy\") pod \"e2e-api-b-bdc256-openshift-default-5cf7d97495-76wgr\" (UID: \"bcdd7ce4-a94f-4e6d-aebc-e7ca25d36b4a\") " pod="openshift-ingress/e2e-api-b-bdc256-openshift-default-5cf7d97495-76wgr"
Oct 30 00:53:24.537413 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:53:24.533250    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z485d\" (UniqueName: \"kubernetes.io/projected/bcdd7ce4-a94f-4e6d-aebc-e7ca25d36b4a-kube-api-access-z485d\") pod \"e2e-api-b-bdc256-openshift-default-5cf7d97495-76wgr\" (UID: \"bcdd7ce4-a94f-4e6d-aebc-e7ca25d36b4a\") " pod="openshift-ingress/e2e-api-b-bdc256-openshift-default-5cf7d97495-76wgr"
Oct 30 00:53:24.549410 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:53:24.544470    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bcdd7ce4-a94f-4e6d-aebc-e7ca25d36b4a-istio-podinfo\") pod \"e2e-api-b-bdc256-openshift-default-5cf7d97495-76wgr\" (UID: \"bcdd7ce4-a94f-4e6d-aebc-e7ca25d36b4a\") " pod="openshift-ingress/e2e-api-b-bdc256-openshift-default-5cf7d97495-76wgr"
Oct 30 00:53:24.549410 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:53:24.546852    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bcdd7ce4-a94f-4e6d-aebc-e7ca25d36b4a-istio-token\") pod \"e2e-api-b-bdc256-openshift-default-5cf7d97495-76wgr\" (UID: \"bcdd7ce4-a94f-4e6d-aebc-e7ca25d36b4a\") " pod="openshift-ingress/e2e-api-b-bdc256-openshift-default-5cf7d97495-76wgr"
Oct 30 00:53:24.592128 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:53:24.590682    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-bdc256-openshift-default-5cf7d97495-76wgr"
Oct 30 00:53:25.136495 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:53:25.136461    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-bdc256-openshift-default-5cf7d97495-76wgr"]
Oct 30 00:53:25.136889 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:53:25.136865    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcdd7ce4_a94f_4e6d_aebc_e7ca25d36b4a.slice/crio-4799dafcdd2933c39d8e11a9c6ef66a63339e8a279419e41de7af5265655a357 WatchSource:0}: Error finding container 4799dafcdd2933c39d8e11a9c6ef66a63339e8a279419e41de7af5265655a357: Status 404 returned error can't find the container with id 4799dafcdd2933c39d8e11a9c6ef66a63339e8a279419e41de7af5265655a357
Oct 30 00:53:25.139613 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:53:25.139582    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 00:53:25.139692 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:53:25.139664    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 00:53:25.139732 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:53:25.139693    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 00:53:26.058790 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:53:26.058682    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-bdc256-openshift-default-5cf7d97495-76wgr" event={"ID":"bcdd7ce4-a94f-4e6d-aebc-e7ca25d36b4a","Type":"ContainerStarted","Data":"8e1a6f5e4f08a1f82b240d45bd29abbd9c2567543a4205deb79fd92c69f4714c"}
Oct 30 00:53:26.058790 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:53:26.058771    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-bdc256-openshift-default-5cf7d97495-76wgr" event={"ID":"bcdd7ce4-a94f-4e6d-aebc-e7ca25d36b4a","Type":"ContainerStarted","Data":"4799dafcdd2933c39d8e11a9c6ef66a63339e8a279419e41de7af5265655a357"}
Oct 30 00:53:26.126086 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:53:26.126027    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-bdc256-openshift-default-5cf7d97495-76wgr" podStartSLOduration=2.126011516 podStartE2EDuration="2.126011516s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:53:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:53:26.122640082 +0000 UTC m=+1301.739072963" watchObservedRunningTime="2026-10-30 00:53:26.126011516 +0000 UTC m=+1301.742444396"
Oct 30 00:53:26.591347 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:53:26.591310    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-bdc256-openshift-default-5cf7d97495-76wgr"
Oct 30 00:53:26.597015 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:53:26.596988    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-bdc256-openshift-default-5cf7d97495-76wgr"
Oct 30 00:53:27.066989 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:53:27.066957    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-bdc256-openshift-default-5cf7d97495-76wgr"
Oct 30 00:53:27.067918 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:53:27.067896    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-bdc256-openshift-default-5cf7d97495-76wgr"
Oct 30 00:53:29.029934 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:53:29.029899    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-655f3853-6rtdl"]
Oct 30 00:53:29.045442 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:53:29.045410    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-655f3853-6rtdl"]
Oct 30 00:53:30.606283 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:53:30.606242    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c4ea6fd-d9b4-4a04-b3cd-dd2707844a2c" path="/var/lib/kubelet/pods/2c4ea6fd-d9b4-4a04-b3cd-dd2707844a2c/volumes"
Oct 30 00:53:31.340774 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:53:31.340740    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7d8z4"]
Oct 30 00:53:33.032843 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:53:33.032804    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a14ef684-fl4gj"]
Oct 30 00:53:33.054780 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:53:33.054743    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a14ef684-fl4gj"]
Oct 30 00:53:34.578403 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:53:34.578140    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7245d167-0ff4-4064-892d-e4d91aed3ee0" path="/var/lib/kubelet/pods/7245d167-0ff4-4064-892d-e4d91aed3ee0/volumes"
Oct 30 00:53:35.281436 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:53:35.280472    5002 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-52650f/simulator-w0-52650f-kserve-58b64766c-94qxx" podUID="659c1425-5b67-4d01-bfc2-f1ba56b9a59b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:53:35.323405 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:53:35.322482    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1a11cf/distinct-w2-1a11cf-kserve-854cf4cdf8-cddw6" podUID="32a03478-46eb-4363-97b6-51a32c4099c7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:53:35.323405 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:53:35.322972    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1a11cf/distinct-2-w2-1a11cf-kserve-758bb6dd57-j5f2k" podUID="f2d66dea-9f0b-43f3-a774-bb065e88f378" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded"
Oct 30 00:53:35.377475 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:53:35.377427    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6f4fba/simulator-w5-6f4fba-kserve-5bf8dfddd5-wcwzh" podUID="3a2c2f8f-649c-4ab6-af98-5d87df0e511f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/ready\": context deadline exceeded"
Oct 30 00:53:35.402911 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:53:35.402639    5002 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-1a11cf/distinct-w2-1a11cf-kserve-854cf4cdf8-cddw6" podUID="32a03478-46eb-4363-97b6-51a32c4099c7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:53:35.760537 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:53:35.760463    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cefb27/simulator-w3-cefb27-kserve-6b669c9d9f-zx2vm" podUID="7121e81e-f903-42c0-a7a4-bc18d49194e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:53:35.825790 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:53:35.825746    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-52650f/simulator-w0-52650f-kserve-58b64766c-94qxx" podUID="659c1425-5b67-4d01-bfc2-f1ba56b9a59b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": context deadline exceeded"
Oct 30 00:53:36.684522 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:53:36.683983    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-52650f/premium-w0-52650f-kserve-c5bf66f9d-qq5dm" podUID="3455d983-b11b-4cee-bfce-d1f399cf9679" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:53:38.041671 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:53:38.041633    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4932e2d1-hlzs8"]
Oct 30 00:53:38.053766 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:53:38.053733    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4932e2d1-hlzs8"]
Oct 30 00:53:38.498027 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:53:38.497992    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d059dbd-6736-48b6-9a97-7c54269ee2e9" path="/var/lib/kubelet/pods/9d059dbd-6736-48b6-9a97-7c54269ee2e9/volumes"
Oct 30 00:53:42.865341 ip-10-0-128-7 kubenswrapper[5002]: E1030 00:53:42.865298    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbjdq" containerName="extractor"
Oct 30 00:53:43.034751 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:53:43.034704    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0ea9289-t5f56"]
Oct 30 00:53:43.046166 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:53:43.046120    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0ea9289-t5f56"]
Oct 30 00:53:44.524466 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:53:44.524428    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba2785d1-40fc-4081-ab4b-71f251ffed23" path="/var/lib/kubelet/pods/ba2785d1-40fc-4081-ab4b-71f251ffed23/volumes"
Oct 30 00:53:46.182425 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:53:46.182390    5002 scope.go:122] "RemoveContainer" containerID="510087b5d66aedeaefb7fd3863474957094da09e5f8e146d718256efcc730d33"
Oct 30 00:53:46.198334 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:53:46.198307    5002 scope.go:122] "RemoveContainer" containerID="29ca663c81e62a7da53693c4b8536acba9b2157c98f854d3a288b6df166484ad"
Oct 30 00:53:48.030174 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:53:48.030138    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bedf5b47-7qtsj"]
Oct 30 00:53:48.044169 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:53:48.044138    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bedf5b47-7qtsj"]
Oct 30 00:53:48.496873 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:53:48.496835    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2ae039e-d1d3-4190-8781-f58a8f778eee" path="/var/lib/kubelet/pods/c2ae039e-d1d3-4190-8781-f58a8f778eee/volumes"
Oct 30 00:53:54.703538 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:53:54.703486    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca5363/distinct-2-w4-ca5363-kserve-576576f9b5-xkkqc" podUID="5de8bebf-cfad-49da-9a98-08aedde4442c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:53:54.853816 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:53:54.853764    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-52650f/simulator-w0-52650f-kserve-58b64766c-94qxx" podUID="659c1425-5b67-4d01-bfc2-f1ba56b9a59b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": context deadline exceeded"
Oct 30 00:53:55.263522 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:53:55.263476    5002 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-52650f/simulator-w0-52650f-kserve-58b64766c-94qxx" podUID="659c1425-5b67-4d01-bfc2-f1ba56b9a59b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:53:55.711537 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:53:55.711485    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca5363/distinct-2-w4-ca5363-kserve-576576f9b5-xkkqc" podUID="5de8bebf-cfad-49da-9a98-08aedde4442c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:53:55.714152 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:53:55.714124    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ca5363/distinct-2-w4-ca5363-kserve-576576f9b5-xkkqc"
Oct 30 00:53:55.864409 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:53:55.863186    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-52650f/simulator-w0-52650f-kserve-58b64766c-94qxx" podUID="659c1425-5b67-4d01-bfc2-f1ba56b9a59b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:53:55.864409 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:53:55.863926    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-52650f/simulator-w0-52650f-kserve-58b64766c-94qxx"
Oct 30 00:53:56.027856 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:53:56.027765    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-29be73/simulator-w1-29be73-kserve-9958d76db-fks2t" podUID="fbb3c806-8967-4d39-84c6-9dc022fc5cae" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:53:56.057666 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:53:56.050735    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-29be73/premium-w1-29be73-kserve-7b777cfc8c-7kgp5" podUID="e6fabbdd-86ca-4cbf-85be-ef250575a355" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:53:56.057666 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:53:56.055481    5002 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-fwrpg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 00:53:56.057666 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:53:56.055549    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-fwrpg" podUID="a498076a-ee51-4409-b7d4-a0fabd0f385e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:53:56.072121 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:53:56.072084    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-34f9d3da-sm7z5"]
Oct 30 00:53:56.072355 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:53:56.072341    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-34f9d3da-sm7z5"]
Oct 30 00:53:56.273404 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:53:56.272835    5002 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-jwx9q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 00:53:56.273404 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:53:56.272929    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-jwx9q" podUID="a4f0d6bc-42a1-4f97-8713-daad565c8d7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:53:56.568708 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:53:56.568668    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="065123cf-d6c8-4282-81b6-7d4fb04c490a" path="/var/lib/kubelet/pods/065123cf-d6c8-4282-81b6-7d4fb04c490a/volumes"
Oct 30 00:53:56.637396 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:53:56.636680    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-52650f/simulator-w0-52650f-kserve-58b64766c-94qxx"
Oct 30 00:53:56.725829 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:53:56.725235    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca5363/distinct-2-w4-ca5363-kserve-576576f9b5-xkkqc" podUID="5de8bebf-cfad-49da-9a98-08aedde4442c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:53:56.877461 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:53:56.816810    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ca5363/distinct-2-w4-ca5363-kserve-576576f9b5-xkkqc"
Oct 30 00:53:56.982833 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:53:56.982795    5002 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-4nbkc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 00:53:56.983953 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:53:56.983916    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-4nbkc" podUID="6259aaf4-6f22-4781-8265-5ed5724b0f41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:54:05.353614 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:05.353575    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-02fbf5-openshift-default-64fc59d485-jhdtc"]
Oct 30 00:54:05.359632 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:05.359608    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-02fbf5-openshift-default-64fc59d485-jhdtc"
Oct 30 00:54:05.388586 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:05.388551    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-02fbf5-openshift-default-64fc59d485-jhdtc"]
Oct 30 00:54:05.408452 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:05.408367    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-02fbf5-openshift-default-dockercfg-hns6d\""
Oct 30 00:54:05.436578 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:05.436546    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58608e31-3cf2-4a5c-9d19-c8c72b774f51-workload-socket\") pod \"e2e-api-b-02fbf5-openshift-default-64fc59d485-jhdtc\" (UID: \"58608e31-3cf2-4a5c-9d19-c8c72b774f51\") " pod="openshift-ingress/e2e-api-b-02fbf5-openshift-default-64fc59d485-jhdtc"
Oct 30 00:54:05.436933 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:05.436906    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58608e31-3cf2-4a5c-9d19-c8c72b774f51-istio-token\") pod \"e2e-api-b-02fbf5-openshift-default-64fc59d485-jhdtc\" (UID: \"58608e31-3cf2-4a5c-9d19-c8c72b774f51\") " pod="openshift-ingress/e2e-api-b-02fbf5-openshift-default-64fc59d485-jhdtc"
Oct 30 00:54:05.437075 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:05.437057    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58608e31-3cf2-4a5c-9d19-c8c72b774f51-istio-envoy\") pod \"e2e-api-b-02fbf5-openshift-default-64fc59d485-jhdtc\" (UID: \"58608e31-3cf2-4a5c-9d19-c8c72b774f51\") " pod="openshift-ingress/e2e-api-b-02fbf5-openshift-default-64fc59d485-jhdtc"
Oct 30 00:54:05.437193 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:05.437176    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58608e31-3cf2-4a5c-9d19-c8c72b774f51-credential-socket\") pod \"e2e-api-b-02fbf5-openshift-default-64fc59d485-jhdtc\" (UID: \"58608e31-3cf2-4a5c-9d19-c8c72b774f51\") " pod="openshift-ingress/e2e-api-b-02fbf5-openshift-default-64fc59d485-jhdtc"
Oct 30 00:54:05.437318 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:05.437303    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4fkl\" (UniqueName: \"kubernetes.io/projected/58608e31-3cf2-4a5c-9d19-c8c72b774f51-kube-api-access-p4fkl\") pod \"e2e-api-b-02fbf5-openshift-default-64fc59d485-jhdtc\" (UID: \"58608e31-3cf2-4a5c-9d19-c8c72b774f51\") " pod="openshift-ingress/e2e-api-b-02fbf5-openshift-default-64fc59d485-jhdtc"
Oct 30 00:54:05.437450 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:05.437434    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58608e31-3cf2-4a5c-9d19-c8c72b774f51-istio-data\") pod \"e2e-api-b-02fbf5-openshift-default-64fc59d485-jhdtc\" (UID: \"58608e31-3cf2-4a5c-9d19-c8c72b774f51\") " pod="openshift-ingress/e2e-api-b-02fbf5-openshift-default-64fc59d485-jhdtc"
Oct 30 00:54:05.437577 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:05.437562    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58608e31-3cf2-4a5c-9d19-c8c72b774f51-istio-podinfo\") pod \"e2e-api-b-02fbf5-openshift-default-64fc59d485-jhdtc\" (UID: \"58608e31-3cf2-4a5c-9d19-c8c72b774f51\") " pod="openshift-ingress/e2e-api-b-02fbf5-openshift-default-64fc59d485-jhdtc"
Oct 30 00:54:05.437827 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:05.437810    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58608e31-3cf2-4a5c-9d19-c8c72b774f51-istiod-ca-cert\") pod \"e2e-api-b-02fbf5-openshift-default-64fc59d485-jhdtc\" (UID: \"58608e31-3cf2-4a5c-9d19-c8c72b774f51\") " pod="openshift-ingress/e2e-api-b-02fbf5-openshift-default-64fc59d485-jhdtc"
Oct 30 00:54:05.437973 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:05.437958    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58608e31-3cf2-4a5c-9d19-c8c72b774f51-workload-certs\") pod \"e2e-api-b-02fbf5-openshift-default-64fc59d485-jhdtc\" (UID: \"58608e31-3cf2-4a5c-9d19-c8c72b774f51\") " pod="openshift-ingress/e2e-api-b-02fbf5-openshift-default-64fc59d485-jhdtc"
Oct 30 00:54:05.541017 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:05.540970    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4fkl\" (UniqueName: \"kubernetes.io/projected/58608e31-3cf2-4a5c-9d19-c8c72b774f51-kube-api-access-p4fkl\") pod \"e2e-api-b-02fbf5-openshift-default-64fc59d485-jhdtc\" (UID: \"58608e31-3cf2-4a5c-9d19-c8c72b774f51\") " pod="openshift-ingress/e2e-api-b-02fbf5-openshift-default-64fc59d485-jhdtc"
Oct 30 00:54:05.575410 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:05.564535    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58608e31-3cf2-4a5c-9d19-c8c72b774f51-istio-data\") pod \"e2e-api-b-02fbf5-openshift-default-64fc59d485-jhdtc\" (UID: \"58608e31-3cf2-4a5c-9d19-c8c72b774f51\") " pod="openshift-ingress/e2e-api-b-02fbf5-openshift-default-64fc59d485-jhdtc"
Oct 30 00:54:05.575410 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:05.564620    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58608e31-3cf2-4a5c-9d19-c8c72b774f51-istio-podinfo\") pod \"e2e-api-b-02fbf5-openshift-default-64fc59d485-jhdtc\" (UID: \"58608e31-3cf2-4a5c-9d19-c8c72b774f51\") " pod="openshift-ingress/e2e-api-b-02fbf5-openshift-default-64fc59d485-jhdtc"
Oct 30 00:54:05.575410 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:05.564832    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58608e31-3cf2-4a5c-9d19-c8c72b774f51-istiod-ca-cert\") pod \"e2e-api-b-02fbf5-openshift-default-64fc59d485-jhdtc\" (UID: \"58608e31-3cf2-4a5c-9d19-c8c72b774f51\") " pod="openshift-ingress/e2e-api-b-02fbf5-openshift-default-64fc59d485-jhdtc"
Oct 30 00:54:05.575410 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:05.564904    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58608e31-3cf2-4a5c-9d19-c8c72b774f51-workload-certs\") pod \"e2e-api-b-02fbf5-openshift-default-64fc59d485-jhdtc\" (UID: \"58608e31-3cf2-4a5c-9d19-c8c72b774f51\") " pod="openshift-ingress/e2e-api-b-02fbf5-openshift-default-64fc59d485-jhdtc"
Oct 30 00:54:05.575410 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:05.565018    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58608e31-3cf2-4a5c-9d19-c8c72b774f51-workload-socket\") pod \"e2e-api-b-02fbf5-openshift-default-64fc59d485-jhdtc\" (UID: \"58608e31-3cf2-4a5c-9d19-c8c72b774f51\") " pod="openshift-ingress/e2e-api-b-02fbf5-openshift-default-64fc59d485-jhdtc"
Oct 30 00:54:05.575410 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:05.565199    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58608e31-3cf2-4a5c-9d19-c8c72b774f51-istio-token\") pod \"e2e-api-b-02fbf5-openshift-default-64fc59d485-jhdtc\" (UID: \"58608e31-3cf2-4a5c-9d19-c8c72b774f51\") " pod="openshift-ingress/e2e-api-b-02fbf5-openshift-default-64fc59d485-jhdtc"
Oct 30 00:54:05.575410 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:05.565240    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58608e31-3cf2-4a5c-9d19-c8c72b774f51-istio-envoy\") pod \"e2e-api-b-02fbf5-openshift-default-64fc59d485-jhdtc\" (UID: \"58608e31-3cf2-4a5c-9d19-c8c72b774f51\") " pod="openshift-ingress/e2e-api-b-02fbf5-openshift-default-64fc59d485-jhdtc"
Oct 30 00:54:05.575410 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:05.565280    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58608e31-3cf2-4a5c-9d19-c8c72b774f51-credential-socket\") pod \"e2e-api-b-02fbf5-openshift-default-64fc59d485-jhdtc\" (UID: \"58608e31-3cf2-4a5c-9d19-c8c72b774f51\") " pod="openshift-ingress/e2e-api-b-02fbf5-openshift-default-64fc59d485-jhdtc"
Oct 30 00:54:05.575410 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:05.565844    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58608e31-3cf2-4a5c-9d19-c8c72b774f51-workload-certs\") pod \"e2e-api-b-02fbf5-openshift-default-64fc59d485-jhdtc\" (UID: \"58608e31-3cf2-4a5c-9d19-c8c72b774f51\") " pod="openshift-ingress/e2e-api-b-02fbf5-openshift-default-64fc59d485-jhdtc"
Oct 30 00:54:05.575410 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:05.565871    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58608e31-3cf2-4a5c-9d19-c8c72b774f51-credential-socket\") pod \"e2e-api-b-02fbf5-openshift-default-64fc59d485-jhdtc\" (UID: \"58608e31-3cf2-4a5c-9d19-c8c72b774f51\") " pod="openshift-ingress/e2e-api-b-02fbf5-openshift-default-64fc59d485-jhdtc"
Oct 30 00:54:05.575410 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:05.566146    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58608e31-3cf2-4a5c-9d19-c8c72b774f51-workload-socket\") pod \"e2e-api-b-02fbf5-openshift-default-64fc59d485-jhdtc\" (UID: \"58608e31-3cf2-4a5c-9d19-c8c72b774f51\") " pod="openshift-ingress/e2e-api-b-02fbf5-openshift-default-64fc59d485-jhdtc"
Oct 30 00:54:05.575410 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:05.566429    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58608e31-3cf2-4a5c-9d19-c8c72b774f51-istio-data\") pod \"e2e-api-b-02fbf5-openshift-default-64fc59d485-jhdtc\" (UID: \"58608e31-3cf2-4a5c-9d19-c8c72b774f51\") " pod="openshift-ingress/e2e-api-b-02fbf5-openshift-default-64fc59d485-jhdtc"
Oct 30 00:54:05.575410 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:05.566553    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58608e31-3cf2-4a5c-9d19-c8c72b774f51-istiod-ca-cert\") pod \"e2e-api-b-02fbf5-openshift-default-64fc59d485-jhdtc\" (UID: \"58608e31-3cf2-4a5c-9d19-c8c72b774f51\") " pod="openshift-ingress/e2e-api-b-02fbf5-openshift-default-64fc59d485-jhdtc"
Oct 30 00:54:05.810942 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:05.810904    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58608e31-3cf2-4a5c-9d19-c8c72b774f51-istio-podinfo\") pod \"e2e-api-b-02fbf5-openshift-default-64fc59d485-jhdtc\" (UID: \"58608e31-3cf2-4a5c-9d19-c8c72b774f51\") " pod="openshift-ingress/e2e-api-b-02fbf5-openshift-default-64fc59d485-jhdtc"
Oct 30 00:54:05.811468 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:05.811436    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58608e31-3cf2-4a5c-9d19-c8c72b774f51-istio-envoy\") pod \"e2e-api-b-02fbf5-openshift-default-64fc59d485-jhdtc\" (UID: \"58608e31-3cf2-4a5c-9d19-c8c72b774f51\") " pod="openshift-ingress/e2e-api-b-02fbf5-openshift-default-64fc59d485-jhdtc"
Oct 30 00:54:05.812367 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:05.812346    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4fkl\" (UniqueName: \"kubernetes.io/projected/58608e31-3cf2-4a5c-9d19-c8c72b774f51-kube-api-access-p4fkl\") pod \"e2e-api-b-02fbf5-openshift-default-64fc59d485-jhdtc\" (UID: \"58608e31-3cf2-4a5c-9d19-c8c72b774f51\") " pod="openshift-ingress/e2e-api-b-02fbf5-openshift-default-64fc59d485-jhdtc"
Oct 30 00:54:05.822773 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:05.822716    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58608e31-3cf2-4a5c-9d19-c8c72b774f51-istio-token\") pod \"e2e-api-b-02fbf5-openshift-default-64fc59d485-jhdtc\" (UID: \"58608e31-3cf2-4a5c-9d19-c8c72b774f51\") " pod="openshift-ingress/e2e-api-b-02fbf5-openshift-default-64fc59d485-jhdtc"
Oct 30 00:54:05.994756 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:05.994716    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-02fbf5-openshift-default-64fc59d485-jhdtc"
Oct 30 00:54:06.328314 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:54:06.328272    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58608e31_3cf2_4a5c_9d19_c8c72b774f51.slice/crio-4b1608c79c2227804d1079325de4c888039a28c1569ed65f1edbe6a5ce241796 WatchSource:0}: Error finding container 4b1608c79c2227804d1079325de4c888039a28c1569ed65f1edbe6a5ce241796: Status 404 returned error can't find the container with id 4b1608c79c2227804d1079325de4c888039a28c1569ed65f1edbe6a5ce241796
Oct 30 00:54:06.329415 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:06.329367    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-02fbf5-openshift-default-64fc59d485-jhdtc"]
Oct 30 00:54:06.331634 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:06.331597    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 00:54:06.331737 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:06.331670    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 00:54:06.331737 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:06.331702    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 00:54:06.872444 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:06.872367    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-02fbf5-openshift-default-64fc59d485-jhdtc" event={"ID":"58608e31-3cf2-4a5c-9d19-c8c72b774f51","Type":"ContainerStarted","Data":"0cf4e79fe5ba7762ac6dae465a00bf85949ae7159a9c4ad0b2992c31d13e23aa"}
Oct 30 00:54:06.872921 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:06.872472    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-02fbf5-openshift-default-64fc59d485-jhdtc" event={"ID":"58608e31-3cf2-4a5c-9d19-c8c72b774f51","Type":"ContainerStarted","Data":"4b1608c79c2227804d1079325de4c888039a28c1569ed65f1edbe6a5ce241796"}
Oct 30 00:54:06.923351 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:06.923300    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-02fbf5-openshift-default-64fc59d485-jhdtc" podStartSLOduration=1.9232875489999999 podStartE2EDuration="1.923287549s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:54:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:54:06.920111759 +0000 UTC m=+1342.536544659" watchObservedRunningTime="2026-10-30 00:54:06.923287549 +0000 UTC m=+1342.539720428"
Oct 30 00:54:06.995503 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:06.995458    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-02fbf5-openshift-default-64fc59d485-jhdtc"
Oct 30 00:54:08.001878 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:08.001841    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-02fbf5-openshift-default-64fc59d485-jhdtc"
Oct 30 00:54:08.889119 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:08.889079    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-02fbf5-openshift-default-64fc59d485-jhdtc"
Oct 30 00:54:08.890265 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:08.890238    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-02fbf5-openshift-default-64fc59d485-jhdtc"
Oct 30 00:54:11.357956 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:11.357919    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-02fbf5-764c8d7f95-qdnr5"]
Oct 30 00:54:11.362353 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:11.362330    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-02fbf5-764c8d7f95-qdnr5"
Oct 30 00:54:11.384622 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:11.384595    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-02fbf5-764c8d7f95-qdnr5"]
Oct 30 00:54:11.396230 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:11.393237    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-02fbf5\""
Oct 30 00:54:11.396230 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:11.393496    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-02fbf5-dockercfg-wgb9c\""
Oct 30 00:54:11.544402 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:11.513849    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c5e0e1c-4526-4b7b-80fb-c65730081763-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-02fbf5-764c8d7f95-qdnr5\" (UID: \"1c5e0e1c-4526-4b7b-80fb-c65730081763\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-02fbf5-764c8d7f95-qdnr5"
Oct 30 00:54:11.613472 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:11.588958    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cr82w\" (UniqueName: \"kubernetes.io/projected/1c5e0e1c-4526-4b7b-80fb-c65730081763-kube-api-access-cr82w\") pod \"payload-pre-processing-e2e-api-b-02fbf5-764c8d7f95-qdnr5\" (UID: \"1c5e0e1c-4526-4b7b-80fb-c65730081763\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-02fbf5-764c8d7f95-qdnr5"
Oct 30 00:54:11.650713 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:11.613693    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-02fbf5-6b759bfc8-7kvrs"]
Oct 30 00:54:11.723156 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:11.671889    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-02fbf5-6b759bfc8-7kvrs"
Oct 30 00:54:11.723156 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:11.696208    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c5e0e1c-4526-4b7b-80fb-c65730081763-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-02fbf5-764c8d7f95-qdnr5\" (UID: \"1c5e0e1c-4526-4b7b-80fb-c65730081763\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-02fbf5-764c8d7f95-qdnr5"
Oct 30 00:54:11.723156 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:11.696321    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cr82w\" (UniqueName: \"kubernetes.io/projected/1c5e0e1c-4526-4b7b-80fb-c65730081763-kube-api-access-cr82w\") pod \"payload-pre-processing-e2e-api-b-02fbf5-764c8d7f95-qdnr5\" (UID: \"1c5e0e1c-4526-4b7b-80fb-c65730081763\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-02fbf5-764c8d7f95-qdnr5"
Oct 30 00:54:11.723156 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:11.720571    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c5e0e1c-4526-4b7b-80fb-c65730081763-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-02fbf5-764c8d7f95-qdnr5\" (UID: \"1c5e0e1c-4526-4b7b-80fb-c65730081763\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-02fbf5-764c8d7f95-qdnr5"
Oct 30 00:54:11.724886 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:11.724861    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-02fbf5-6b759bfc8-7kvrs"]
Oct 30 00:54:11.795452 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:11.793334    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cr82w\" (UniqueName: \"kubernetes.io/projected/1c5e0e1c-4526-4b7b-80fb-c65730081763-kube-api-access-cr82w\") pod \"payload-pre-processing-e2e-api-b-02fbf5-764c8d7f95-qdnr5\" (UID: \"1c5e0e1c-4526-4b7b-80fb-c65730081763\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-02fbf5-764c8d7f95-qdnr5"
Oct 30 00:54:11.798543 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:11.798500    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87a023ea-b2a5-438d-b211-983374118e75-plugins-config-volume\") pod \"payload-processing-e2e-api-b-02fbf5-6b759bfc8-7kvrs\" (UID: \"87a023ea-b2a5-438d-b211-983374118e75\") " pod="openshift-ingress/payload-processing-e2e-api-b-02fbf5-6b759bfc8-7kvrs"
Oct 30 00:54:11.798706 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:11.798557    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgp57\" (UniqueName: \"kubernetes.io/projected/87a023ea-b2a5-438d-b211-983374118e75-kube-api-access-tgp57\") pod \"payload-processing-e2e-api-b-02fbf5-6b759bfc8-7kvrs\" (UID: \"87a023ea-b2a5-438d-b211-983374118e75\") " pod="openshift-ingress/payload-processing-e2e-api-b-02fbf5-6b759bfc8-7kvrs"
Oct 30 00:54:11.900014 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:11.899909    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87a023ea-b2a5-438d-b211-983374118e75-plugins-config-volume\") pod \"payload-processing-e2e-api-b-02fbf5-6b759bfc8-7kvrs\" (UID: \"87a023ea-b2a5-438d-b211-983374118e75\") " pod="openshift-ingress/payload-processing-e2e-api-b-02fbf5-6b759bfc8-7kvrs"
Oct 30 00:54:11.900014 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:11.899964    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgp57\" (UniqueName: \"kubernetes.io/projected/87a023ea-b2a5-438d-b211-983374118e75-kube-api-access-tgp57\") pod \"payload-processing-e2e-api-b-02fbf5-6b759bfc8-7kvrs\" (UID: \"87a023ea-b2a5-438d-b211-983374118e75\") " pod="openshift-ingress/payload-processing-e2e-api-b-02fbf5-6b759bfc8-7kvrs"
Oct 30 00:54:11.900705 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:11.900683    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87a023ea-b2a5-438d-b211-983374118e75-plugins-config-volume\") pod \"payload-processing-e2e-api-b-02fbf5-6b759bfc8-7kvrs\" (UID: \"87a023ea-b2a5-438d-b211-983374118e75\") " pod="openshift-ingress/payload-processing-e2e-api-b-02fbf5-6b759bfc8-7kvrs"
Oct 30 00:54:11.925106 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:11.925075    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgp57\" (UniqueName: \"kubernetes.io/projected/87a023ea-b2a5-438d-b211-983374118e75-kube-api-access-tgp57\") pod \"payload-processing-e2e-api-b-02fbf5-6b759bfc8-7kvrs\" (UID: \"87a023ea-b2a5-438d-b211-983374118e75\") " pod="openshift-ingress/payload-processing-e2e-api-b-02fbf5-6b759bfc8-7kvrs"
Oct 30 00:54:11.978202 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:11.978161    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-02fbf5-764c8d7f95-qdnr5"
Oct 30 00:54:12.052958 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:12.052772    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-02fbf5-6b759bfc8-7kvrs"
Oct 30 00:54:12.287816 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:12.287779    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-02fbf5-764c8d7f95-qdnr5"]
Oct 30 00:54:12.423406 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:12.423355    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-02fbf5-6b759bfc8-7kvrs"]
Oct 30 00:54:12.727406 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:12.715555    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7d8z4"]
Oct 30 00:54:12.901682 ip-10-0-128-7 kubenswrapper[5002]: E1030 00:54:12.901633    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbjdq" containerName="extractor"
Oct 30 00:54:12.944808 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:12.944760    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-02fbf5-6b759bfc8-7kvrs" event={"ID":"87a023ea-b2a5-438d-b211-983374118e75","Type":"ContainerStarted","Data":"fe32d6c434906f369be04944285863cf409d69efb2fd1f383e4c76aa81ab0092"}
Oct 30 00:54:12.947260 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:12.947231    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-02fbf5-764c8d7f95-qdnr5" event={"ID":"1c5e0e1c-4526-4b7b-80fb-c65730081763","Type":"ContainerStarted","Data":"e8b037aade058da052efe2e96f6d3e8b538501ce313d95bdabaabdf4d90d2505"}
Oct 30 00:54:19.120358 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:19.120320    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-02fbf5-6b759bfc8-7kvrs" event={"ID":"87a023ea-b2a5-438d-b211-983374118e75","Type":"ContainerStarted","Data":"f1e4eccc7c7314a783029ba7c62a0400cdebc2cad765d2884707d2ce0456e065"}
Oct 30 00:54:19.120812 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:19.120419    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-02fbf5-6b759bfc8-7kvrs"
Oct 30 00:54:19.121860 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:19.121837    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-02fbf5-764c8d7f95-qdnr5" event={"ID":"1c5e0e1c-4526-4b7b-80fb-c65730081763","Type":"ContainerStarted","Data":"a48b62526972779f30269ac650ada11591fa7475aea4363d44f0491c63e93eb9"}
Oct 30 00:54:19.122176 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:19.121964    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-02fbf5-764c8d7f95-qdnr5"
Oct 30 00:54:19.169406 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:19.165023    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-02fbf5-6b759bfc8-7kvrs" podStartSLOduration=2.355046766 podStartE2EDuration="8.16500506s" totalImagesPullingTime="5.809958294s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:54:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:54:19.163958933 +0000 UTC m=+1354.780391815" watchObservedRunningTime="2026-10-30 00:54:19.16500506 +0000 UTC m=+1354.781437941"
Oct 30 00:54:19.208686 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:19.208620    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-02fbf5-764c8d7f95-qdnr5" podStartSLOduration=2.083992414 podStartE2EDuration="8.208603189s" totalImagesPullingTime="6.124610775s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:54:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:54:19.207728594 +0000 UTC m=+1354.824161494" watchObservedRunningTime="2026-10-30 00:54:19.208603189 +0000 UTC m=+1354.825036084"
Oct 30 00:54:25.342510 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:25.342475    5002 generic.go:356] "Generic (PLEG): container finished" podID="bcdd7ce4-a94f-4e6d-aebc-e7ca25d36b4a" containerID="8e1a6f5e4f08a1f82b240d45bd29abbd9c2567543a4205deb79fd92c69f4714c" exitCode=0
Oct 30 00:54:25.342989 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:25.342577    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-bdc256-openshift-default-5cf7d97495-76wgr" event={"ID":"bcdd7ce4-a94f-4e6d-aebc-e7ca25d36b4a","Type":"ContainerDied","Data":"8e1a6f5e4f08a1f82b240d45bd29abbd9c2567543a4205deb79fd92c69f4714c"}
Oct 30 00:54:25.343163 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:25.343136    5002 scope.go:122] "RemoveContainer" containerID="8e1a6f5e4f08a1f82b240d45bd29abbd9c2567543a4205deb79fd92c69f4714c"
Oct 30 00:54:25.345077 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:25.345046    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 00:54:25.345175 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:25.345117    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 00:54:25.345175 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:25.345151    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 00:54:26.384399 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:26.381179    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-bdc256-openshift-default-5cf7d97495-76wgr" event={"ID":"bcdd7ce4-a94f-4e6d-aebc-e7ca25d36b4a","Type":"ContainerStarted","Data":"4a6d6ab8e1b9dc36ab6c48798066ffe77ce17701c29662248408e0814f525233"}
Oct 30 00:54:26.591892 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:26.591854    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-bdc256-openshift-default-5cf7d97495-76wgr"
Oct 30 00:54:26.593602 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:26.593571    5002 patch_prober.go:29] interesting pod/e2e-api-b-bdc256-openshift-default-5cf7d97495-76wgr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Oct 30 00:54:26.593742 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:26.593630    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-bdc256-openshift-default-5cf7d97495-76wgr" podUID="bcdd7ce4-a94f-4e6d-aebc-e7ca25d36b4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Oct 30 00:54:27.071401 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:27.070489    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-bdc256-openshift-default-5cf7d97495-76wgr"
Oct 30 00:54:27.285239 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:27.285201    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1fa555-rqbw9"]
Oct 30 00:54:27.290453 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:27.290427    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1fa555-rqbw9"
Oct 30 00:54:27.301094 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:27.301065    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1fa555-rqbw9"]
Oct 30 00:54:27.314427 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:27.313719    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 00:54:27.314427 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:27.314196    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 00:54:27.314427 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:27.314268    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5phsh\""
Oct 30 00:54:27.327877 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:27.327804    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 00:54:27.382451 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:27.370509    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5160630-28d3-4af0-a8e9-d4b08e9f70dd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-1fa555-rqbw9\" (UID: \"e5160630-28d3-4af0-a8e9-d4b08e9f70dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1fa555-rqbw9"
Oct 30 00:54:27.382867 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:27.382834    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5160630-28d3-4af0-a8e9-d4b08e9f70dd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-1fa555-rqbw9\" (UID: \"e5160630-28d3-4af0-a8e9-d4b08e9f70dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1fa555-rqbw9"
Oct 30 00:54:27.383089 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:27.383070    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glzs4\" (UniqueName: \"kubernetes.io/projected/e5160630-28d3-4af0-a8e9-d4b08e9f70dd-kube-api-access-glzs4\") pod \"maas-api-revoke-keys-e2e-api-a-1fa555-rqbw9\" (UID: \"e5160630-28d3-4af0-a8e9-d4b08e9f70dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1fa555-rqbw9"
Oct 30 00:54:27.484810 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:27.484756    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5160630-28d3-4af0-a8e9-d4b08e9f70dd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-1fa555-rqbw9\" (UID: \"e5160630-28d3-4af0-a8e9-d4b08e9f70dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1fa555-rqbw9"
Oct 30 00:54:27.485335 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:27.484869    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5160630-28d3-4af0-a8e9-d4b08e9f70dd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-1fa555-rqbw9\" (UID: \"e5160630-28d3-4af0-a8e9-d4b08e9f70dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1fa555-rqbw9"
Oct 30 00:54:27.485335 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:27.484919    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glzs4\" (UniqueName: \"kubernetes.io/projected/e5160630-28d3-4af0-a8e9-d4b08e9f70dd-kube-api-access-glzs4\") pod \"maas-api-revoke-keys-e2e-api-a-1fa555-rqbw9\" (UID: \"e5160630-28d3-4af0-a8e9-d4b08e9f70dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1fa555-rqbw9"
Oct 30 00:54:27.485719 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:27.485658    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5160630-28d3-4af0-a8e9-d4b08e9f70dd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-1fa555-rqbw9\" (UID: \"e5160630-28d3-4af0-a8e9-d4b08e9f70dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1fa555-rqbw9"
Oct 30 00:54:27.487761 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:27.487734    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5160630-28d3-4af0-a8e9-d4b08e9f70dd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-1fa555-rqbw9\" (UID: \"e5160630-28d3-4af0-a8e9-d4b08e9f70dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1fa555-rqbw9"
Oct 30 00:54:27.510308 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:27.510281    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-glzs4\" (UniqueName: \"kubernetes.io/projected/e5160630-28d3-4af0-a8e9-d4b08e9f70dd-kube-api-access-glzs4\") pod \"maas-api-revoke-keys-e2e-api-a-1fa555-rqbw9\" (UID: \"e5160630-28d3-4af0-a8e9-d4b08e9f70dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1fa555-rqbw9"
Oct 30 00:54:27.592270 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:27.592179    5002 patch_prober.go:29] interesting pod/e2e-api-b-bdc256-openshift-default-5cf7d97495-76wgr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Oct 30 00:54:27.592270 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:27.592241    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-bdc256-openshift-default-5cf7d97495-76wgr" podUID="bcdd7ce4-a94f-4e6d-aebc-e7ca25d36b4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Oct 30 00:54:27.616206 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:27.616170    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1fa555-rqbw9"
Oct 30 00:54:28.185782 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:28.185744    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1fa555-rqbw9"]
Oct 30 00:54:28.186090 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:54:28.186063    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5160630_28d3_4af0_a8e9_d4b08e9f70dd.slice/crio-dc2dd7414d4f4180d8a29250c4627977623def64f5ffec8040a2c70f24f02704 WatchSource:0}: Error finding container dc2dd7414d4f4180d8a29250c4627977623def64f5ffec8040a2c70f24f02704: Status 404 returned error can't find the container with id dc2dd7414d4f4180d8a29250c4627977623def64f5ffec8040a2c70f24f02704
Oct 30 00:54:28.401630 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:28.401603    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1fa555-rqbw9" event={"ID":"e5160630-28d3-4af0-a8e9-d4b08e9f70dd","Type":"ContainerStarted","Data":"dc2dd7414d4f4180d8a29250c4627977623def64f5ffec8040a2c70f24f02704"}
Oct 30 00:54:28.591493 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:28.591452    5002 patch_prober.go:29] interesting pod/e2e-api-b-bdc256-openshift-default-5cf7d97495-76wgr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Oct 30 00:54:28.591926 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:28.591516    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-bdc256-openshift-default-5cf7d97495-76wgr" podUID="bcdd7ce4-a94f-4e6d-aebc-e7ca25d36b4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Oct 30 00:54:29.659960 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:29.652414    5002 patch_prober.go:29] interesting pod/e2e-api-b-bdc256-openshift-default-5cf7d97495-76wgr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Oct 30 00:54:29.659960 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:29.655224    5002 generic.go:356] "Generic (PLEG): container finished" podID="e5160630-28d3-4af0-a8e9-d4b08e9f70dd" containerID="75b34039a14a9c761bf08a16681aa616314578152997eb8a3d36095952eddb15" exitCode=0
Oct 30 00:54:29.659960 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:29.655663    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1fa555-rqbw9" event={"ID":"e5160630-28d3-4af0-a8e9-d4b08e9f70dd","Type":"ContainerDied","Data":"75b34039a14a9c761bf08a16681aa616314578152997eb8a3d36095952eddb15"}
Oct 30 00:54:29.695785 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:29.689206    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-bdc256-openshift-default-5cf7d97495-76wgr" podUID="bcdd7ce4-a94f-4e6d-aebc-e7ca25d36b4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Oct 30 00:54:30.319719 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:30.319691    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-02fbf5-764c8d7f95-qdnr5"
Oct 30 00:54:30.323362 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:30.323340    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-02fbf5-6b759bfc8-7kvrs"
Oct 30 00:54:30.593448 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:30.593354    5002 patch_prober.go:29] interesting pod/e2e-api-b-bdc256-openshift-default-5cf7d97495-76wgr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Oct 30 00:54:30.593619 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:30.593438    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-bdc256-openshift-default-5cf7d97495-76wgr" podUID="bcdd7ce4-a94f-4e6d-aebc-e7ca25d36b4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Oct 30 00:54:30.894458 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:30.894431    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1fa555-rqbw9"
Oct 30 00:54:30.991002 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:30.990959    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5160630-28d3-4af0-a8e9-d4b08e9f70dd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5160630-28d3-4af0-a8e9-d4b08e9f70dd-maas-api-openshift-ca\") pod \"e5160630-28d3-4af0-a8e9-d4b08e9f70dd\" (UID: \"e5160630-28d3-4af0-a8e9-d4b08e9f70dd\") "
Oct 30 00:54:31.004426 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:30.991049    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e5160630-28d3-4af0-a8e9-d4b08e9f70dd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5160630-28d3-4af0-a8e9-d4b08e9f70dd-maas-api-xks-ca\") pod \"e5160630-28d3-4af0-a8e9-d4b08e9f70dd\" (UID: \"e5160630-28d3-4af0-a8e9-d4b08e9f70dd\") "
Oct 30 00:54:31.004426 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:30.991531    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5160630-28d3-4af0-a8e9-d4b08e9f70dd-kube-api-access-glzs4\" (UniqueName: \"kubernetes.io/projected/e5160630-28d3-4af0-a8e9-d4b08e9f70dd-kube-api-access-glzs4\") pod \"e5160630-28d3-4af0-a8e9-d4b08e9f70dd\" (UID: \"e5160630-28d3-4af0-a8e9-d4b08e9f70dd\") "
Oct 30 00:54:31.004426 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:30.997725    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5160630-28d3-4af0-a8e9-d4b08e9f70dd-kube-api-access-glzs4" pod "e5160630-28d3-4af0-a8e9-d4b08e9f70dd" (UID: "e5160630-28d3-4af0-a8e9-d4b08e9f70dd"). InnerVolumeSpecName "kube-api-access-glzs4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:54:31.004426 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:30.998138    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5160630-28d3-4af0-a8e9-d4b08e9f70dd-maas-api-openshift-ca" pod "e5160630-28d3-4af0-a8e9-d4b08e9f70dd" (UID: "e5160630-28d3-4af0-a8e9-d4b08e9f70dd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:54:31.055415 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:31.032023    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5160630-28d3-4af0-a8e9-d4b08e9f70dd-maas-api-xks-ca" pod "e5160630-28d3-4af0-a8e9-d4b08e9f70dd" (UID: "e5160630-28d3-4af0-a8e9-d4b08e9f70dd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:54:31.129575 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:31.129467    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5160630-28d3-4af0-a8e9-d4b08e9f70dd-maas-api-openshift-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:54:31.129575 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:31.129511    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5160630-28d3-4af0-a8e9-d4b08e9f70dd-maas-api-xks-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:54:31.129575 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:31.129527    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-glzs4\" (UniqueName: \"kubernetes.io/projected/e5160630-28d3-4af0-a8e9-d4b08e9f70dd-kube-api-access-glzs4\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:54:31.591676 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:31.591633    5002 patch_prober.go:29] interesting pod/e2e-api-b-bdc256-openshift-default-5cf7d97495-76wgr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Oct 30 00:54:31.591852 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:31.591703    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-bdc256-openshift-default-5cf7d97495-76wgr" podUID="bcdd7ce4-a94f-4e6d-aebc-e7ca25d36b4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Oct 30 00:54:31.690515 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:31.690479    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1fa555-rqbw9"
Oct 30 00:54:31.690694 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:31.690511    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1fa555-rqbw9" event={"ID":"e5160630-28d3-4af0-a8e9-d4b08e9f70dd","Type":"ContainerDied","Data":"dc2dd7414d4f4180d8a29250c4627977623def64f5ffec8040a2c70f24f02704"}
Oct 30 00:54:31.690694 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:31.690573    5002 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dc2dd7414d4f4180d8a29250c4627977623def64f5ffec8040a2c70f24f02704"
Oct 30 00:54:32.596272 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:32.596242    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-bdc256-openshift-default-5cf7d97495-76wgr"
Oct 30 00:54:32.597699 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:32.597675    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-bdc256-openshift-default-5cf7d97495-76wgr"
Oct 30 00:54:36.467101 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:36.467065    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-znxq4"]
Oct 30 00:54:36.467710 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:36.467695    5002 cpu_manager.go:414] "containerMap: removing container" podUID="e5160630-28d3-4af0-a8e9-d4b08e9f70dd" containerName="revoke-keys"
Oct 30 00:54:36.467760 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:36.467711    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5160630-28d3-4af0-a8e9-d4b08e9f70dd" containerName="revoke-keys"
Oct 30 00:54:36.467840 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:36.467831    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5160630-28d3-4af0-a8e9-d4b08e9f70dd" containerName="revoke-keys"
Oct 30 00:54:36.471777 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:36.471756    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-znxq4"
Oct 30 00:54:36.507120 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:36.507085    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-znxq4"]
Oct 30 00:54:36.622704 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:36.622666    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c61db96f-6bcd-462f-9957-c920245070c6-credential-socket\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-znxq4\" (UID: \"c61db96f-6bcd-462f-9957-c920245070c6\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-znxq4"
Oct 30 00:54:36.622704 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:36.622709    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c61db96f-6bcd-462f-9957-c920245070c6-workload-socket\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-znxq4\" (UID: \"c61db96f-6bcd-462f-9957-c920245070c6\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-znxq4"
Oct 30 00:54:36.622917 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:36.622730    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znzzx\" (UniqueName: \"kubernetes.io/projected/c61db96f-6bcd-462f-9957-c920245070c6-kube-api-access-znzzx\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-znxq4\" (UID: \"c61db96f-6bcd-462f-9957-c920245070c6\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-znxq4"
Oct 30 00:54:36.622955 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:36.622910    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c61db96f-6bcd-462f-9957-c920245070c6-istio-token\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-znxq4\" (UID: \"c61db96f-6bcd-462f-9957-c920245070c6\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-znxq4"
Oct 30 00:54:36.622990 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:36.622978    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c61db96f-6bcd-462f-9957-c920245070c6-istiod-ca-cert\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-znxq4\" (UID: \"c61db96f-6bcd-462f-9957-c920245070c6\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-znxq4"
Oct 30 00:54:36.623170 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:36.623146    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c61db96f-6bcd-462f-9957-c920245070c6-istio-data\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-znxq4\" (UID: \"c61db96f-6bcd-462f-9957-c920245070c6\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-znxq4"
Oct 30 00:54:36.623297 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:36.623282    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c61db96f-6bcd-462f-9957-c920245070c6-istio-envoy\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-znxq4\" (UID: \"c61db96f-6bcd-462f-9957-c920245070c6\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-znxq4"
Oct 30 00:54:36.623332 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:36.623309    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c61db96f-6bcd-462f-9957-c920245070c6-istio-podinfo\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-znxq4\" (UID: \"c61db96f-6bcd-462f-9957-c920245070c6\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-znxq4"
Oct 30 00:54:36.623445 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:36.623413    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c61db96f-6bcd-462f-9957-c920245070c6-workload-certs\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-znxq4\" (UID: \"c61db96f-6bcd-462f-9957-c920245070c6\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-znxq4"
Oct 30 00:54:36.725082 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:36.725045    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c61db96f-6bcd-462f-9957-c920245070c6-istio-envoy\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-znxq4\" (UID: \"c61db96f-6bcd-462f-9957-c920245070c6\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-znxq4"
Oct 30 00:54:36.725363 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:36.725335    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c61db96f-6bcd-462f-9957-c920245070c6-istio-podinfo\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-znxq4\" (UID: \"c61db96f-6bcd-462f-9957-c920245070c6\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-znxq4"
Oct 30 00:54:36.725523 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:36.725466    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c61db96f-6bcd-462f-9957-c920245070c6-workload-certs\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-znxq4\" (UID: \"c61db96f-6bcd-462f-9957-c920245070c6\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-znxq4"
Oct 30 00:54:36.725622 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:36.725599    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c61db96f-6bcd-462f-9957-c920245070c6-credential-socket\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-znxq4\" (UID: \"c61db96f-6bcd-462f-9957-c920245070c6\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-znxq4"
Oct 30 00:54:36.725700 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:36.725643    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c61db96f-6bcd-462f-9957-c920245070c6-workload-socket\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-znxq4\" (UID: \"c61db96f-6bcd-462f-9957-c920245070c6\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-znxq4"
Oct 30 00:54:36.725700 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:36.725673    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znzzx\" (UniqueName: \"kubernetes.io/projected/c61db96f-6bcd-462f-9957-c920245070c6-kube-api-access-znzzx\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-znxq4\" (UID: \"c61db96f-6bcd-462f-9957-c920245070c6\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-znxq4"
Oct 30 00:54:36.725818 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:36.725729    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c61db96f-6bcd-462f-9957-c920245070c6-istio-token\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-znxq4\" (UID: \"c61db96f-6bcd-462f-9957-c920245070c6\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-znxq4"
Oct 30 00:54:36.725818 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:36.725785    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c61db96f-6bcd-462f-9957-c920245070c6-istiod-ca-cert\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-znxq4\" (UID: \"c61db96f-6bcd-462f-9957-c920245070c6\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-znxq4"
Oct 30 00:54:36.725931 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:36.725888    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c61db96f-6bcd-462f-9957-c920245070c6-istio-data\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-znxq4\" (UID: \"c61db96f-6bcd-462f-9957-c920245070c6\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-znxq4"
Oct 30 00:54:36.725989 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:36.725933    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c61db96f-6bcd-462f-9957-c920245070c6-workload-certs\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-znxq4\" (UID: \"c61db96f-6bcd-462f-9957-c920245070c6\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-znxq4"
Oct 30 00:54:36.726289 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:36.726262    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c61db96f-6bcd-462f-9957-c920245070c6-istio-data\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-znxq4\" (UID: \"c61db96f-6bcd-462f-9957-c920245070c6\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-znxq4"
Oct 30 00:54:36.726725 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:36.726702    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c61db96f-6bcd-462f-9957-c920245070c6-credential-socket\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-znxq4\" (UID: \"c61db96f-6bcd-462f-9957-c920245070c6\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-znxq4"
Oct 30 00:54:36.727620 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:36.727071    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c61db96f-6bcd-462f-9957-c920245070c6-istiod-ca-cert\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-znxq4\" (UID: \"c61db96f-6bcd-462f-9957-c920245070c6\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-znxq4"
Oct 30 00:54:36.727620 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:36.727461    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c61db96f-6bcd-462f-9957-c920245070c6-workload-socket\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-znxq4\" (UID: \"c61db96f-6bcd-462f-9957-c920245070c6\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-znxq4"
Oct 30 00:54:36.734278 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:36.732737    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c61db96f-6bcd-462f-9957-c920245070c6-istio-podinfo\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-znxq4\" (UID: \"c61db96f-6bcd-462f-9957-c920245070c6\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-znxq4"
Oct 30 00:54:36.740936 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:36.740909    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c61db96f-6bcd-462f-9957-c920245070c6-istio-envoy\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-znxq4\" (UID: \"c61db96f-6bcd-462f-9957-c920245070c6\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-znxq4"
Oct 30 00:54:36.752071 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:36.752038    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-znzzx\" (UniqueName: \"kubernetes.io/projected/c61db96f-6bcd-462f-9957-c920245070c6-kube-api-access-znzzx\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-znxq4\" (UID: \"c61db96f-6bcd-462f-9957-c920245070c6\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-znxq4"
Oct 30 00:54:36.752215 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:36.752164    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c61db96f-6bcd-462f-9957-c920245070c6-istio-token\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-znxq4\" (UID: \"c61db96f-6bcd-462f-9957-c920245070c6\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-znxq4"
Oct 30 00:54:36.789491 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:36.789455    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-znxq4"
Oct 30 00:54:37.119462 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:54:37.119422    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc61db96f_6bcd_462f_9957_c920245070c6.slice/crio-7c54953bebbd351c55001ab37df75edfc59956db809e924cc101700c6e238027 WatchSource:0}: Error finding container 7c54953bebbd351c55001ab37df75edfc59956db809e924cc101700c6e238027: Status 404 returned error can't find the container with id 7c54953bebbd351c55001ab37df75edfc59956db809e924cc101700c6e238027
Oct 30 00:54:37.120932 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:37.120907    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-znxq4"]
Oct 30 00:54:37.122578 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:37.122545    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 00:54:37.122691 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:37.122623    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 00:54:37.122691 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:37.122671    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 00:54:37.347963 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:37.347918    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-24zch"]
Oct 30 00:54:37.348160 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:37.348127    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-24zch" podUID="35e8d90c-3415-41a3-8b4f-03a3c738cc99" containerName="istio-proxy" containerID="cri-o://6589a90942f3e31764a49a2d424e3f285e5224c7ba7c9a4a33e1f71345b0622e" gracePeriod=30
Oct 30 00:54:37.752676 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:37.752633    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-znxq4" event={"ID":"c61db96f-6bcd-462f-9957-c920245070c6","Type":"ContainerStarted","Data":"6452327d99acd74fd2b200c6d4939407a52f0ef83d35586c5e35bfe98114dbe8"}
Oct 30 00:54:37.753784 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:37.753242    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-znxq4" event={"ID":"c61db96f-6bcd-462f-9957-c920245070c6","Type":"ContainerStarted","Data":"7c54953bebbd351c55001ab37df75edfc59956db809e924cc101700c6e238027"}
Oct 30 00:54:37.790710 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:37.790641    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-znxq4"
Oct 30 00:54:37.814062 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:37.813989    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-znxq4" podStartSLOduration=1.813969932 podStartE2EDuration="1.813969932s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:54:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:54:37.808162256 +0000 UTC m=+1373.424595135" watchObservedRunningTime="2026-10-30 00:54:37.813969932 +0000 UTC m=+1373.430402812"
Oct 30 00:54:38.925325 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:38.925281    5002 patch_prober.go:29] interesting pod/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-znxq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body=
Oct 30 00:54:38.925883 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:38.925347    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-znxq4" podUID="c61db96f-6bcd-462f-9957-c920245070c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused"
Oct 30 00:54:39.790842 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:39.790796    5002 patch_prober.go:29] interesting pod/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-znxq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body=
Oct 30 00:54:39.791033 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:39.790848    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-znxq4" podUID="c61db96f-6bcd-462f-9957-c920245070c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused"
Oct 30 00:54:40.538500 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:40.537934    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-6444d7-openshift-default-7bf76bdcd9-6nnq2"]
Oct 30 00:54:40.542928 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:40.542902    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6444d7-openshift-default-7bf76bdcd9-6nnq2"
Oct 30 00:54:40.580505 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:40.580458    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-6444d7-openshift-default-dockercfg-84h8h\""
Oct 30 00:54:40.601763 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:40.601734    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-6444d7-openshift-default-7bf76bdcd9-6nnq2"]
Oct 30 00:54:40.632978 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:40.632937    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/750a98de-3674-4c5b-835c-48b928841cd7-workload-certs\") pod \"e2e-inf-a-6444d7-openshift-default-7bf76bdcd9-6nnq2\" (UID: \"750a98de-3674-4c5b-835c-48b928841cd7\") " pod="openshift-ingress/e2e-inf-a-6444d7-openshift-default-7bf76bdcd9-6nnq2"
Oct 30 00:54:40.633173 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:40.632990    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/750a98de-3674-4c5b-835c-48b928841cd7-istio-token\") pod \"e2e-inf-a-6444d7-openshift-default-7bf76bdcd9-6nnq2\" (UID: \"750a98de-3674-4c5b-835c-48b928841cd7\") " pod="openshift-ingress/e2e-inf-a-6444d7-openshift-default-7bf76bdcd9-6nnq2"
Oct 30 00:54:40.633173 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:40.633045    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/750a98de-3674-4c5b-835c-48b928841cd7-credential-socket\") pod \"e2e-inf-a-6444d7-openshift-default-7bf76bdcd9-6nnq2\" (UID: \"750a98de-3674-4c5b-835c-48b928841cd7\") " pod="openshift-ingress/e2e-inf-a-6444d7-openshift-default-7bf76bdcd9-6nnq2"
Oct 30 00:54:40.633173 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:40.633083    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/750a98de-3674-4c5b-835c-48b928841cd7-istiod-ca-cert\") pod \"e2e-inf-a-6444d7-openshift-default-7bf76bdcd9-6nnq2\" (UID: \"750a98de-3674-4c5b-835c-48b928841cd7\") " pod="openshift-ingress/e2e-inf-a-6444d7-openshift-default-7bf76bdcd9-6nnq2"
Oct 30 00:54:40.633173 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:40.633155    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/750a98de-3674-4c5b-835c-48b928841cd7-istio-podinfo\") pod \"e2e-inf-a-6444d7-openshift-default-7bf76bdcd9-6nnq2\" (UID: \"750a98de-3674-4c5b-835c-48b928841cd7\") " pod="openshift-ingress/e2e-inf-a-6444d7-openshift-default-7bf76bdcd9-6nnq2"
Oct 30 00:54:40.633433 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:40.633187    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/750a98de-3674-4c5b-835c-48b928841cd7-istio-data\") pod \"e2e-inf-a-6444d7-openshift-default-7bf76bdcd9-6nnq2\" (UID: \"750a98de-3674-4c5b-835c-48b928841cd7\") " pod="openshift-ingress/e2e-inf-a-6444d7-openshift-default-7bf76bdcd9-6nnq2"
Oct 30 00:54:40.633433 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:40.633283    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/750a98de-3674-4c5b-835c-48b928841cd7-workload-socket\") pod \"e2e-inf-a-6444d7-openshift-default-7bf76bdcd9-6nnq2\" (UID: \"750a98de-3674-4c5b-835c-48b928841cd7\") " pod="openshift-ingress/e2e-inf-a-6444d7-openshift-default-7bf76bdcd9-6nnq2"
Oct 30 00:54:40.633433 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:40.633317    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/750a98de-3674-4c5b-835c-48b928841cd7-istio-envoy\") pod \"e2e-inf-a-6444d7-openshift-default-7bf76bdcd9-6nnq2\" (UID: \"750a98de-3674-4c5b-835c-48b928841cd7\") " pod="openshift-ingress/e2e-inf-a-6444d7-openshift-default-7bf76bdcd9-6nnq2"
Oct 30 00:54:40.633433 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:40.633365    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cz6qq\" (UniqueName: \"kubernetes.io/projected/750a98de-3674-4c5b-835c-48b928841cd7-kube-api-access-cz6qq\") pod \"e2e-inf-a-6444d7-openshift-default-7bf76bdcd9-6nnq2\" (UID: \"750a98de-3674-4c5b-835c-48b928841cd7\") " pod="openshift-ingress/e2e-inf-a-6444d7-openshift-default-7bf76bdcd9-6nnq2"
Oct 30 00:54:40.742405 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:40.741441    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/750a98de-3674-4c5b-835c-48b928841cd7-workload-certs\") pod \"e2e-inf-a-6444d7-openshift-default-7bf76bdcd9-6nnq2\" (UID: \"750a98de-3674-4c5b-835c-48b928841cd7\") " pod="openshift-ingress/e2e-inf-a-6444d7-openshift-default-7bf76bdcd9-6nnq2"
Oct 30 00:54:40.742405 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:40.741504    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/750a98de-3674-4c5b-835c-48b928841cd7-istio-token\") pod \"e2e-inf-a-6444d7-openshift-default-7bf76bdcd9-6nnq2\" (UID: \"750a98de-3674-4c5b-835c-48b928841cd7\") " pod="openshift-ingress/e2e-inf-a-6444d7-openshift-default-7bf76bdcd9-6nnq2"
Oct 30 00:54:40.742405 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:40.741564    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/750a98de-3674-4c5b-835c-48b928841cd7-credential-socket\") pod \"e2e-inf-a-6444d7-openshift-default-7bf76bdcd9-6nnq2\" (UID: \"750a98de-3674-4c5b-835c-48b928841cd7\") " pod="openshift-ingress/e2e-inf-a-6444d7-openshift-default-7bf76bdcd9-6nnq2"
Oct 30 00:54:40.742405 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:40.741606    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/750a98de-3674-4c5b-835c-48b928841cd7-istiod-ca-cert\") pod \"e2e-inf-a-6444d7-openshift-default-7bf76bdcd9-6nnq2\" (UID: \"750a98de-3674-4c5b-835c-48b928841cd7\") " pod="openshift-ingress/e2e-inf-a-6444d7-openshift-default-7bf76bdcd9-6nnq2"
Oct 30 00:54:40.742405 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:40.741687    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/750a98de-3674-4c5b-835c-48b928841cd7-istio-podinfo\") pod \"e2e-inf-a-6444d7-openshift-default-7bf76bdcd9-6nnq2\" (UID: \"750a98de-3674-4c5b-835c-48b928841cd7\") " pod="openshift-ingress/e2e-inf-a-6444d7-openshift-default-7bf76bdcd9-6nnq2"
Oct 30 00:54:40.742405 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:40.741725    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/750a98de-3674-4c5b-835c-48b928841cd7-istio-data\") pod \"e2e-inf-a-6444d7-openshift-default-7bf76bdcd9-6nnq2\" (UID: \"750a98de-3674-4c5b-835c-48b928841cd7\") " pod="openshift-ingress/e2e-inf-a-6444d7-openshift-default-7bf76bdcd9-6nnq2"
Oct 30 00:54:40.742405 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:40.741864    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/750a98de-3674-4c5b-835c-48b928841cd7-workload-socket\") pod \"e2e-inf-a-6444d7-openshift-default-7bf76bdcd9-6nnq2\" (UID: \"750a98de-3674-4c5b-835c-48b928841cd7\") " pod="openshift-ingress/e2e-inf-a-6444d7-openshift-default-7bf76bdcd9-6nnq2"
Oct 30 00:54:40.742405 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:40.741903    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/750a98de-3674-4c5b-835c-48b928841cd7-istio-envoy\") pod \"e2e-inf-a-6444d7-openshift-default-7bf76bdcd9-6nnq2\" (UID: \"750a98de-3674-4c5b-835c-48b928841cd7\") " pod="openshift-ingress/e2e-inf-a-6444d7-openshift-default-7bf76bdcd9-6nnq2"
Oct 30 00:54:40.742405 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:40.741965    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cz6qq\" (UniqueName: \"kubernetes.io/projected/750a98de-3674-4c5b-835c-48b928841cd7-kube-api-access-cz6qq\") pod \"e2e-inf-a-6444d7-openshift-default-7bf76bdcd9-6nnq2\" (UID: \"750a98de-3674-4c5b-835c-48b928841cd7\") " pod="openshift-ingress/e2e-inf-a-6444d7-openshift-default-7bf76bdcd9-6nnq2"
Oct 30 00:54:40.743259 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:40.743232    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/750a98de-3674-4c5b-835c-48b928841cd7-istiod-ca-cert\") pod \"e2e-inf-a-6444d7-openshift-default-7bf76bdcd9-6nnq2\" (UID: \"750a98de-3674-4c5b-835c-48b928841cd7\") " pod="openshift-ingress/e2e-inf-a-6444d7-openshift-default-7bf76bdcd9-6nnq2"
Oct 30 00:54:40.807472 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:40.743812    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/750a98de-3674-4c5b-835c-48b928841cd7-workload-socket\") pod \"e2e-inf-a-6444d7-openshift-default-7bf76bdcd9-6nnq2\" (UID: \"750a98de-3674-4c5b-835c-48b928841cd7\") " pod="openshift-ingress/e2e-inf-a-6444d7-openshift-default-7bf76bdcd9-6nnq2"
Oct 30 00:54:40.807472 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:40.744060    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/750a98de-3674-4c5b-835c-48b928841cd7-istio-data\") pod \"e2e-inf-a-6444d7-openshift-default-7bf76bdcd9-6nnq2\" (UID: \"750a98de-3674-4c5b-835c-48b928841cd7\") " pod="openshift-ingress/e2e-inf-a-6444d7-openshift-default-7bf76bdcd9-6nnq2"
Oct 30 00:54:40.807472 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:40.751121    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/750a98de-3674-4c5b-835c-48b928841cd7-workload-certs\") pod \"e2e-inf-a-6444d7-openshift-default-7bf76bdcd9-6nnq2\" (UID: \"750a98de-3674-4c5b-835c-48b928841cd7\") " pod="openshift-ingress/e2e-inf-a-6444d7-openshift-default-7bf76bdcd9-6nnq2"
Oct 30 00:54:40.807472 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:40.751397    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/750a98de-3674-4c5b-835c-48b928841cd7-credential-socket\") pod \"e2e-inf-a-6444d7-openshift-default-7bf76bdcd9-6nnq2\" (UID: \"750a98de-3674-4c5b-835c-48b928841cd7\") " pod="openshift-ingress/e2e-inf-a-6444d7-openshift-default-7bf76bdcd9-6nnq2"
Oct 30 00:54:40.807472 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:40.775040    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/750a98de-3674-4c5b-835c-48b928841cd7-istio-podinfo\") pod \"e2e-inf-a-6444d7-openshift-default-7bf76bdcd9-6nnq2\" (UID: \"750a98de-3674-4c5b-835c-48b928841cd7\") " pod="openshift-ingress/e2e-inf-a-6444d7-openshift-default-7bf76bdcd9-6nnq2"
Oct 30 00:54:40.807472 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:40.792309    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/750a98de-3674-4c5b-835c-48b928841cd7-istio-envoy\") pod \"e2e-inf-a-6444d7-openshift-default-7bf76bdcd9-6nnq2\" (UID: \"750a98de-3674-4c5b-835c-48b928841cd7\") " pod="openshift-ingress/e2e-inf-a-6444d7-openshift-default-7bf76bdcd9-6nnq2"
Oct 30 00:54:40.807472 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:40.803676    5002 patch_prober.go:29] interesting pod/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-znxq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body=
Oct 30 00:54:40.807472 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:40.803726    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-znxq4" podUID="c61db96f-6bcd-462f-9957-c920245070c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused"
Oct 30 00:54:40.807472 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:40.807068    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/750a98de-3674-4c5b-835c-48b928841cd7-istio-token\") pod \"e2e-inf-a-6444d7-openshift-default-7bf76bdcd9-6nnq2\" (UID: \"750a98de-3674-4c5b-835c-48b928841cd7\") " pod="openshift-ingress/e2e-inf-a-6444d7-openshift-default-7bf76bdcd9-6nnq2"
Oct 30 00:54:40.822953 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:40.822920    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cz6qq\" (UniqueName: \"kubernetes.io/projected/750a98de-3674-4c5b-835c-48b928841cd7-kube-api-access-cz6qq\") pod \"e2e-inf-a-6444d7-openshift-default-7bf76bdcd9-6nnq2\" (UID: \"750a98de-3674-4c5b-835c-48b928841cd7\") " pod="openshift-ingress/e2e-inf-a-6444d7-openshift-default-7bf76bdcd9-6nnq2"
Oct 30 00:54:40.884919 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:40.884885    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6444d7-openshift-default-7bf76bdcd9-6nnq2"
Oct 30 00:54:41.471417 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:54:41.450321    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod750a98de_3674_4c5b_835c_48b928841cd7.slice/crio-0f9701dab16f9cb7495f70e43cef32036dfabda278ca66981c1040d7db2df347 WatchSource:0}: Error finding container 0f9701dab16f9cb7495f70e43cef32036dfabda278ca66981c1040d7db2df347: Status 404 returned error can't find the container with id 0f9701dab16f9cb7495f70e43cef32036dfabda278ca66981c1040d7db2df347
Oct 30 00:54:41.471417 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:41.451481    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-6444d7-openshift-default-7bf76bdcd9-6nnq2"]
Oct 30 00:54:41.471417 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:41.455152    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 00:54:41.471417 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:41.455246    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 00:54:41.471417 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:41.455290    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 00:54:41.802489 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:41.796798    5002 patch_prober.go:29] interesting pod/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-znxq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body=
Oct 30 00:54:41.802489 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:41.796859    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-znxq4" podUID="c61db96f-6bcd-462f-9957-c920245070c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused"
Oct 30 00:54:42.045466 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:42.045411    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6444d7-openshift-default-7bf76bdcd9-6nnq2" event={"ID":"750a98de-3674-4c5b-835c-48b928841cd7","Type":"ContainerStarted","Data":"b9052f3a47331b79001912f9b046332d2fc3bf08b67fde883354b6e30c262916"}
Oct 30 00:54:42.045780 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:42.045751    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6444d7-openshift-default-7bf76bdcd9-6nnq2" event={"ID":"750a98de-3674-4c5b-835c-48b928841cd7","Type":"ContainerStarted","Data":"0f9701dab16f9cb7495f70e43cef32036dfabda278ca66981c1040d7db2df347"}
Oct 30 00:54:42.106613 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:42.106498    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-6444d7-openshift-default-7bf76bdcd9-6nnq2" podStartSLOduration=2.106481603 podStartE2EDuration="2.106481603s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:54:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:54:42.105684468 +0000 UTC m=+1377.722117373" watchObservedRunningTime="2026-10-30 00:54:42.106481603 +0000 UTC m=+1377.722914483"
Oct 30 00:54:42.791204 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:42.790761    5002 patch_prober.go:29] interesting pod/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-znxq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body=
Oct 30 00:54:42.791204 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:42.790827    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-znxq4" podUID="c61db96f-6bcd-462f-9957-c920245070c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused"
Oct 30 00:54:42.892255 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:42.892176    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-6444d7-openshift-default-7bf76bdcd9-6nnq2"
Oct 30 00:54:42.902527 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:42.902494    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-6444d7-openshift-default-7bf76bdcd9-6nnq2"
Oct 30 00:54:42.903339 ip-10-0-128-7 kubenswrapper[5002]: E1030 00:54:42.902958    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbjdq" containerName="extractor"
Oct 30 00:54:43.186404 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:43.185858    5002 generic.go:356] "Generic (PLEG): container finished" podID="35e8d90c-3415-41a3-8b4f-03a3c738cc99" containerID="6589a90942f3e31764a49a2d424e3f285e5224c7ba7c9a4a33e1f71345b0622e" exitCode=0
Oct 30 00:54:43.203786 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:43.188001    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-24zch" event={"ID":"35e8d90c-3415-41a3-8b4f-03a3c738cc99","Type":"ContainerDied","Data":"6589a90942f3e31764a49a2d424e3f285e5224c7ba7c9a4a33e1f71345b0622e"}
Oct 30 00:54:43.203786 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:43.188087    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-6444d7-openshift-default-7bf76bdcd9-6nnq2"
Oct 30 00:54:43.219725 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:43.218958    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-6444d7-openshift-default-7bf76bdcd9-6nnq2"
Oct 30 00:54:43.431983 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:43.431947    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-24zch"
Oct 30 00:54:43.529219 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:43.529188    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/35e8d90c-3415-41a3-8b4f-03a3c738cc99-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35e8d90c-3415-41a3-8b4f-03a3c738cc99-istio-podinfo\") pod \"35e8d90c-3415-41a3-8b4f-03a3c738cc99\" (UID: \"35e8d90c-3415-41a3-8b4f-03a3c738cc99\") "
Oct 30 00:54:43.529413 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:43.529255    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35e8d90c-3415-41a3-8b4f-03a3c738cc99-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35e8d90c-3415-41a3-8b4f-03a3c738cc99-workload-socket\") pod \"35e8d90c-3415-41a3-8b4f-03a3c738cc99\" (UID: \"35e8d90c-3415-41a3-8b4f-03a3c738cc99\") "
Oct 30 00:54:43.529413 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:43.529328    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35e8d90c-3415-41a3-8b4f-03a3c738cc99-istio-token\" (UniqueName: \"kubernetes.io/projected/35e8d90c-3415-41a3-8b4f-03a3c738cc99-istio-token\") pod \"35e8d90c-3415-41a3-8b4f-03a3c738cc99\" (UID: \"35e8d90c-3415-41a3-8b4f-03a3c738cc99\") "
Oct 30 00:54:43.529413 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:43.529353    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35e8d90c-3415-41a3-8b4f-03a3c738cc99-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35e8d90c-3415-41a3-8b4f-03a3c738cc99-workload-certs\") pod \"35e8d90c-3415-41a3-8b4f-03a3c738cc99\" (UID: \"35e8d90c-3415-41a3-8b4f-03a3c738cc99\") "
Oct 30 00:54:43.529413 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:43.529407    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35e8d90c-3415-41a3-8b4f-03a3c738cc99-kube-api-access-xz6rj\" (UniqueName: \"kubernetes.io/projected/35e8d90c-3415-41a3-8b4f-03a3c738cc99-kube-api-access-xz6rj\") pod \"35e8d90c-3415-41a3-8b4f-03a3c738cc99\" (UID: \"35e8d90c-3415-41a3-8b4f-03a3c738cc99\") "
Oct 30 00:54:43.529595 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:43.529471    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35e8d90c-3415-41a3-8b4f-03a3c738cc99-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35e8d90c-3415-41a3-8b4f-03a3c738cc99-credential-socket\") pod \"35e8d90c-3415-41a3-8b4f-03a3c738cc99\" (UID: \"35e8d90c-3415-41a3-8b4f-03a3c738cc99\") "
Oct 30 00:54:43.529595 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:43.529525    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35e8d90c-3415-41a3-8b4f-03a3c738cc99-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35e8d90c-3415-41a3-8b4f-03a3c738cc99-istiod-ca-cert\") pod \"35e8d90c-3415-41a3-8b4f-03a3c738cc99\" (UID: \"35e8d90c-3415-41a3-8b4f-03a3c738cc99\") "
Oct 30 00:54:43.529595 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:43.529576    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35e8d90c-3415-41a3-8b4f-03a3c738cc99-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35e8d90c-3415-41a3-8b4f-03a3c738cc99-istio-data\") pod \"35e8d90c-3415-41a3-8b4f-03a3c738cc99\" (UID: \"35e8d90c-3415-41a3-8b4f-03a3c738cc99\") "
Oct 30 00:54:43.529687 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:43.529598    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35e8d90c-3415-41a3-8b4f-03a3c738cc99-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35e8d90c-3415-41a3-8b4f-03a3c738cc99-istio-envoy\") pod \"35e8d90c-3415-41a3-8b4f-03a3c738cc99\" (UID: \"35e8d90c-3415-41a3-8b4f-03a3c738cc99\") "
Oct 30 00:54:43.529815 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:43.529781    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35e8d90c-3415-41a3-8b4f-03a3c738cc99-workload-certs" pod "35e8d90c-3415-41a3-8b4f-03a3c738cc99" (UID: "35e8d90c-3415-41a3-8b4f-03a3c738cc99"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:54:43.530247 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:43.530223    5002 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35e8d90c-3415-41a3-8b4f-03a3c738cc99-workload-certs\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:54:43.530507 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:43.530484    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35e8d90c-3415-41a3-8b4f-03a3c738cc99-workload-socket" pod "35e8d90c-3415-41a3-8b4f-03a3c738cc99" (UID: "35e8d90c-3415-41a3-8b4f-03a3c738cc99"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:54:43.530750 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:43.530726    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35e8d90c-3415-41a3-8b4f-03a3c738cc99-istiod-ca-cert" pod "35e8d90c-3415-41a3-8b4f-03a3c738cc99" (UID: "35e8d90c-3415-41a3-8b4f-03a3c738cc99"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:54:43.531313 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:43.531287    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35e8d90c-3415-41a3-8b4f-03a3c738cc99-istio-data" pod "35e8d90c-3415-41a3-8b4f-03a3c738cc99" (UID: "35e8d90c-3415-41a3-8b4f-03a3c738cc99"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:54:43.531431 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:43.531359    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35e8d90c-3415-41a3-8b4f-03a3c738cc99-credential-socket" pod "35e8d90c-3415-41a3-8b4f-03a3c738cc99" (UID: "35e8d90c-3415-41a3-8b4f-03a3c738cc99"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:54:43.533206 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:43.533177    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35e8d90c-3415-41a3-8b4f-03a3c738cc99-istio-envoy" pod "35e8d90c-3415-41a3-8b4f-03a3c738cc99" (UID: "35e8d90c-3415-41a3-8b4f-03a3c738cc99"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:54:43.533304 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:43.533171    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/35e8d90c-3415-41a3-8b4f-03a3c738cc99-istio-podinfo" pod "35e8d90c-3415-41a3-8b4f-03a3c738cc99" (UID: "35e8d90c-3415-41a3-8b4f-03a3c738cc99"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 00:54:43.534080 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:43.534052    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35e8d90c-3415-41a3-8b4f-03a3c738cc99-istio-token" pod "35e8d90c-3415-41a3-8b4f-03a3c738cc99" (UID: "35e8d90c-3415-41a3-8b4f-03a3c738cc99"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:54:43.534168 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:43.534095    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35e8d90c-3415-41a3-8b4f-03a3c738cc99-kube-api-access-xz6rj" pod "35e8d90c-3415-41a3-8b4f-03a3c738cc99" (UID: "35e8d90c-3415-41a3-8b4f-03a3c738cc99"). InnerVolumeSpecName "kube-api-access-xz6rj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:54:43.631588 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:43.631496    5002 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35e8d90c-3415-41a3-8b4f-03a3c738cc99-istio-token\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:54:43.631588 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:43.631527    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xz6rj\" (UniqueName: \"kubernetes.io/projected/35e8d90c-3415-41a3-8b4f-03a3c738cc99-kube-api-access-xz6rj\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:54:43.631588 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:43.631541    5002 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35e8d90c-3415-41a3-8b4f-03a3c738cc99-credential-socket\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:54:43.631588 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:43.631554    5002 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35e8d90c-3415-41a3-8b4f-03a3c738cc99-istiod-ca-cert\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:54:43.631588 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:43.631565    5002 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35e8d90c-3415-41a3-8b4f-03a3c738cc99-istio-data\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:54:43.631588 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:43.631576    5002 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35e8d90c-3415-41a3-8b4f-03a3c738cc99-istio-envoy\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:54:43.631588 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:43.631586    5002 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35e8d90c-3415-41a3-8b4f-03a3c738cc99-istio-podinfo\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:54:43.631910 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:43.631599    5002 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35e8d90c-3415-41a3-8b4f-03a3c738cc99-workload-socket\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:54:43.790517 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:43.790468    5002 patch_prober.go:29] interesting pod/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-znxq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body=
Oct 30 00:54:43.790730 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:43.790535    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-znxq4" podUID="c61db96f-6bcd-462f-9957-c920245070c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused"
Oct 30 00:54:44.196988 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:44.196952    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-24zch"
Oct 30 00:54:44.197526 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:44.196943    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-24zch" event={"ID":"35e8d90c-3415-41a3-8b4f-03a3c738cc99","Type":"ContainerDied","Data":"f62c7b40e212388685ad3b849283d69d825c1c47e2d33166df0b7e05d4f043b1"}
Oct 30 00:54:44.197526 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:44.197181    5002 scope.go:122] "RemoveContainer" containerID="6589a90942f3e31764a49a2d424e3f285e5224c7ba7c9a4a33e1f71345b0622e"
Oct 30 00:54:44.260267 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:44.260223    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-24zch"]
Oct 30 00:54:44.272363 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:44.272324    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-24zch"]
Oct 30 00:54:44.503493 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:44.502984    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35e8d90c-3415-41a3-8b4f-03a3c738cc99" path="/var/lib/kubelet/pods/35e8d90c-3415-41a3-8b4f-03a3c738cc99/volumes"
Oct 30 00:54:44.797942 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:44.797862    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-znxq4"
Oct 30 00:54:44.798425 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:44.798404    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-znxq4"
Oct 30 00:54:44.799189 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:44.799166    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-znxq4"
Oct 30 00:54:46.238722 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:46.238690    5002 scope.go:122] "RemoveContainer" containerID="830eba8dee9e1a4663aa2b46c5160b2a979fbf4b7e953ee008cd061682b7080d"
Oct 30 00:54:46.248967 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:46.248945    5002 scope.go:122] "RemoveContainer" containerID="d5f162f015609b4bd5ea36e6e2b021b373b068a5b8583b3c405da39041cb6ea2"
Oct 30 00:54:46.261209 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:46.261186    5002 scope.go:122] "RemoveContainer" containerID="b845670c42362ad9fce39f5ea737b4b1a0a3ccc248c5696f4211b44100feff89"
Oct 30 00:54:46.273108 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:46.273083    5002 scope.go:122] "RemoveContainer" containerID="75c04f95617817d91a0ca70e23eac377d3467d5d890ea544d89e63b4e05807a6"
Oct 30 00:54:46.287470 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:46.287449    5002 scope.go:122] "RemoveContainer" containerID="042df1b81a33a7400feaae8b97e2c4fda608860736e198543d7dd492020384e2"
Oct 30 00:54:46.301355 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:46.300555    5002 scope.go:122] "RemoveContainer" containerID="b49d2e7601a7b6e1316fba4311502facbca84ede0e3d7b1d333e648f1044dd46"
Oct 30 00:54:54.745979 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:54:54.745942    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7d8z4"]
Oct 30 00:55:06.679880 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:55:06.679835    5002 generic.go:356] "Generic (PLEG): container finished" podID="58608e31-3cf2-4a5c-9d19-c8c72b774f51" containerID="0cf4e79fe5ba7762ac6dae465a00bf85949ae7159a9c4ad0b2992c31d13e23aa" exitCode=0
Oct 30 00:55:06.680460 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:55:06.679903    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-02fbf5-openshift-default-64fc59d485-jhdtc" event={"ID":"58608e31-3cf2-4a5c-9d19-c8c72b774f51","Type":"ContainerDied","Data":"0cf4e79fe5ba7762ac6dae465a00bf85949ae7159a9c4ad0b2992c31d13e23aa"}
Oct 30 00:55:06.680535 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:55:06.680461    5002 scope.go:122] "RemoveContainer" containerID="0cf4e79fe5ba7762ac6dae465a00bf85949ae7159a9c4ad0b2992c31d13e23aa"
Oct 30 00:55:06.682410 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:55:06.682351    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 00:55:06.682509 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:55:06.682439    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 00:55:06.682509 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:55:06.682475    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 00:55:07.697104 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:55:07.697060    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-02fbf5-openshift-default-64fc59d485-jhdtc" event={"ID":"58608e31-3cf2-4a5c-9d19-c8c72b774f51","Type":"ContainerStarted","Data":"2c078af2f907835699d23134075d192736e875a49f894125f413bb89a83d2870"}
Oct 30 00:55:07.995810 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:55:07.995714    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-02fbf5-openshift-default-64fc59d485-jhdtc"
Oct 30 00:55:07.997264 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:55:07.997231    5002 patch_prober.go:29] interesting pod/e2e-api-b-02fbf5-openshift-default-64fc59d485-jhdtc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body=
Oct 30 00:55:07.997373 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:55:07.997332    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-02fbf5-openshift-default-64fc59d485-jhdtc" podUID="58608e31-3cf2-4a5c-9d19-c8c72b774f51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused"
Oct 30 00:55:08.889189 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:55:08.889156    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-02fbf5-openshift-default-64fc59d485-jhdtc"
Oct 30 00:55:08.996105 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:55:08.996061    5002 patch_prober.go:29] interesting pod/e2e-api-b-02fbf5-openshift-default-64fc59d485-jhdtc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body=
Oct 30 00:55:08.996283 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:55:08.996136    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-02fbf5-openshift-default-64fc59d485-jhdtc" podUID="58608e31-3cf2-4a5c-9d19-c8c72b774f51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused"
Oct 30 00:55:09.995573 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:55:09.995530    5002 patch_prober.go:29] interesting pod/e2e-api-b-02fbf5-openshift-default-64fc59d485-jhdtc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body=
Oct 30 00:55:09.995997 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:55:09.995595    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-02fbf5-openshift-default-64fc59d485-jhdtc" podUID="58608e31-3cf2-4a5c-9d19-c8c72b774f51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused"
Oct 30 00:55:10.999904 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:55:10.999860    5002 patch_prober.go:29] interesting pod/e2e-api-b-02fbf5-openshift-default-64fc59d485-jhdtc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body=
Oct 30 00:55:11.000900 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:55:11.000505    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-02fbf5-openshift-default-64fc59d485-jhdtc" podUID="58608e31-3cf2-4a5c-9d19-c8c72b774f51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused"
Oct 30 00:55:11.996275 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:55:11.996228    5002 patch_prober.go:29] interesting pod/e2e-api-b-02fbf5-openshift-default-64fc59d485-jhdtc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body=
Oct 30 00:55:11.996528 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:55:11.996291    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-02fbf5-openshift-default-64fc59d485-jhdtc" podUID="58608e31-3cf2-4a5c-9d19-c8c72b774f51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused"
Oct 30 00:55:12.898258 ip-10-0-128-7 kubenswrapper[5002]: E1030 00:55:12.898209    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbjdq" containerName="extractor"
Oct 30 00:55:12.996401 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:55:12.995685    5002 patch_prober.go:29] interesting pod/e2e-api-b-02fbf5-openshift-default-64fc59d485-jhdtc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body=
Oct 30 00:55:12.996401 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:55:12.995738    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-02fbf5-openshift-default-64fc59d485-jhdtc" podUID="58608e31-3cf2-4a5c-9d19-c8c72b774f51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused"
Oct 30 00:55:14.017203 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:55:14.016968    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-02fbf5-openshift-default-64fc59d485-jhdtc"
Oct 30 00:55:14.018567 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:55:14.018536    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-02fbf5-openshift-default-64fc59d485-jhdtc"
Oct 30 00:55:17.549984 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:55:17.549950    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1fa555-rqbw9"]
Oct 30 00:55:17.563704 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:55:17.563664    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1fa555-rqbw9"]
Oct 30 00:55:17.626258 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:55:17.626219    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d788ba-5c7d7"]
Oct 30 00:55:17.627547 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:55:17.627515    5002 cpu_manager.go:414] "containerMap: removing container" podUID="35e8d90c-3415-41a3-8b4f-03a3c738cc99" containerName="istio-proxy"
Oct 30 00:55:17.627728 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:55:17.627713    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35e8d90c-3415-41a3-8b4f-03a3c738cc99" containerName="istio-proxy"
Oct 30 00:55:17.628019 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:55:17.628001    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="35e8d90c-3415-41a3-8b4f-03a3c738cc99" containerName="istio-proxy"
Oct 30 00:55:17.637804 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:55:17.637778    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d788ba-5c7d7"
Oct 30 00:55:17.649114 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:55:17.648428    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d788ba-5c7d7"]
Oct 30 00:55:17.655906 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:55:17.655878    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 00:55:17.656829 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:55:17.656809    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 00:55:17.657085 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:55:17.657069    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5phsh\""
Oct 30 00:55:17.657246 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:55:17.657233    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 00:55:17.699042 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:55:17.699000    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb965d3f-ff97-4e79-a803-adf9e2022788-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-d788ba-5c7d7\" (UID: \"fb965d3f-ff97-4e79-a803-adf9e2022788\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d788ba-5c7d7"
Oct 30 00:55:17.699331 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:55:17.699118    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4sjl\" (UniqueName: \"kubernetes.io/projected/fb965d3f-ff97-4e79-a803-adf9e2022788-kube-api-access-p4sjl\") pod \"maas-api-revoke-keys-e2e-api-a-d788ba-5c7d7\" (UID: \"fb965d3f-ff97-4e79-a803-adf9e2022788\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d788ba-5c7d7"
Oct 30 00:55:17.699331 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:55:17.699197    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb965d3f-ff97-4e79-a803-adf9e2022788-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-d788ba-5c7d7\" (UID: \"fb965d3f-ff97-4e79-a803-adf9e2022788\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d788ba-5c7d7"
Oct 30 00:55:17.800412 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:55:17.800319    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb965d3f-ff97-4e79-a803-adf9e2022788-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-d788ba-5c7d7\" (UID: \"fb965d3f-ff97-4e79-a803-adf9e2022788\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d788ba-5c7d7"
Oct 30 00:55:17.800557 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:55:17.800519    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb965d3f-ff97-4e79-a803-adf9e2022788-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-d788ba-5c7d7\" (UID: \"fb965d3f-ff97-4e79-a803-adf9e2022788\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d788ba-5c7d7"
Oct 30 00:55:17.800606 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:55:17.800576    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4sjl\" (UniqueName: \"kubernetes.io/projected/fb965d3f-ff97-4e79-a803-adf9e2022788-kube-api-access-p4sjl\") pod \"maas-api-revoke-keys-e2e-api-a-d788ba-5c7d7\" (UID: \"fb965d3f-ff97-4e79-a803-adf9e2022788\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d788ba-5c7d7"
Oct 30 00:55:17.801183 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:55:17.801158    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb965d3f-ff97-4e79-a803-adf9e2022788-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-d788ba-5c7d7\" (UID: \"fb965d3f-ff97-4e79-a803-adf9e2022788\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d788ba-5c7d7"
Oct 30 00:55:17.802786 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:55:17.802760    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb965d3f-ff97-4e79-a803-adf9e2022788-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-d788ba-5c7d7\" (UID: \"fb965d3f-ff97-4e79-a803-adf9e2022788\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d788ba-5c7d7"
Oct 30 00:55:17.837103 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:55:17.837074    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4sjl\" (UniqueName: \"kubernetes.io/projected/fb965d3f-ff97-4e79-a803-adf9e2022788-kube-api-access-p4sjl\") pod \"maas-api-revoke-keys-e2e-api-a-d788ba-5c7d7\" (UID: \"fb965d3f-ff97-4e79-a803-adf9e2022788\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d788ba-5c7d7"
Oct 30 00:55:17.961923 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:55:17.961889    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d788ba-5c7d7"
Oct 30 00:55:18.265508 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:55:18.265463    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb965d3f_ff97_4e79_a803_adf9e2022788.slice/crio-ff01b46f370b9c8f3d81bf900caed025f02837b511ea3024715005afbba2df5e WatchSource:0}: Error finding container ff01b46f370b9c8f3d81bf900caed025f02837b511ea3024715005afbba2df5e: Status 404 returned error can't find the container with id ff01b46f370b9c8f3d81bf900caed025f02837b511ea3024715005afbba2df5e
Oct 30 00:55:18.267326 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:55:18.267298    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d788ba-5c7d7"]
Oct 30 00:55:18.499035 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:55:18.498999    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5160630-28d3-4af0-a8e9-d4b08e9f70dd" path="/var/lib/kubelet/pods/e5160630-28d3-4af0-a8e9-d4b08e9f70dd/volumes"
Oct 30 00:55:19.075740 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:55:19.075698    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d788ba-5c7d7" event={"ID":"fb965d3f-ff97-4e79-a803-adf9e2022788","Type":"ContainerStarted","Data":"3e61e37ea7be59c1f590eaf6a33ae6e26cc8aef860019cdf5bce700c0cebeb04"}
Oct 30 00:55:19.076336 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:55:19.076309    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d788ba-5c7d7" event={"ID":"fb965d3f-ff97-4e79-a803-adf9e2022788","Type":"ContainerStarted","Data":"ff01b46f370b9c8f3d81bf900caed025f02837b511ea3024715005afbba2df5e"}
Oct 30 00:55:19.124312 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:55:19.124242    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d788ba-5c7d7" podStartSLOduration=2.124223493 podStartE2EDuration="2.124223493s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:55:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:55:19.122568685 +0000 UTC m=+1414.739001565" watchObservedRunningTime="2026-10-30 00:55:19.124223493 +0000 UTC m=+1414.740656374"
Oct 30 00:55:20.084790 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:55:20.084751    5002 generic.go:356] "Generic (PLEG): container finished" podID="fb965d3f-ff97-4e79-a803-adf9e2022788" containerID="3e61e37ea7be59c1f590eaf6a33ae6e26cc8aef860019cdf5bce700c0cebeb04" exitCode=0
Oct 30 00:55:20.085184 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:55:20.084840    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d788ba-5c7d7" event={"ID":"fb965d3f-ff97-4e79-a803-adf9e2022788","Type":"ContainerDied","Data":"3e61e37ea7be59c1f590eaf6a33ae6e26cc8aef860019cdf5bce700c0cebeb04"}
Oct 30 00:55:21.357574 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:55:21.357542    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d788ba-5c7d7"
Oct 30 00:55:21.481068 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:55:21.481033    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb965d3f-ff97-4e79-a803-adf9e2022788-kube-api-access-p4sjl\" (UniqueName: \"kubernetes.io/projected/fb965d3f-ff97-4e79-a803-adf9e2022788-kube-api-access-p4sjl\") pod \"fb965d3f-ff97-4e79-a803-adf9e2022788\" (UID: \"fb965d3f-ff97-4e79-a803-adf9e2022788\") "
Oct 30 00:55:21.481273 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:55:21.481106    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb965d3f-ff97-4e79-a803-adf9e2022788-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb965d3f-ff97-4e79-a803-adf9e2022788-maas-api-xks-ca\") pod \"fb965d3f-ff97-4e79-a803-adf9e2022788\" (UID: \"fb965d3f-ff97-4e79-a803-adf9e2022788\") "
Oct 30 00:55:21.481349 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:55:21.481274    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb965d3f-ff97-4e79-a803-adf9e2022788-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb965d3f-ff97-4e79-a803-adf9e2022788-maas-api-openshift-ca\") pod \"fb965d3f-ff97-4e79-a803-adf9e2022788\" (UID: \"fb965d3f-ff97-4e79-a803-adf9e2022788\") "
Oct 30 00:55:21.481657 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:55:21.481631    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb965d3f-ff97-4e79-a803-adf9e2022788-maas-api-openshift-ca" pod "fb965d3f-ff97-4e79-a803-adf9e2022788" (UID: "fb965d3f-ff97-4e79-a803-adf9e2022788"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:55:21.482043 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:55:21.482023    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb965d3f-ff97-4e79-a803-adf9e2022788-maas-api-openshift-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:55:21.483770 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:55:21.483746    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb965d3f-ff97-4e79-a803-adf9e2022788-maas-api-xks-ca" pod "fb965d3f-ff97-4e79-a803-adf9e2022788" (UID: "fb965d3f-ff97-4e79-a803-adf9e2022788"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:55:21.483898 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:55:21.483787    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb965d3f-ff97-4e79-a803-adf9e2022788-kube-api-access-p4sjl" pod "fb965d3f-ff97-4e79-a803-adf9e2022788" (UID: "fb965d3f-ff97-4e79-a803-adf9e2022788"). InnerVolumeSpecName "kube-api-access-p4sjl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:55:21.583479 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:55:21.583436    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p4sjl\" (UniqueName: \"kubernetes.io/projected/fb965d3f-ff97-4e79-a803-adf9e2022788-kube-api-access-p4sjl\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:55:21.583479 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:55:21.583475    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb965d3f-ff97-4e79-a803-adf9e2022788-maas-api-xks-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:55:22.102837 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:55:22.102805    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d788ba-5c7d7"
Oct 30 00:55:22.103137 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:55:22.102824    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d788ba-5c7d7" event={"ID":"fb965d3f-ff97-4e79-a803-adf9e2022788","Type":"ContainerDied","Data":"ff01b46f370b9c8f3d81bf900caed025f02837b511ea3024715005afbba2df5e"}
Oct 30 00:55:22.103333 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:55:22.103317    5002 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ff01b46f370b9c8f3d81bf900caed025f02837b511ea3024715005afbba2df5e"
Oct 30 00:55:25.376145 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:55:25.376109    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bdc256-n7svj"]
Oct 30 00:55:25.377612 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:55:25.377583    5002 cpu_manager.go:414] "containerMap: removing container" podUID="fb965d3f-ff97-4e79-a803-adf9e2022788" containerName="revoke-keys"
Oct 30 00:55:25.377612 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:55:25.377609    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb965d3f-ff97-4e79-a803-adf9e2022788" containerName="revoke-keys"
Oct 30 00:55:25.377868 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:55:25.377845    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb965d3f-ff97-4e79-a803-adf9e2022788" containerName="revoke-keys"
Oct 30 00:55:25.382992 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:55:25.382974    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bdc256-n7svj"
Oct 30 00:55:25.400987 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:55:25.400963    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bdc256-n7svj"]
Oct 30 00:55:25.409328 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:55:25.409298    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 00:55:25.409499 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:55:25.409297    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 00:55:25.458807 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:55:25.458776    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 00:55:25.458807 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:55:25.458786    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5phsh\""
Oct 30 00:55:25.478834 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:55:25.478795    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9fa8adb3-1011-4865-93d9-27f1f187a327-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-bdc256-n7svj\" (UID: \"9fa8adb3-1011-4865-93d9-27f1f187a327\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bdc256-n7svj"
Oct 30 00:55:25.478992 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:55:25.478839    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckcjb\" (UniqueName: \"kubernetes.io/projected/9fa8adb3-1011-4865-93d9-27f1f187a327-kube-api-access-ckcjb\") pod \"maas-api-revoke-keys-e2e-api-b-bdc256-n7svj\" (UID: \"9fa8adb3-1011-4865-93d9-27f1f187a327\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bdc256-n7svj"
Oct 30 00:55:25.479042 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:55:25.479002    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9fa8adb3-1011-4865-93d9-27f1f187a327-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-bdc256-n7svj\" (UID: \"9fa8adb3-1011-4865-93d9-27f1f187a327\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bdc256-n7svj"
Oct 30 00:55:25.580806 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:55:25.580755    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9fa8adb3-1011-4865-93d9-27f1f187a327-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-bdc256-n7svj\" (UID: \"9fa8adb3-1011-4865-93d9-27f1f187a327\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bdc256-n7svj"
Oct 30 00:55:25.581067 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:55:25.581043    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckcjb\" (UniqueName: \"kubernetes.io/projected/9fa8adb3-1011-4865-93d9-27f1f187a327-kube-api-access-ckcjb\") pod \"maas-api-revoke-keys-e2e-api-b-bdc256-n7svj\" (UID: \"9fa8adb3-1011-4865-93d9-27f1f187a327\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bdc256-n7svj"
Oct 30 00:55:25.581424 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:55:25.581397    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9fa8adb3-1011-4865-93d9-27f1f187a327-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-bdc256-n7svj\" (UID: \"9fa8adb3-1011-4865-93d9-27f1f187a327\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bdc256-n7svj"
Oct 30 00:55:25.581663 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:55:25.581640    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9fa8adb3-1011-4865-93d9-27f1f187a327-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-bdc256-n7svj\" (UID: \"9fa8adb3-1011-4865-93d9-27f1f187a327\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bdc256-n7svj"
Oct 30 00:55:25.583896 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:55:25.583867    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9fa8adb3-1011-4865-93d9-27f1f187a327-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-bdc256-n7svj\" (UID: \"9fa8adb3-1011-4865-93d9-27f1f187a327\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bdc256-n7svj"
Oct 30 00:55:25.606508 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:55:25.606481    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckcjb\" (UniqueName: \"kubernetes.io/projected/9fa8adb3-1011-4865-93d9-27f1f187a327-kube-api-access-ckcjb\") pod \"maas-api-revoke-keys-e2e-api-b-bdc256-n7svj\" (UID: \"9fa8adb3-1011-4865-93d9-27f1f187a327\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bdc256-n7svj"
Oct 30 00:55:25.701641 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:55:25.701608    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bdc256-n7svj"
Oct 30 00:55:26.004626 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:55:26.004595    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fa8adb3_1011_4865_93d9_27f1f187a327.slice/crio-afc764fdec877ca84990c6e812ed4b512d1f987e0b7a119ef72b242291e807f5 WatchSource:0}: Error finding container afc764fdec877ca84990c6e812ed4b512d1f987e0b7a119ef72b242291e807f5: Status 404 returned error can't find the container with id afc764fdec877ca84990c6e812ed4b512d1f987e0b7a119ef72b242291e807f5
Oct 30 00:55:26.006998 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:55:26.006970    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bdc256-n7svj"]
Oct 30 00:55:26.147784 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:55:26.147750    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bdc256-n7svj" event={"ID":"9fa8adb3-1011-4865-93d9-27f1f187a327","Type":"ContainerStarted","Data":"afc764fdec877ca84990c6e812ed4b512d1f987e0b7a119ef72b242291e807f5"}
Oct 30 00:55:27.157426 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:55:27.157320    5002 generic.go:356] "Generic (PLEG): container finished" podID="9fa8adb3-1011-4865-93d9-27f1f187a327" containerID="06cd0bbfb970047f19c28fe86cf05a040fd55a7b4c561fcc3e9e6130741aa439" exitCode=0
Oct 30 00:55:27.157855 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:55:27.157415    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bdc256-n7svj" event={"ID":"9fa8adb3-1011-4865-93d9-27f1f187a327","Type":"ContainerDied","Data":"06cd0bbfb970047f19c28fe86cf05a040fd55a7b4c561fcc3e9e6130741aa439"}
Oct 30 00:55:28.407261 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:55:28.407232    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bdc256-n7svj"
Oct 30 00:55:28.427159 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:55:28.427085    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9fa8adb3-1011-4865-93d9-27f1f187a327-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9fa8adb3-1011-4865-93d9-27f1f187a327-maas-api-openshift-ca\") pod \"9fa8adb3-1011-4865-93d9-27f1f187a327\" (UID: \"9fa8adb3-1011-4865-93d9-27f1f187a327\") "
Oct 30 00:55:28.427296 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:55:28.427265    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fa8adb3-1011-4865-93d9-27f1f187a327-kube-api-access-ckcjb\" (UniqueName: \"kubernetes.io/projected/9fa8adb3-1011-4865-93d9-27f1f187a327-kube-api-access-ckcjb\") pod \"9fa8adb3-1011-4865-93d9-27f1f187a327\" (UID: \"9fa8adb3-1011-4865-93d9-27f1f187a327\") "
Oct 30 00:55:28.427419 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:55:28.427399    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9fa8adb3-1011-4865-93d9-27f1f187a327-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9fa8adb3-1011-4865-93d9-27f1f187a327-maas-api-xks-ca\") pod \"9fa8adb3-1011-4865-93d9-27f1f187a327\" (UID: \"9fa8adb3-1011-4865-93d9-27f1f187a327\") "
Oct 30 00:55:28.427508 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:55:28.427486    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fa8adb3-1011-4865-93d9-27f1f187a327-maas-api-openshift-ca" pod "9fa8adb3-1011-4865-93d9-27f1f187a327" (UID: "9fa8adb3-1011-4865-93d9-27f1f187a327"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:55:28.427973 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:55:28.427952    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9fa8adb3-1011-4865-93d9-27f1f187a327-maas-api-openshift-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:55:28.429423 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:55:28.429397    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fa8adb3-1011-4865-93d9-27f1f187a327-kube-api-access-ckcjb" pod "9fa8adb3-1011-4865-93d9-27f1f187a327" (UID: "9fa8adb3-1011-4865-93d9-27f1f187a327"). InnerVolumeSpecName "kube-api-access-ckcjb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:55:28.429518 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:55:28.429474    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9fa8adb3-1011-4865-93d9-27f1f187a327-maas-api-xks-ca" pod "9fa8adb3-1011-4865-93d9-27f1f187a327" (UID: "9fa8adb3-1011-4865-93d9-27f1f187a327"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:55:28.539406 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:55:28.530164    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ckcjb\" (UniqueName: \"kubernetes.io/projected/9fa8adb3-1011-4865-93d9-27f1f187a327-kube-api-access-ckcjb\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:55:28.539406 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:55:28.530203    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9fa8adb3-1011-4865-93d9-27f1f187a327-maas-api-xks-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:55:29.176855 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:55:29.176824    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bdc256-n7svj"
Oct 30 00:55:29.177054 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:55:29.176862    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bdc256-n7svj" event={"ID":"9fa8adb3-1011-4865-93d9-27f1f187a327","Type":"ContainerDied","Data":"afc764fdec877ca84990c6e812ed4b512d1f987e0b7a119ef72b242291e807f5"}
Oct 30 00:55:29.177054 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:55:29.176909    5002 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="afc764fdec877ca84990c6e812ed4b512d1f987e0b7a119ef72b242291e807f5"
Oct 30 00:55:32.539491 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:55:32.539455    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7d8z4"]
Oct 30 00:55:36.294798 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:55:36.294754    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1a11cf/distinct-w2-1a11cf-kserve-854cf4cdf8-cddw6" podUID="32a03478-46eb-4363-97b6-51a32c4099c7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded"
Oct 30 00:55:36.321544 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:55:36.321149    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1a11cf/distinct-2-w2-1a11cf-kserve-758bb6dd57-j5f2k" podUID="f2d66dea-9f0b-43f3-a774-bb065e88f378" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:55:36.378401 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:55:36.368484    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6f4fba/simulator-w5-6f4fba-kserve-5bf8dfddd5-wcwzh" podUID="3a2c2f8f-649c-4ab6-af98-5d87df0e511f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:55:36.642346 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:55:36.642256    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-52650f/simulator-w0-52650f-kserve-58b64766c-94qxx" podUID="659c1425-5b67-4d01-bfc2-f1ba56b9a59b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:55:37.047425 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:55:37.045201    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-29be73/simulator-w1-29be73-kserve-9958d76db-fks2t" podUID="fbb3c806-8967-4d39-84c6-9dc022fc5cae" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/ready\": context deadline exceeded"
Oct 30 00:55:37.399473 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:55:37.399085    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-021d6d/simulator-w6-021d6d-kserve-6cc97fdf86-dqtpw" podUID="50586f76-4f3b-4e7a-83ac-6c573faee415" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:55:37.714530 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:55:37.714479    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-52650f/simulator-w0-52650f-kserve-58b64766c-94qxx" podUID="659c1425-5b67-4d01-bfc2-f1ba56b9a59b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:55:37.720414 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:55:37.715194    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-52650f/simulator-w0-52650f-kserve-58b64766c-94qxx"
Oct 30 00:55:37.730508 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:55:37.730465    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca5363/distinct-2-w4-ca5363-kserve-576576f9b5-xkkqc" podUID="5de8bebf-cfad-49da-9a98-08aedde4442c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:55:37.769526 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:55:37.769478    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cefb27/simulator-w3-cefb27-kserve-6b669c9d9f-zx2vm" podUID="7121e81e-f903-42c0-a7a4-bc18d49194e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:55:38.047620 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:55:38.047495    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-29be73/simulator-w1-29be73-kserve-9958d76db-fks2t" podUID="fbb3c806-8967-4d39-84c6-9dc022fc5cae" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:55:38.086405 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:55:38.068788    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-29be73/simulator-w1-29be73-kserve-9958d76db-fks2t"
Oct 30 00:55:38.412690 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:55:38.412470    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-021d6d/simulator-w6-021d6d-kserve-6cc97fdf86-dqtpw" podUID="50586f76-4f3b-4e7a-83ac-6c573faee415" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:55:38.427338 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:55:38.427305    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-021d6d/simulator-w6-021d6d-kserve-6cc97fdf86-dqtpw"
Oct 30 00:55:38.440213 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:55:38.440170    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1a11cf/distinct-2-w2-1a11cf-kserve-758bb6dd57-j5f2k" podUID="f2d66dea-9f0b-43f3-a774-bb065e88f378" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:55:38.519661 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:55:38.519619    5002 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-29be73/premium-w1-29be73-kserve-7b777cfc8c-7kgp5" podUID="e6fabbdd-86ca-4cbf-85be-ef250575a355" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/health\": context deadline exceeded"
Oct 30 00:55:38.715413 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:55:38.715348    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-52650f/simulator-w0-52650f-kserve-58b64766c-94qxx" podUID="659c1425-5b67-4d01-bfc2-f1ba56b9a59b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:55:38.736320 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:55:38.736280    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca5363/distinct-2-w4-ca5363-kserve-576576f9b5-xkkqc" podUID="5de8bebf-cfad-49da-9a98-08aedde4442c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:55:38.816116 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:55:38.816079    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ca5363/distinct-2-w4-ca5363-kserve-576576f9b5-xkkqc"
Oct 30 00:55:38.995463 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:55:38.972110    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-52650f/distinct-w0-52650f-kserve-d85899b5-vlhps" podUID="ea1850d9-abba-4977-a540-f6e0eaefb17d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:55:39.062404 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:55:39.047168    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-52650f/simulator-w0-52650f-kserve-58b64766c-94qxx"
Oct 30 00:55:39.184405 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:55:39.183693    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-29be73/premium-w1-29be73-kserve-7b777cfc8c-7kgp5" podUID="e6fabbdd-86ca-4cbf-85be-ef250575a355" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": context deadline exceeded"
Oct 30 00:55:39.209405 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:55:39.186397    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-29be73/simulator-w1-29be73-kserve-9958d76db-fks2t" podUID="fbb3c806-8967-4d39-84c6-9dc022fc5cae" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:55:39.414555 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:55:39.414455    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-021d6d/simulator-w6-021d6d-kserve-6cc97fdf86-dqtpw" podUID="50586f76-4f3b-4e7a-83ac-6c573faee415" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:55:39.529839 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:55:39.529808    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-021d6d/simulator-w6-021d6d-kserve-6cc97fdf86-dqtpw"
Oct 30 00:55:39.741432 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:55:39.739470    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca5363/distinct-2-w4-ca5363-kserve-576576f9b5-xkkqc" podUID="5de8bebf-cfad-49da-9a98-08aedde4442c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Oct 30 00:55:39.768116 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:55:39.767869    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cefb27/simulator-w3-cefb27-kserve-6b669c9d9f-zx2vm" podUID="7121e81e-f903-42c0-a7a4-bc18d49194e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/ready\": context deadline exceeded"
Oct 30 00:55:39.857746 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:55:39.857714    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-29be73/simulator-w1-29be73-kserve-9958d76db-fks2t"
Oct 30 00:55:40.143280 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:55:40.143176    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-52650f/simulator-w0-52650f-kserve-58b64766c-94qxx" podUID="659c1425-5b67-4d01-bfc2-f1ba56b9a59b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:55:40.235047 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:55:40.219513    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-29be73/premium-w1-29be73-kserve-7b777cfc8c-7kgp5" podUID="e6fabbdd-86ca-4cbf-85be-ef250575a355" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": context deadline exceeded"
Oct 30 00:55:40.235047 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:55:40.220234    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-29be73/premium-w1-29be73-kserve-7b777cfc8c-7kgp5"
Oct 30 00:55:40.235047 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:55:40.231249    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ca5363/distinct-2-w4-ca5363-kserve-576576f9b5-xkkqc"
Oct 30 00:55:40.235047 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:55:40.234646    5002 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-ca5363/distinct-2-w4-ca5363-kserve-576576f9b5-xkkqc" podUID="5de8bebf-cfad-49da-9a98-08aedde4442c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:55:40.236690 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:55:40.236600    5002 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-1a11cf/distinct-2-w2-1a11cf-kserve-758bb6dd57-j5f2k" podUID="f2d66dea-9f0b-43f3-a774-bb065e88f378" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:55:40.315403 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:55:40.315150    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1a11cf/distinct-2-w2-1a11cf-kserve-758bb6dd57-j5f2k" podUID="f2d66dea-9f0b-43f3-a774-bb065e88f378" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:55:40.342822 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:55:40.342793    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-29be73/premium-w1-29be73-kserve-7b777cfc8c-7kgp5"
Oct 30 00:55:42.870784 ip-10-0-128-7 kubenswrapper[5002]: E1030 00:55:42.870741    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbjdq" containerName="extractor"
Oct 30 00:55:44.228524 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:55:44.228486    5002 patch_prober.go:29] interesting pod/e2e-inf-a-6444d7-openshift-default-7bf76bdcd9-6nnq2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 00:55:44.228984 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:55:44.228553    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-6444d7-openshift-default-7bf76bdcd9-6nnq2" podUID="750a98de-3674-4c5b-835c-48b928841cd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:55:46.371866 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:55:46.371836    5002 scope.go:122] "RemoveContainer" containerID="ce0bf28697e53c08afa8e664a0b806a1f15a282be73cfbd3c6817f1d8ea07722"
Oct 30 00:55:46.928416 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:55:46.928357    5002 generic.go:356] "Generic (PLEG): container finished" podID="750a98de-3674-4c5b-835c-48b928841cd7" containerID="b9052f3a47331b79001912f9b046332d2fc3bf08b67fde883354b6e30c262916" exitCode=0
Oct 30 00:55:46.928587 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:55:46.928434    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6444d7-openshift-default-7bf76bdcd9-6nnq2" event={"ID":"750a98de-3674-4c5b-835c-48b928841cd7","Type":"ContainerDied","Data":"b9052f3a47331b79001912f9b046332d2fc3bf08b67fde883354b6e30c262916"}
Oct 30 00:55:46.929016 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:55:46.928998    5002 scope.go:122] "RemoveContainer" containerID="b9052f3a47331b79001912f9b046332d2fc3bf08b67fde883354b6e30c262916"
Oct 30 00:55:46.930579 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:55:46.930548    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 00:55:46.930639 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:55:46.930620    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 00:55:46.930673 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:55:46.930652    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 00:55:47.937829 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:55:47.937796    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6444d7-openshift-default-7bf76bdcd9-6nnq2" event={"ID":"750a98de-3674-4c5b-835c-48b928841cd7","Type":"ContainerStarted","Data":"128a0167b68df93e5d9adce649b2359b6064d025992b8ab66c9709b7e511ebff"}
Oct 30 00:55:48.886531 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:55:48.886428    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-6444d7-openshift-default-7bf76bdcd9-6nnq2"
Oct 30 00:55:48.886811 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:55:48.886780    5002 patch_prober.go:29] interesting pod/e2e-inf-a-6444d7-openshift-default-7bf76bdcd9-6nnq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body=
Oct 30 00:55:48.886883 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:55:48.886844    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6444d7-openshift-default-7bf76bdcd9-6nnq2" podUID="750a98de-3674-4c5b-835c-48b928841cd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused"
Oct 30 00:55:49.893403 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:55:49.889361    5002 patch_prober.go:29] interesting pod/e2e-inf-a-6444d7-openshift-default-7bf76bdcd9-6nnq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body=
Oct 30 00:55:49.893403 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:55:49.889456    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6444d7-openshift-default-7bf76bdcd9-6nnq2" podUID="750a98de-3674-4c5b-835c-48b928841cd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused"
Oct 30 00:55:50.885888 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:55:50.885847    5002 patch_prober.go:29] interesting pod/e2e-inf-a-6444d7-openshift-default-7bf76bdcd9-6nnq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body=
Oct 30 00:55:50.886088 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:55:50.885933    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6444d7-openshift-default-7bf76bdcd9-6nnq2" podUID="750a98de-3674-4c5b-835c-48b928841cd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused"
Oct 30 00:55:51.886620 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:55:51.886582    5002 patch_prober.go:29] interesting pod/e2e-inf-a-6444d7-openshift-default-7bf76bdcd9-6nnq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body=
Oct 30 00:55:51.887092 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:55:51.886653    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6444d7-openshift-default-7bf76bdcd9-6nnq2" podUID="750a98de-3674-4c5b-835c-48b928841cd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused"
Oct 30 00:55:52.886845 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:55:52.886806    5002 patch_prober.go:29] interesting pod/e2e-inf-a-6444d7-openshift-default-7bf76bdcd9-6nnq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body=
Oct 30 00:55:52.887523 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:55:52.887487    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6444d7-openshift-default-7bf76bdcd9-6nnq2" podUID="750a98de-3674-4c5b-835c-48b928841cd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused"
Oct 30 00:55:54.026407 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:55:54.022824    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-6444d7-openshift-default-7bf76bdcd9-6nnq2"
Oct 30 00:55:54.026407 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:55:54.024742    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-6444d7-openshift-default-7bf76bdcd9-6nnq2"
Oct 30 00:55:54.073819 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:55:54.073779    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-6444d7-openshift-default-7bf76bdcd9-6nnq2"
Oct 30 00:56:00.040799 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:00.040751    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f92e816-s28n5"]
Oct 30 00:56:00.054090 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:00.054058    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f92e816-s28n5"]
Oct 30 00:56:00.498250 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:00.498207    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87b20090-dc73-4f5b-a8bb-082d2a4365ec" path="/var/lib/kubelet/pods/87b20090-dc73-4f5b-a8bb-082d2a4365ec/volumes"
Oct 30 00:56:01.814034 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:01.813987    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-d7a6db25/e2e-lifecycle-model-d7a6db25-kserve-556c6db9f7-nsdhk"]
Oct 30 00:56:04.246737 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:04.246703    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d788ba-5c7d7"]
Oct 30 00:56:04.285409 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:04.276126    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d788ba-5c7d7"]
Oct 30 00:56:04.497909 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:04.497826    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb965d3f-ff97-4e79-a803-adf9e2022788" path="/var/lib/kubelet/pods/fb965d3f-ff97-4e79-a803-adf9e2022788/volumes"
Oct 30 00:56:12.380580 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:12.380543    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-02fbf5-w477t"]
Oct 30 00:56:12.381618 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:12.381597    5002 cpu_manager.go:414] "containerMap: removing container" podUID="9fa8adb3-1011-4865-93d9-27f1f187a327" containerName="revoke-keys"
Oct 30 00:56:12.381722 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:12.381622    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fa8adb3-1011-4865-93d9-27f1f187a327" containerName="revoke-keys"
Oct 30 00:56:12.381821 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:12.381808    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="9fa8adb3-1011-4865-93d9-27f1f187a327" containerName="revoke-keys"
Oct 30 00:56:12.386276 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:12.386253    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-02fbf5-w477t"
Oct 30 00:56:12.404217 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:12.404191    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-02fbf5-w477t"]
Oct 30 00:56:12.413206 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:12.413180    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 00:56:12.413357 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:12.413232    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 00:56:12.415578 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:12.415536    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 00:56:12.415578 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:12.415579    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5phsh\""
Oct 30 00:56:12.454533 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:12.454432    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1178ba7-6655-4ecd-b2f5-79477a445322-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-02fbf5-w477t\" (UID: \"b1178ba7-6655-4ecd-b2f5-79477a445322\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-02fbf5-w477t"
Oct 30 00:56:12.454941 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:12.454864    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1178ba7-6655-4ecd-b2f5-79477a445322-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-02fbf5-w477t\" (UID: \"b1178ba7-6655-4ecd-b2f5-79477a445322\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-02fbf5-w477t"
Oct 30 00:56:12.455257 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:12.455093    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jg9d\" (UniqueName: \"kubernetes.io/projected/b1178ba7-6655-4ecd-b2f5-79477a445322-kube-api-access-9jg9d\") pod \"maas-api-revoke-keys-e2e-api-b-02fbf5-w477t\" (UID: \"b1178ba7-6655-4ecd-b2f5-79477a445322\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-02fbf5-w477t"
Oct 30 00:56:12.556812 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:12.556775    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1178ba7-6655-4ecd-b2f5-79477a445322-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-02fbf5-w477t\" (UID: \"b1178ba7-6655-4ecd-b2f5-79477a445322\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-02fbf5-w477t"
Oct 30 00:56:12.557012 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:12.556859    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1178ba7-6655-4ecd-b2f5-79477a445322-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-02fbf5-w477t\" (UID: \"b1178ba7-6655-4ecd-b2f5-79477a445322\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-02fbf5-w477t"
Oct 30 00:56:12.557231 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:12.556961    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jg9d\" (UniqueName: \"kubernetes.io/projected/b1178ba7-6655-4ecd-b2f5-79477a445322-kube-api-access-9jg9d\") pod \"maas-api-revoke-keys-e2e-api-b-02fbf5-w477t\" (UID: \"b1178ba7-6655-4ecd-b2f5-79477a445322\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-02fbf5-w477t"
Oct 30 00:56:12.557744 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:12.557719    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1178ba7-6655-4ecd-b2f5-79477a445322-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-02fbf5-w477t\" (UID: \"b1178ba7-6655-4ecd-b2f5-79477a445322\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-02fbf5-w477t"
Oct 30 00:56:12.559329 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:12.559306    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1178ba7-6655-4ecd-b2f5-79477a445322-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-02fbf5-w477t\" (UID: \"b1178ba7-6655-4ecd-b2f5-79477a445322\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-02fbf5-w477t"
Oct 30 00:56:12.581928 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:12.581902    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jg9d\" (UniqueName: \"kubernetes.io/projected/b1178ba7-6655-4ecd-b2f5-79477a445322-kube-api-access-9jg9d\") pod \"maas-api-revoke-keys-e2e-api-b-02fbf5-w477t\" (UID: \"b1178ba7-6655-4ecd-b2f5-79477a445322\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-02fbf5-w477t"
Oct 30 00:56:12.705932 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:12.705902    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-02fbf5-w477t"
Oct 30 00:56:12.871165 ip-10-0-128-7 kubenswrapper[5002]: E1030 00:56:12.871100    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbjdq" containerName="extractor"
Oct 30 00:56:13.045234 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:13.045173    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-02fbf5-w477t"]
Oct 30 00:56:13.217025 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:13.216986    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-02fbf5-w477t" event={"ID":"b1178ba7-6655-4ecd-b2f5-79477a445322","Type":"ContainerStarted","Data":"ff2192e2a5edf29816257c8316c6bbc52b6bf212537b643b1740a927cc2fe682"}
Oct 30 00:56:13.364324 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:13.364221    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7d8z4"]
Oct 30 00:56:14.224825 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:14.224793    5002 generic.go:356] "Generic (PLEG): container finished" podID="b1178ba7-6655-4ecd-b2f5-79477a445322" containerID="b5c7f311d5708ffb1d37bd7acf0d68522dca6d386b116400a59b37e10188ff1e" exitCode=0
Oct 30 00:56:14.225239 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:14.224873    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-02fbf5-w477t" event={"ID":"b1178ba7-6655-4ecd-b2f5-79477a445322","Type":"ContainerDied","Data":"b5c7f311d5708ffb1d37bd7acf0d68522dca6d386b116400a59b37e10188ff1e"}
Oct 30 00:56:15.410099 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:15.410067    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-02fbf5-w477t"
Oct 30 00:56:15.533346 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:15.533267    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b1178ba7-6655-4ecd-b2f5-79477a445322-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1178ba7-6655-4ecd-b2f5-79477a445322-maas-api-openshift-ca\") pod \"b1178ba7-6655-4ecd-b2f5-79477a445322\" (UID: \"b1178ba7-6655-4ecd-b2f5-79477a445322\") "
Oct 30 00:56:15.533511 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:15.533469    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b1178ba7-6655-4ecd-b2f5-79477a445322-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1178ba7-6655-4ecd-b2f5-79477a445322-maas-api-xks-ca\") pod \"b1178ba7-6655-4ecd-b2f5-79477a445322\" (UID: \"b1178ba7-6655-4ecd-b2f5-79477a445322\") "
Oct 30 00:56:15.533557 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:15.533519    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1178ba7-6655-4ecd-b2f5-79477a445322-kube-api-access-9jg9d\" (UniqueName: \"kubernetes.io/projected/b1178ba7-6655-4ecd-b2f5-79477a445322-kube-api-access-9jg9d\") pod \"b1178ba7-6655-4ecd-b2f5-79477a445322\" (UID: \"b1178ba7-6655-4ecd-b2f5-79477a445322\") "
Oct 30 00:56:15.533706 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:15.533678    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1178ba7-6655-4ecd-b2f5-79477a445322-maas-api-openshift-ca" pod "b1178ba7-6655-4ecd-b2f5-79477a445322" (UID: "b1178ba7-6655-4ecd-b2f5-79477a445322"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:56:15.534093 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:15.534072    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1178ba7-6655-4ecd-b2f5-79477a445322-maas-api-openshift-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:56:15.535673 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:15.535648    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1178ba7-6655-4ecd-b2f5-79477a445322-kube-api-access-9jg9d" pod "b1178ba7-6655-4ecd-b2f5-79477a445322" (UID: "b1178ba7-6655-4ecd-b2f5-79477a445322"). InnerVolumeSpecName "kube-api-access-9jg9d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:56:15.535673 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:15.535667    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1178ba7-6655-4ecd-b2f5-79477a445322-maas-api-xks-ca" pod "b1178ba7-6655-4ecd-b2f5-79477a445322" (UID: "b1178ba7-6655-4ecd-b2f5-79477a445322"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:56:15.635640 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:15.635602    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1178ba7-6655-4ecd-b2f5-79477a445322-maas-api-xks-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:56:15.635640 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:15.635641    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9jg9d\" (UniqueName: \"kubernetes.io/projected/b1178ba7-6655-4ecd-b2f5-79477a445322-kube-api-access-9jg9d\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:56:16.239444 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:16.239403    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-02fbf5-w477t"
Oct 30 00:56:16.239665 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:16.239413    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-02fbf5-w477t" event={"ID":"b1178ba7-6655-4ecd-b2f5-79477a445322","Type":"ContainerDied","Data":"ff2192e2a5edf29816257c8316c6bbc52b6bf212537b643b1740a927cc2fe682"}
Oct 30 00:56:16.239665 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:16.239530    5002 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ff2192e2a5edf29816257c8316c6bbc52b6bf212537b643b1740a927cc2fe682"
Oct 30 00:56:16.836557 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:16.836493    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-mt-d7a6db25/e2e-lifecycle-model-d7a6db25-kserve-556c6db9f7-nsdhk" podUID="c783e213-5fa1-4e80-9f2d-66b362015d39" containerName="main" containerID="cri-o://8bd15bc70e4c41517f36d790600910b34ea8e3b13e0c55ba5ab961cf78099d69" gracePeriod=45
Oct 30 00:56:17.139708 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:17.139675    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-d7a6db25/e2e-lifecycle-model-d7a6db25-kserve-556c6db9f7-nsdhk"
Oct 30 00:56:17.250759 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:17.250720    5002 generic.go:356] "Generic (PLEG): container finished" podID="c783e213-5fa1-4e80-9f2d-66b362015d39" containerID="8bd15bc70e4c41517f36d790600910b34ea8e3b13e0c55ba5ab961cf78099d69" exitCode=0
Oct 30 00:56:17.250920 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:17.250781    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-d7a6db25/e2e-lifecycle-model-d7a6db25-kserve-556c6db9f7-nsdhk"
Oct 30 00:56:17.250920 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:17.250782    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-d7a6db25/e2e-lifecycle-model-d7a6db25-kserve-556c6db9f7-nsdhk" event={"ID":"c783e213-5fa1-4e80-9f2d-66b362015d39","Type":"ContainerDied","Data":"8bd15bc70e4c41517f36d790600910b34ea8e3b13e0c55ba5ab961cf78099d69"}
Oct 30 00:56:17.250920 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:17.250913    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-d7a6db25/e2e-lifecycle-model-d7a6db25-kserve-556c6db9f7-nsdhk" event={"ID":"c783e213-5fa1-4e80-9f2d-66b362015d39","Type":"ContainerDied","Data":"42fa9d5d6590de0094b07bb763e61e5da7abc5c06ce0cf9df874f9d0d5dcdf0e"}
Oct 30 00:56:17.251054 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:17.250940    5002 scope.go:122] "RemoveContainer" containerID="8bd15bc70e4c41517f36d790600910b34ea8e3b13e0c55ba5ab961cf78099d69"
Oct 30 00:56:17.261558 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:17.261530    5002 scope.go:122] "RemoveContainer" containerID="8bd15bc70e4c41517f36d790600910b34ea8e3b13e0c55ba5ab961cf78099d69"
Oct 30 00:56:17.261893 ip-10-0-128-7 kubenswrapper[5002]: E1030 00:56:17.261871    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8bd15bc70e4c41517f36d790600910b34ea8e3b13e0c55ba5ab961cf78099d69\": container with ID starting with 8bd15bc70e4c41517f36d790600910b34ea8e3b13e0c55ba5ab961cf78099d69 not found: ID does not exist" containerID="8bd15bc70e4c41517f36d790600910b34ea8e3b13e0c55ba5ab961cf78099d69"
Oct 30 00:56:17.261962 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:17.261906    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8bd15bc70e4c41517f36d790600910b34ea8e3b13e0c55ba5ab961cf78099d69"} err="failed to get container status \"8bd15bc70e4c41517f36d790600910b34ea8e3b13e0c55ba5ab961cf78099d69\": rpc error: code = NotFound desc = could not find container \"8bd15bc70e4c41517f36d790600910b34ea8e3b13e0c55ba5ab961cf78099d69\": container with ID starting with 8bd15bc70e4c41517f36d790600910b34ea8e3b13e0c55ba5ab961cf78099d69 not found: ID does not exist"
Oct 30 00:56:17.261962 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:17.261929    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c783e213-5fa1-4e80-9f2d-66b362015d39-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c783e213-5fa1-4e80-9f2d-66b362015d39-tmp-dir\") pod \"c783e213-5fa1-4e80-9f2d-66b362015d39\" (UID: \"c783e213-5fa1-4e80-9f2d-66b362015d39\") "
Oct 30 00:56:17.262130 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:17.262112    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c783e213-5fa1-4e80-9f2d-66b362015d39-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c783e213-5fa1-4e80-9f2d-66b362015d39-dshm\") pod \"c783e213-5fa1-4e80-9f2d-66b362015d39\" (UID: \"c783e213-5fa1-4e80-9f2d-66b362015d39\") "
Oct 30 00:56:17.262555 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:17.262172    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c783e213-5fa1-4e80-9f2d-66b362015d39-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c783e213-5fa1-4e80-9f2d-66b362015d39-model-cache\") pod \"c783e213-5fa1-4e80-9f2d-66b362015d39\" (UID: \"c783e213-5fa1-4e80-9f2d-66b362015d39\") "
Oct 30 00:56:17.262555 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:17.262217    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c783e213-5fa1-4e80-9f2d-66b362015d39-tmp-dir" pod "c783e213-5fa1-4e80-9f2d-66b362015d39" (UID: "c783e213-5fa1-4e80-9f2d-66b362015d39"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:56:17.262555 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:17.262306    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c783e213-5fa1-4e80-9f2d-66b362015d39-home\" (UniqueName: \"kubernetes.io/empty-dir/c783e213-5fa1-4e80-9f2d-66b362015d39-home\") pod \"c783e213-5fa1-4e80-9f2d-66b362015d39\" (UID: \"c783e213-5fa1-4e80-9f2d-66b362015d39\") "
Oct 30 00:56:17.262555 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:17.262348    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c783e213-5fa1-4e80-9f2d-66b362015d39-tls-certs\" (UniqueName: \"kubernetes.io/secret/c783e213-5fa1-4e80-9f2d-66b362015d39-tls-certs\") pod \"c783e213-5fa1-4e80-9f2d-66b362015d39\" (UID: \"c783e213-5fa1-4e80-9f2d-66b362015d39\") "
Oct 30 00:56:17.262555 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:17.262452    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c783e213-5fa1-4e80-9f2d-66b362015d39-model-cache" pod "c783e213-5fa1-4e80-9f2d-66b362015d39" (UID: "c783e213-5fa1-4e80-9f2d-66b362015d39"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:56:17.262555 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:17.262551    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c783e213-5fa1-4e80-9f2d-66b362015d39-kube-api-access-59kx9\" (UniqueName: \"kubernetes.io/projected/c783e213-5fa1-4e80-9f2d-66b362015d39-kube-api-access-59kx9\") pod \"c783e213-5fa1-4e80-9f2d-66b362015d39\" (UID: \"c783e213-5fa1-4e80-9f2d-66b362015d39\") "
Oct 30 00:56:17.262863 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:17.262638    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c783e213-5fa1-4e80-9f2d-66b362015d39-home" pod "c783e213-5fa1-4e80-9f2d-66b362015d39" (UID: "c783e213-5fa1-4e80-9f2d-66b362015d39"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:56:17.263325 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:17.263299    5002 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c783e213-5fa1-4e80-9f2d-66b362015d39-model-cache\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:56:17.263325 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:17.263327    5002 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c783e213-5fa1-4e80-9f2d-66b362015d39-home\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:56:17.263325 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:17.263336    5002 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c783e213-5fa1-4e80-9f2d-66b362015d39-tmp-dir\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:56:17.264931 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:17.264908    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c783e213-5fa1-4e80-9f2d-66b362015d39-tls-certs" pod "c783e213-5fa1-4e80-9f2d-66b362015d39" (UID: "c783e213-5fa1-4e80-9f2d-66b362015d39"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:56:17.265033 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:17.264965    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c783e213-5fa1-4e80-9f2d-66b362015d39-dshm" pod "c783e213-5fa1-4e80-9f2d-66b362015d39" (UID: "c783e213-5fa1-4e80-9f2d-66b362015d39"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:56:17.265147 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:17.265133    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c783e213-5fa1-4e80-9f2d-66b362015d39-kube-api-access-59kx9" pod "c783e213-5fa1-4e80-9f2d-66b362015d39" (UID: "c783e213-5fa1-4e80-9f2d-66b362015d39"). InnerVolumeSpecName "kube-api-access-59kx9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:56:17.364451 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:17.364355    5002 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c783e213-5fa1-4e80-9f2d-66b362015d39-dshm\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:56:17.364451 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:17.364397    5002 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c783e213-5fa1-4e80-9f2d-66b362015d39-tls-certs\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:56:17.364451 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:17.364408    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-59kx9\" (UniqueName: \"kubernetes.io/projected/c783e213-5fa1-4e80-9f2d-66b362015d39-kube-api-access-59kx9\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:56:17.606787 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:17.606749    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-d7a6db25/e2e-lifecycle-model-d7a6db25-kserve-556c6db9f7-nsdhk"]
Oct 30 00:56:17.620338 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:17.620267    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-d7a6db25/e2e-lifecycle-model-d7a6db25-kserve-556c6db9f7-nsdhk"]
Oct 30 00:56:18.498480 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:18.498447    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c783e213-5fa1-4e80-9f2d-66b362015d39" path="/var/lib/kubelet/pods/c783e213-5fa1-4e80-9f2d-66b362015d39/volumes"
Oct 30 00:56:22.842533 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:22.842490    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-02fbf5-6b759bfc8-7kvrs"]
Oct 30 00:56:22.867950 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:22.867915    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-02fbf5-764c8d7f95-qdnr5"]
Oct 30 00:56:27.843915 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:27.843834    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-02fbf5-6b759bfc8-7kvrs" podUID="87a023ea-b2a5-438d-b211-983374118e75" containerName="payload-processing" containerID="cri-o://f1e4eccc7c7314a783029ba7c62a0400cdebc2cad765d2884707d2ce0456e065" gracePeriod=25
Oct 30 00:56:27.868407 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:27.868338    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-02fbf5-764c8d7f95-qdnr5" podUID="1c5e0e1c-4526-4b7b-80fb-c65730081763" containerName="payload-pre-processing" containerID="cri-o://a48b62526972779f30269ac650ada11591fa7475aea4363d44f0491c63e93eb9" gracePeriod=25
Oct 30 00:56:28.268885 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:28.268528    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-02fbf5-6b759bfc8-7kvrs"
Oct 30 00:56:28.361073 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:28.354699    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87a023ea-b2a5-438d-b211-983374118e75-kube-api-access-tgp57\" (UniqueName: \"kubernetes.io/projected/87a023ea-b2a5-438d-b211-983374118e75-kube-api-access-tgp57\") pod \"87a023ea-b2a5-438d-b211-983374118e75\" (UID: \"87a023ea-b2a5-438d-b211-983374118e75\") "
Oct 30 00:56:28.361073 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:28.354762    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87a023ea-b2a5-438d-b211-983374118e75-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87a023ea-b2a5-438d-b211-983374118e75-plugins-config-volume\") pod \"87a023ea-b2a5-438d-b211-983374118e75\" (UID: \"87a023ea-b2a5-438d-b211-983374118e75\") "
Oct 30 00:56:28.361594 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:28.361532    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87a023ea-b2a5-438d-b211-983374118e75-plugins-config-volume" pod "87a023ea-b2a5-438d-b211-983374118e75" (UID: "87a023ea-b2a5-438d-b211-983374118e75"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:56:28.406830 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:28.406799    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87a023ea-b2a5-438d-b211-983374118e75-kube-api-access-tgp57" pod "87a023ea-b2a5-438d-b211-983374118e75" (UID: "87a023ea-b2a5-438d-b211-983374118e75"). InnerVolumeSpecName "kube-api-access-tgp57". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:56:28.494464 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:28.468758    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tgp57\" (UniqueName: \"kubernetes.io/projected/87a023ea-b2a5-438d-b211-983374118e75-kube-api-access-tgp57\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:56:28.494464 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:28.468798    5002 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87a023ea-b2a5-438d-b211-983374118e75-plugins-config-volume\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:56:28.600684 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:28.595239    5002 generic.go:356] "Generic (PLEG): container finished" podID="1c5e0e1c-4526-4b7b-80fb-c65730081763" containerID="a48b62526972779f30269ac650ada11591fa7475aea4363d44f0491c63e93eb9" exitCode=0
Oct 30 00:56:28.603908 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:28.603372    5002 generic.go:356] "Generic (PLEG): container finished" podID="87a023ea-b2a5-438d-b211-983374118e75" containerID="f1e4eccc7c7314a783029ba7c62a0400cdebc2cad765d2884707d2ce0456e065" exitCode=0
Oct 30 00:56:28.603908 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:28.603516    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-02fbf5-6b759bfc8-7kvrs"
Oct 30 00:56:28.626533 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:28.626478    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-02fbf5-764c8d7f95-qdnr5"
Oct 30 00:56:28.716242 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:28.716203    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-02fbf5-764c8d7f95-qdnr5" event={"ID":"1c5e0e1c-4526-4b7b-80fb-c65730081763","Type":"ContainerDied","Data":"a48b62526972779f30269ac650ada11591fa7475aea4363d44f0491c63e93eb9"}
Oct 30 00:56:28.716404 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:28.716274    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-02fbf5-764c8d7f95-qdnr5" event={"ID":"1c5e0e1c-4526-4b7b-80fb-c65730081763","Type":"ContainerDied","Data":"e8b037aade058da052efe2e96f6d3e8b538501ce313d95bdabaabdf4d90d2505"}
Oct 30 00:56:28.716404 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:28.716299    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-02fbf5-6b759bfc8-7kvrs" event={"ID":"87a023ea-b2a5-438d-b211-983374118e75","Type":"ContainerDied","Data":"f1e4eccc7c7314a783029ba7c62a0400cdebc2cad765d2884707d2ce0456e065"}
Oct 30 00:56:28.716404 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:28.716314    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-02fbf5-6b759bfc8-7kvrs" event={"ID":"87a023ea-b2a5-438d-b211-983374118e75","Type":"ContainerDied","Data":"fe32d6c434906f369be04944285863cf409d69efb2fd1f383e4c76aa81ab0092"}
Oct 30 00:56:28.716404 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:28.716336    5002 scope.go:122] "RemoveContainer" containerID="a48b62526972779f30269ac650ada11591fa7475aea4363d44f0491c63e93eb9"
Oct 30 00:56:28.731352 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:28.730873    5002 scope.go:122] "RemoveContainer" containerID="a48b62526972779f30269ac650ada11591fa7475aea4363d44f0491c63e93eb9"
Oct 30 00:56:28.731352 ip-10-0-128-7 kubenswrapper[5002]: E1030 00:56:28.731161    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a48b62526972779f30269ac650ada11591fa7475aea4363d44f0491c63e93eb9\": container with ID starting with a48b62526972779f30269ac650ada11591fa7475aea4363d44f0491c63e93eb9 not found: ID does not exist" containerID="a48b62526972779f30269ac650ada11591fa7475aea4363d44f0491c63e93eb9"
Oct 30 00:56:28.731352 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:28.731187    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a48b62526972779f30269ac650ada11591fa7475aea4363d44f0491c63e93eb9"} err="failed to get container status \"a48b62526972779f30269ac650ada11591fa7475aea4363d44f0491c63e93eb9\": rpc error: code = NotFound desc = could not find container \"a48b62526972779f30269ac650ada11591fa7475aea4363d44f0491c63e93eb9\": container with ID starting with a48b62526972779f30269ac650ada11591fa7475aea4363d44f0491c63e93eb9 not found: ID does not exist"
Oct 30 00:56:28.731352 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:28.731203    5002 scope.go:122] "RemoveContainer" containerID="f1e4eccc7c7314a783029ba7c62a0400cdebc2cad765d2884707d2ce0456e065"
Oct 30 00:56:28.731352 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:28.731256    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c5e0e1c-4526-4b7b-80fb-c65730081763-kube-api-access-cr82w\" (UniqueName: \"kubernetes.io/projected/1c5e0e1c-4526-4b7b-80fb-c65730081763-kube-api-access-cr82w\") pod \"1c5e0e1c-4526-4b7b-80fb-c65730081763\" (UID: \"1c5e0e1c-4526-4b7b-80fb-c65730081763\") "
Oct 30 00:56:28.731864 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:28.731373    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c5e0e1c-4526-4b7b-80fb-c65730081763-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c5e0e1c-4526-4b7b-80fb-c65730081763-plugins-config-volume\") pod \"1c5e0e1c-4526-4b7b-80fb-c65730081763\" (UID: \"1c5e0e1c-4526-4b7b-80fb-c65730081763\") "
Oct 30 00:56:28.732205 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:28.732183    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c5e0e1c-4526-4b7b-80fb-c65730081763-plugins-config-volume" pod "1c5e0e1c-4526-4b7b-80fb-c65730081763" (UID: "1c5e0e1c-4526-4b7b-80fb-c65730081763"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:56:28.733981 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:28.733952    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c5e0e1c-4526-4b7b-80fb-c65730081763-kube-api-access-cr82w" pod "1c5e0e1c-4526-4b7b-80fb-c65730081763" (UID: "1c5e0e1c-4526-4b7b-80fb-c65730081763"). InnerVolumeSpecName "kube-api-access-cr82w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:56:28.741095 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:28.741074    5002 scope.go:122] "RemoveContainer" containerID="f1e4eccc7c7314a783029ba7c62a0400cdebc2cad765d2884707d2ce0456e065"
Oct 30 00:56:28.747444 ip-10-0-128-7 kubenswrapper[5002]: E1030 00:56:28.741757    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1e4eccc7c7314a783029ba7c62a0400cdebc2cad765d2884707d2ce0456e065\": container with ID starting with f1e4eccc7c7314a783029ba7c62a0400cdebc2cad765d2884707d2ce0456e065 not found: ID does not exist" containerID="f1e4eccc7c7314a783029ba7c62a0400cdebc2cad765d2884707d2ce0456e065"
Oct 30 00:56:28.747444 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:28.741787    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1e4eccc7c7314a783029ba7c62a0400cdebc2cad765d2884707d2ce0456e065"} err="failed to get container status \"f1e4eccc7c7314a783029ba7c62a0400cdebc2cad765d2884707d2ce0456e065\": rpc error: code = NotFound desc = could not find container \"f1e4eccc7c7314a783029ba7c62a0400cdebc2cad765d2884707d2ce0456e065\": container with ID starting with f1e4eccc7c7314a783029ba7c62a0400cdebc2cad765d2884707d2ce0456e065 not found: ID does not exist"
Oct 30 00:56:28.761166 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:28.761109    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-02fbf5-6b759bfc8-7kvrs"]
Oct 30 00:56:28.768514 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:28.768487    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-02fbf5-6b759bfc8-7kvrs"]
Oct 30 00:56:28.832945 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:28.832914    5002 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c5e0e1c-4526-4b7b-80fb-c65730081763-plugins-config-volume\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:56:28.832945 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:28.832941    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cr82w\" (UniqueName: \"kubernetes.io/projected/1c5e0e1c-4526-4b7b-80fb-c65730081763-kube-api-access-cr82w\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:56:29.180809 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:29.180719    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bdc256-n7svj"]
Oct 30 00:56:29.188847 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:29.188815    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bdc256-n7svj"]
Oct 30 00:56:29.613485 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:29.613456    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-02fbf5-764c8d7f95-qdnr5"
Oct 30 00:56:29.673355 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:29.673312    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-02fbf5-764c8d7f95-qdnr5"]
Oct 30 00:56:29.687665 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:29.687621    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-02fbf5-764c8d7f95-qdnr5"]
Oct 30 00:56:30.495360 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:30.495327    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c5e0e1c-4526-4b7b-80fb-c65730081763" path="/var/lib/kubelet/pods/1c5e0e1c-4526-4b7b-80fb-c65730081763/volumes"
Oct 30 00:56:30.495986 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:30.495968    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87a023ea-b2a5-438d-b211-983374118e75" path="/var/lib/kubelet/pods/87a023ea-b2a5-438d-b211-983374118e75/volumes"
Oct 30 00:56:30.496421 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:30.496408    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9fa8adb3-1011-4865-93d9-27f1f187a327" path="/var/lib/kubelet/pods/9fa8adb3-1011-4865-93d9-27f1f187a327/volumes"
Oct 30 00:56:33.679178 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:33.679136    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-52785f5e-9fgf7"]
Oct 30 00:56:33.679809 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:33.679791    5002 cpu_manager.go:414] "containerMap: removing container" podUID="1c5e0e1c-4526-4b7b-80fb-c65730081763" containerName="payload-pre-processing"
Oct 30 00:56:33.679809 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:33.679809    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c5e0e1c-4526-4b7b-80fb-c65730081763" containerName="payload-pre-processing"
Oct 30 00:56:33.679936 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:33.679829    5002 cpu_manager.go:414] "containerMap: removing container" podUID="b1178ba7-6655-4ecd-b2f5-79477a445322" containerName="revoke-keys"
Oct 30 00:56:33.679936 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:33.679834    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1178ba7-6655-4ecd-b2f5-79477a445322" containerName="revoke-keys"
Oct 30 00:56:33.679936 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:33.679850    5002 cpu_manager.go:414] "containerMap: removing container" podUID="c783e213-5fa1-4e80-9f2d-66b362015d39" containerName="main"
Oct 30 00:56:33.679936 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:33.679855    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c783e213-5fa1-4e80-9f2d-66b362015d39" containerName="main"
Oct 30 00:56:33.679936 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:33.679864    5002 cpu_manager.go:414] "containerMap: removing container" podUID="87a023ea-b2a5-438d-b211-983374118e75" containerName="payload-processing"
Oct 30 00:56:33.679936 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:33.679869    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87a023ea-b2a5-438d-b211-983374118e75" containerName="payload-processing"
Oct 30 00:56:33.680241 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:33.679977    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c5e0e1c-4526-4b7b-80fb-c65730081763" containerName="payload-pre-processing"
Oct 30 00:56:33.680241 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:33.679989    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1178ba7-6655-4ecd-b2f5-79477a445322" containerName="revoke-keys"
Oct 30 00:56:33.680241 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:33.679997    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="c783e213-5fa1-4e80-9f2d-66b362015d39" containerName="main"
Oct 30 00:56:33.680241 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:33.680004    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="87a023ea-b2a5-438d-b211-983374118e75" containerName="payload-processing"
Oct 30 00:56:33.685085 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:33.685061    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-52785f5e-9fgf7"
Oct 30 00:56:33.702462 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:33.702433    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-52785f5e-9fgf7"]
Oct 30 00:56:33.711849 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:33.711819    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 00:56:33.712835 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:33.712557    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5phsh\""
Oct 30 00:56:33.712835 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:33.712611    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 00:56:33.715294 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:33.715268    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 00:56:33.815955 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:33.815915    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51a535ea-8c8a-46e1-a624-3d8f324e97fc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-52785f5e-9fgf7\" (UID: \"51a535ea-8c8a-46e1-a624-3d8f324e97fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-52785f5e-9fgf7"
Oct 30 00:56:33.816171 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:33.816003    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51a535ea-8c8a-46e1-a624-3d8f324e97fc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-52785f5e-9fgf7\" (UID: \"51a535ea-8c8a-46e1-a624-3d8f324e97fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-52785f5e-9fgf7"
Oct 30 00:56:33.816171 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:33.816113    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wbxn\" (UniqueName: \"kubernetes.io/projected/51a535ea-8c8a-46e1-a624-3d8f324e97fc-kube-api-access-8wbxn\") pod \"maas-api-revoke-keys-e2e-mt-52785f5e-9fgf7\" (UID: \"51a535ea-8c8a-46e1-a624-3d8f324e97fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-52785f5e-9fgf7"
Oct 30 00:56:33.917678 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:33.917633    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51a535ea-8c8a-46e1-a624-3d8f324e97fc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-52785f5e-9fgf7\" (UID: \"51a535ea-8c8a-46e1-a624-3d8f324e97fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-52785f5e-9fgf7"
Oct 30 00:56:33.917853 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:33.917700    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51a535ea-8c8a-46e1-a624-3d8f324e97fc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-52785f5e-9fgf7\" (UID: \"51a535ea-8c8a-46e1-a624-3d8f324e97fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-52785f5e-9fgf7"
Oct 30 00:56:33.917853 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:33.917778    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wbxn\" (UniqueName: \"kubernetes.io/projected/51a535ea-8c8a-46e1-a624-3d8f324e97fc-kube-api-access-8wbxn\") pod \"maas-api-revoke-keys-e2e-mt-52785f5e-9fgf7\" (UID: \"51a535ea-8c8a-46e1-a624-3d8f324e97fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-52785f5e-9fgf7"
Oct 30 00:56:33.918290 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:33.918270    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51a535ea-8c8a-46e1-a624-3d8f324e97fc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-52785f5e-9fgf7\" (UID: \"51a535ea-8c8a-46e1-a624-3d8f324e97fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-52785f5e-9fgf7"
Oct 30 00:56:33.920198 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:33.920175    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51a535ea-8c8a-46e1-a624-3d8f324e97fc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-52785f5e-9fgf7\" (UID: \"51a535ea-8c8a-46e1-a624-3d8f324e97fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-52785f5e-9fgf7"
Oct 30 00:56:33.942949 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:33.942920    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wbxn\" (UniqueName: \"kubernetes.io/projected/51a535ea-8c8a-46e1-a624-3d8f324e97fc-kube-api-access-8wbxn\") pod \"maas-api-revoke-keys-e2e-mt-52785f5e-9fgf7\" (UID: \"51a535ea-8c8a-46e1-a624-3d8f324e97fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-52785f5e-9fgf7"
Oct 30 00:56:34.000365 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:34.000325    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-52785f5e-9fgf7"
Oct 30 00:56:34.310639 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:56:34.310600    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51a535ea_8c8a_46e1_a624_3d8f324e97fc.slice/crio-4b49594c3182ad1c75834ccea459aacd7f940c9fcc66a6257be2aeef4cad4bca WatchSource:0}: Error finding container 4b49594c3182ad1c75834ccea459aacd7f940c9fcc66a6257be2aeef4cad4bca: Status 404 returned error can't find the container with id 4b49594c3182ad1c75834ccea459aacd7f940c9fcc66a6257be2aeef4cad4bca
Oct 30 00:56:34.311549 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:34.311527    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-52785f5e-9fgf7"]
Oct 30 00:56:34.652693 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:34.652652    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-52785f5e-9fgf7" event={"ID":"51a535ea-8c8a-46e1-a624-3d8f324e97fc","Type":"ContainerStarted","Data":"4b49594c3182ad1c75834ccea459aacd7f940c9fcc66a6257be2aeef4cad4bca"}
Oct 30 00:56:35.766402 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:35.766170    5002 generic.go:356] "Generic (PLEG): container finished" podID="51a535ea-8c8a-46e1-a624-3d8f324e97fc" containerID="12d3c7094425048a7b6c8bb7e91e4cc973c74ace818b2f54f358df9e5236b73b" exitCode=0
Oct 30 00:56:35.766402 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:35.766248    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-52785f5e-9fgf7" event={"ID":"51a535ea-8c8a-46e1-a624-3d8f324e97fc","Type":"ContainerDied","Data":"12d3c7094425048a7b6c8bb7e91e4cc973c74ace818b2f54f358df9e5236b73b"}
Oct 30 00:56:36.098274 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:36.098171    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-bdc256-openshift-default-5cf7d97495-76wgr"]
Oct 30 00:56:36.098564 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:36.098506    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-bdc256-openshift-default-5cf7d97495-76wgr" podUID="bcdd7ce4-a94f-4e6d-aebc-e7ca25d36b4a" containerName="istio-proxy" containerID="cri-o://4a6d6ab8e1b9dc36ab6c48798066ffe77ce17701c29662248408e0814f525233" gracePeriod=30
Oct 30 00:56:37.192403 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:37.190070    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-52785f5e-9fgf7"
Oct 30 00:56:37.363403 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:37.362973    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/51a535ea-8c8a-46e1-a624-3d8f324e97fc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51a535ea-8c8a-46e1-a624-3d8f324e97fc-maas-api-xks-ca\") pod \"51a535ea-8c8a-46e1-a624-3d8f324e97fc\" (UID: \"51a535ea-8c8a-46e1-a624-3d8f324e97fc\") "
Oct 30 00:56:37.363403 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:37.363212    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51a535ea-8c8a-46e1-a624-3d8f324e97fc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51a535ea-8c8a-46e1-a624-3d8f324e97fc-maas-api-openshift-ca\") pod \"51a535ea-8c8a-46e1-a624-3d8f324e97fc\" (UID: \"51a535ea-8c8a-46e1-a624-3d8f324e97fc\") "
Oct 30 00:56:37.363403 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:37.363295    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51a535ea-8c8a-46e1-a624-3d8f324e97fc-kube-api-access-8wbxn\" (UniqueName: \"kubernetes.io/projected/51a535ea-8c8a-46e1-a624-3d8f324e97fc-kube-api-access-8wbxn\") pod \"51a535ea-8c8a-46e1-a624-3d8f324e97fc\" (UID: \"51a535ea-8c8a-46e1-a624-3d8f324e97fc\") "
Oct 30 00:56:37.382680 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:37.372332    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51a535ea-8c8a-46e1-a624-3d8f324e97fc-maas-api-openshift-ca" pod "51a535ea-8c8a-46e1-a624-3d8f324e97fc" (UID: "51a535ea-8c8a-46e1-a624-3d8f324e97fc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:56:37.406465 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:37.393546    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51a535ea-8c8a-46e1-a624-3d8f324e97fc-kube-api-access-8wbxn" pod "51a535ea-8c8a-46e1-a624-3d8f324e97fc" (UID: "51a535ea-8c8a-46e1-a624-3d8f324e97fc"). InnerVolumeSpecName "kube-api-access-8wbxn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:56:37.469622 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:37.467766    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51a535ea-8c8a-46e1-a624-3d8f324e97fc-maas-api-xks-ca" pod "51a535ea-8c8a-46e1-a624-3d8f324e97fc" (UID: "51a535ea-8c8a-46e1-a624-3d8f324e97fc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:56:37.473643 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:37.473610    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51a535ea-8c8a-46e1-a624-3d8f324e97fc-maas-api-xks-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:56:37.473794 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:37.473652    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51a535ea-8c8a-46e1-a624-3d8f324e97fc-maas-api-openshift-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:56:37.473794 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:37.473670    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8wbxn\" (UniqueName: \"kubernetes.io/projected/51a535ea-8c8a-46e1-a624-3d8f324e97fc-kube-api-access-8wbxn\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:56:37.491039 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:37.491006    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-x8wgq"]
Oct 30 00:56:37.496434 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:37.496334    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-x8wgq" podUID="c534def0-b8f3-4be1-8f47-72bffc1234e1" containerName="istio-proxy" containerID="cri-o://1f4486a5b827e8159aab1d7ebd1bac12a4071091b2a303abb7a71ed51131269d" gracePeriod=30
Oct 30 00:56:37.908295 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:37.908207    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-52785f5e-9fgf7" event={"ID":"51a535ea-8c8a-46e1-a624-3d8f324e97fc","Type":"ContainerDied","Data":"4b49594c3182ad1c75834ccea459aacd7f940c9fcc66a6257be2aeef4cad4bca"}
Oct 30 00:56:37.908295 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:37.908285    5002 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4b49594c3182ad1c75834ccea459aacd7f940c9fcc66a6257be2aeef4cad4bca"
Oct 30 00:56:37.908295 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:37.908226    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-52785f5e-9fgf7"
Oct 30 00:56:41.157716 ip-10-0-128-7 kubenswrapper[5002]: E1030 00:56:41.157680    5002 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcdd7ce4_a94f_4e6d_aebc_e7ca25d36b4a.slice/crio-4a6d6ab8e1b9dc36ab6c48798066ffe77ce17701c29662248408e0814f525233.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 00:56:41.418139 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:41.418111    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-bdc256-openshift-default-5cf7d97495-76wgr"
Oct 30 00:56:41.515760 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:41.515593    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bcdd7ce4-a94f-4e6d-aebc-e7ca25d36b4a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bcdd7ce4-a94f-4e6d-aebc-e7ca25d36b4a-istiod-ca-cert\") pod \"bcdd7ce4-a94f-4e6d-aebc-e7ca25d36b4a\" (UID: \"bcdd7ce4-a94f-4e6d-aebc-e7ca25d36b4a\") "
Oct 30 00:56:41.516485 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:41.516460    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bcdd7ce4-a94f-4e6d-aebc-e7ca25d36b4a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcdd7ce4-a94f-4e6d-aebc-e7ca25d36b4a-workload-socket\") pod \"bcdd7ce4-a94f-4e6d-aebc-e7ca25d36b4a\" (UID: \"bcdd7ce4-a94f-4e6d-aebc-e7ca25d36b4a\") "
Oct 30 00:56:41.517027 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:41.517009    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bcdd7ce4-a94f-4e6d-aebc-e7ca25d36b4a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bcdd7ce4-a94f-4e6d-aebc-e7ca25d36b4a-istio-podinfo\") pod \"bcdd7ce4-a94f-4e6d-aebc-e7ca25d36b4a\" (UID: \"bcdd7ce4-a94f-4e6d-aebc-e7ca25d36b4a\") "
Oct 30 00:56:41.517577 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:41.517554    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bcdd7ce4-a94f-4e6d-aebc-e7ca25d36b4a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bcdd7ce4-a94f-4e6d-aebc-e7ca25d36b4a-workload-certs\") pod \"bcdd7ce4-a94f-4e6d-aebc-e7ca25d36b4a\" (UID: \"bcdd7ce4-a94f-4e6d-aebc-e7ca25d36b4a\") "
Oct 30 00:56:41.518012 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:41.517993    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bcdd7ce4-a94f-4e6d-aebc-e7ca25d36b4a-istio-token\" (UniqueName: \"kubernetes.io/projected/bcdd7ce4-a94f-4e6d-aebc-e7ca25d36b4a-istio-token\") pod \"bcdd7ce4-a94f-4e6d-aebc-e7ca25d36b4a\" (UID: \"bcdd7ce4-a94f-4e6d-aebc-e7ca25d36b4a\") "
Oct 30 00:56:41.518500 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:41.516361    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bcdd7ce4-a94f-4e6d-aebc-e7ca25d36b4a-istiod-ca-cert" pod "bcdd7ce4-a94f-4e6d-aebc-e7ca25d36b4a" (UID: "bcdd7ce4-a94f-4e6d-aebc-e7ca25d36b4a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:56:41.518634 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:41.516925    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcdd7ce4-a94f-4e6d-aebc-e7ca25d36b4a-workload-socket" pod "bcdd7ce4-a94f-4e6d-aebc-e7ca25d36b4a" (UID: "bcdd7ce4-a94f-4e6d-aebc-e7ca25d36b4a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:56:41.518755 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:41.517940    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcdd7ce4-a94f-4e6d-aebc-e7ca25d36b4a-workload-certs" pod "bcdd7ce4-a94f-4e6d-aebc-e7ca25d36b4a" (UID: "bcdd7ce4-a94f-4e6d-aebc-e7ca25d36b4a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:56:41.518890 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:41.518873    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bcdd7ce4-a94f-4e6d-aebc-e7ca25d36b4a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bcdd7ce4-a94f-4e6d-aebc-e7ca25d36b4a-istio-data\") pod \"bcdd7ce4-a94f-4e6d-aebc-e7ca25d36b4a\" (UID: \"bcdd7ce4-a94f-4e6d-aebc-e7ca25d36b4a\") "
Oct 30 00:56:41.519310 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:41.519205    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcdd7ce4-a94f-4e6d-aebc-e7ca25d36b4a-istio-data" pod "bcdd7ce4-a94f-4e6d-aebc-e7ca25d36b4a" (UID: "bcdd7ce4-a94f-4e6d-aebc-e7ca25d36b4a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:56:41.519477 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:41.519460    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bcdd7ce4-a94f-4e6d-aebc-e7ca25d36b4a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bcdd7ce4-a94f-4e6d-aebc-e7ca25d36b4a-istio-envoy\") pod \"bcdd7ce4-a94f-4e6d-aebc-e7ca25d36b4a\" (UID: \"bcdd7ce4-a94f-4e6d-aebc-e7ca25d36b4a\") "
Oct 30 00:56:41.519916 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:41.519898    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bcdd7ce4-a94f-4e6d-aebc-e7ca25d36b4a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcdd7ce4-a94f-4e6d-aebc-e7ca25d36b4a-credential-socket\") pod \"bcdd7ce4-a94f-4e6d-aebc-e7ca25d36b4a\" (UID: \"bcdd7ce4-a94f-4e6d-aebc-e7ca25d36b4a\") "
Oct 30 00:56:41.520053 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:41.520038    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bcdd7ce4-a94f-4e6d-aebc-e7ca25d36b4a-kube-api-access-z485d\" (UniqueName: \"kubernetes.io/projected/bcdd7ce4-a94f-4e6d-aebc-e7ca25d36b4a-kube-api-access-z485d\") pod \"bcdd7ce4-a94f-4e6d-aebc-e7ca25d36b4a\" (UID: \"bcdd7ce4-a94f-4e6d-aebc-e7ca25d36b4a\") "
Oct 30 00:56:41.520483 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:41.520457    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcdd7ce4-a94f-4e6d-aebc-e7ca25d36b4a-credential-socket" pod "bcdd7ce4-a94f-4e6d-aebc-e7ca25d36b4a" (UID: "bcdd7ce4-a94f-4e6d-aebc-e7ca25d36b4a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:56:41.521298 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:41.521205    5002 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bcdd7ce4-a94f-4e6d-aebc-e7ca25d36b4a-istiod-ca-cert\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:56:41.521451 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:41.521421    5002 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcdd7ce4-a94f-4e6d-aebc-e7ca25d36b4a-workload-socket\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:56:41.521451 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:41.521447    5002 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bcdd7ce4-a94f-4e6d-aebc-e7ca25d36b4a-workload-certs\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:56:41.521649 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:41.521461    5002 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bcdd7ce4-a94f-4e6d-aebc-e7ca25d36b4a-istio-data\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:56:41.521649 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:41.521475    5002 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcdd7ce4-a94f-4e6d-aebc-e7ca25d36b4a-credential-socket\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:56:41.536446 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:41.525745    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcdd7ce4-a94f-4e6d-aebc-e7ca25d36b4a-istio-envoy" pod "bcdd7ce4-a94f-4e6d-aebc-e7ca25d36b4a" (UID: "bcdd7ce4-a94f-4e6d-aebc-e7ca25d36b4a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:56:41.536446 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:41.525934    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcdd7ce4-a94f-4e6d-aebc-e7ca25d36b4a-istio-token" pod "bcdd7ce4-a94f-4e6d-aebc-e7ca25d36b4a" (UID: "bcdd7ce4-a94f-4e6d-aebc-e7ca25d36b4a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:56:41.536446 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:41.525976    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bcdd7ce4-a94f-4e6d-aebc-e7ca25d36b4a-istio-podinfo" pod "bcdd7ce4-a94f-4e6d-aebc-e7ca25d36b4a" (UID: "bcdd7ce4-a94f-4e6d-aebc-e7ca25d36b4a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 00:56:41.536446 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:41.526150    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcdd7ce4-a94f-4e6d-aebc-e7ca25d36b4a-kube-api-access-z485d" pod "bcdd7ce4-a94f-4e6d-aebc-e7ca25d36b4a" (UID: "bcdd7ce4-a94f-4e6d-aebc-e7ca25d36b4a"). InnerVolumeSpecName "kube-api-access-z485d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:56:41.623181 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:41.623144    5002 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bcdd7ce4-a94f-4e6d-aebc-e7ca25d36b4a-istio-podinfo\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:56:41.623181 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:41.623176    5002 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bcdd7ce4-a94f-4e6d-aebc-e7ca25d36b4a-istio-token\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:56:41.623181 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:41.623185    5002 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bcdd7ce4-a94f-4e6d-aebc-e7ca25d36b4a-istio-envoy\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:56:41.623432 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:41.623193    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z485d\" (UniqueName: \"kubernetes.io/projected/bcdd7ce4-a94f-4e6d-aebc-e7ca25d36b4a-kube-api-access-z485d\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:56:41.991169 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:41.991132    5002 generic.go:356] "Generic (PLEG): container finished" podID="bcdd7ce4-a94f-4e6d-aebc-e7ca25d36b4a" containerID="4a6d6ab8e1b9dc36ab6c48798066ffe77ce17701c29662248408e0814f525233" exitCode=0
Oct 30 00:56:41.991331 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:41.991217    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-bdc256-openshift-default-5cf7d97495-76wgr" event={"ID":"bcdd7ce4-a94f-4e6d-aebc-e7ca25d36b4a","Type":"ContainerDied","Data":"4a6d6ab8e1b9dc36ab6c48798066ffe77ce17701c29662248408e0814f525233"}
Oct 30 00:56:41.991331 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:41.991224    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-bdc256-openshift-default-5cf7d97495-76wgr"
Oct 30 00:56:41.991331 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:41.991323    5002 scope.go:122] "RemoveContainer" containerID="4a6d6ab8e1b9dc36ab6c48798066ffe77ce17701c29662248408e0814f525233"
Oct 30 00:56:41.991537 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:41.991311    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-bdc256-openshift-default-5cf7d97495-76wgr" event={"ID":"bcdd7ce4-a94f-4e6d-aebc-e7ca25d36b4a","Type":"ContainerDied","Data":"4799dafcdd2933c39d8e11a9c6ef66a63339e8a279419e41de7af5265655a357"}
Oct 30 00:56:42.003202 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:42.003185    5002 scope.go:122] "RemoveContainer" containerID="8e1a6f5e4f08a1f82b240d45bd29abbd9c2567543a4205deb79fd92c69f4714c"
Oct 30 00:56:42.014690 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:42.014666    5002 scope.go:122] "RemoveContainer" containerID="4a6d6ab8e1b9dc36ab6c48798066ffe77ce17701c29662248408e0814f525233"
Oct 30 00:56:42.014943 ip-10-0-128-7 kubenswrapper[5002]: E1030 00:56:42.014918    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a6d6ab8e1b9dc36ab6c48798066ffe77ce17701c29662248408e0814f525233\": container with ID starting with 4a6d6ab8e1b9dc36ab6c48798066ffe77ce17701c29662248408e0814f525233 not found: ID does not exist" containerID="4a6d6ab8e1b9dc36ab6c48798066ffe77ce17701c29662248408e0814f525233"
Oct 30 00:56:42.015025 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:42.014956    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a6d6ab8e1b9dc36ab6c48798066ffe77ce17701c29662248408e0814f525233"} err="failed to get container status \"4a6d6ab8e1b9dc36ab6c48798066ffe77ce17701c29662248408e0814f525233\": rpc error: code = NotFound desc = could not find container \"4a6d6ab8e1b9dc36ab6c48798066ffe77ce17701c29662248408e0814f525233\": container with ID starting with 4a6d6ab8e1b9dc36ab6c48798066ffe77ce17701c29662248408e0814f525233 not found: ID does not exist"
Oct 30 00:56:42.015025 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:42.014979    5002 scope.go:122] "RemoveContainer" containerID="8e1a6f5e4f08a1f82b240d45bd29abbd9c2567543a4205deb79fd92c69f4714c"
Oct 30 00:56:42.015242 ip-10-0-128-7 kubenswrapper[5002]: E1030 00:56:42.015224    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e1a6f5e4f08a1f82b240d45bd29abbd9c2567543a4205deb79fd92c69f4714c\": container with ID starting with 8e1a6f5e4f08a1f82b240d45bd29abbd9c2567543a4205deb79fd92c69f4714c not found: ID does not exist" containerID="8e1a6f5e4f08a1f82b240d45bd29abbd9c2567543a4205deb79fd92c69f4714c"
Oct 30 00:56:42.015277 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:42.015249    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e1a6f5e4f08a1f82b240d45bd29abbd9c2567543a4205deb79fd92c69f4714c"} err="failed to get container status \"8e1a6f5e4f08a1f82b240d45bd29abbd9c2567543a4205deb79fd92c69f4714c\": rpc error: code = NotFound desc = could not find container \"8e1a6f5e4f08a1f82b240d45bd29abbd9c2567543a4205deb79fd92c69f4714c\": container with ID starting with 8e1a6f5e4f08a1f82b240d45bd29abbd9c2567543a4205deb79fd92c69f4714c not found: ID does not exist"
Oct 30 00:56:42.054315 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:42.054281    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-bdc256-openshift-default-5cf7d97495-76wgr"]
Oct 30 00:56:42.066583 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:42.066550    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-bdc256-openshift-default-5cf7d97495-76wgr"]
Oct 30 00:56:42.497178 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:42.497142    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bcdd7ce4-a94f-4e6d-aebc-e7ca25d36b4a" path="/var/lib/kubelet/pods/bcdd7ce4-a94f-4e6d-aebc-e7ca25d36b4a/volumes"
Oct 30 00:56:42.920598 ip-10-0-128-7 kubenswrapper[5002]: E1030 00:56:42.920568    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbjdq" containerName="extractor"
Oct 30 00:56:43.009135 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:43.009094    5002 generic.go:356] "Generic (PLEG): container finished" podID="c534def0-b8f3-4be1-8f47-72bffc1234e1" containerID="1f4486a5b827e8159aab1d7ebd1bac12a4071091b2a303abb7a71ed51131269d" exitCode=0
Oct 30 00:56:43.009310 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:43.009151    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-x8wgq" event={"ID":"c534def0-b8f3-4be1-8f47-72bffc1234e1","Type":"ContainerDied","Data":"1f4486a5b827e8159aab1d7ebd1bac12a4071091b2a303abb7a71ed51131269d"}
Oct 30 00:56:43.084575 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:43.084546    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-x8wgq"
Oct 30 00:56:43.120046 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:43.120012    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c534def0-b8f3-4be1-8f47-72bffc1234e1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c534def0-b8f3-4be1-8f47-72bffc1234e1-workload-socket\") pod \"c534def0-b8f3-4be1-8f47-72bffc1234e1\" (UID: \"c534def0-b8f3-4be1-8f47-72bffc1234e1\") "
Oct 30 00:56:43.120199 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:43.120062    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c534def0-b8f3-4be1-8f47-72bffc1234e1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c534def0-b8f3-4be1-8f47-72bffc1234e1-credential-socket\") pod \"c534def0-b8f3-4be1-8f47-72bffc1234e1\" (UID: \"c534def0-b8f3-4be1-8f47-72bffc1234e1\") "
Oct 30 00:56:43.120199 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:43.120089    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c534def0-b8f3-4be1-8f47-72bffc1234e1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c534def0-b8f3-4be1-8f47-72bffc1234e1-istio-data\") pod \"c534def0-b8f3-4be1-8f47-72bffc1234e1\" (UID: \"c534def0-b8f3-4be1-8f47-72bffc1234e1\") "
Oct 30 00:56:43.120199 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:43.120119    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c534def0-b8f3-4be1-8f47-72bffc1234e1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c534def0-b8f3-4be1-8f47-72bffc1234e1-istio-podinfo\") pod \"c534def0-b8f3-4be1-8f47-72bffc1234e1\" (UID: \"c534def0-b8f3-4be1-8f47-72bffc1234e1\") "
Oct 30 00:56:43.120391 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:43.120229    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c534def0-b8f3-4be1-8f47-72bffc1234e1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c534def0-b8f3-4be1-8f47-72bffc1234e1-istio-envoy\") pod \"c534def0-b8f3-4be1-8f47-72bffc1234e1\" (UID: \"c534def0-b8f3-4be1-8f47-72bffc1234e1\") "
Oct 30 00:56:43.120391 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:43.120350    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c534def0-b8f3-4be1-8f47-72bffc1234e1-istio-token\" (UniqueName: \"kubernetes.io/projected/c534def0-b8f3-4be1-8f47-72bffc1234e1-istio-token\") pod \"c534def0-b8f3-4be1-8f47-72bffc1234e1\" (UID: \"c534def0-b8f3-4be1-8f47-72bffc1234e1\") "
Oct 30 00:56:43.120506 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:43.120391    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c534def0-b8f3-4be1-8f47-72bffc1234e1-workload-socket" pod "c534def0-b8f3-4be1-8f47-72bffc1234e1" (UID: "c534def0-b8f3-4be1-8f47-72bffc1234e1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:56:43.120506 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:43.120404    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c534def0-b8f3-4be1-8f47-72bffc1234e1-credential-socket" pod "c534def0-b8f3-4be1-8f47-72bffc1234e1" (UID: "c534def0-b8f3-4be1-8f47-72bffc1234e1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:56:43.120506 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:43.120424    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c534def0-b8f3-4be1-8f47-72bffc1234e1-istio-data" pod "c534def0-b8f3-4be1-8f47-72bffc1234e1" (UID: "c534def0-b8f3-4be1-8f47-72bffc1234e1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:56:43.120506 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:43.120467    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c534def0-b8f3-4be1-8f47-72bffc1234e1-kube-api-access-fgxbc\" (UniqueName: \"kubernetes.io/projected/c534def0-b8f3-4be1-8f47-72bffc1234e1-kube-api-access-fgxbc\") pod \"c534def0-b8f3-4be1-8f47-72bffc1234e1\" (UID: \"c534def0-b8f3-4be1-8f47-72bffc1234e1\") "
Oct 30 00:56:43.120699 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:43.120578    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c534def0-b8f3-4be1-8f47-72bffc1234e1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c534def0-b8f3-4be1-8f47-72bffc1234e1-istiod-ca-cert\") pod \"c534def0-b8f3-4be1-8f47-72bffc1234e1\" (UID: \"c534def0-b8f3-4be1-8f47-72bffc1234e1\") "
Oct 30 00:56:43.120699 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:43.120631    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c534def0-b8f3-4be1-8f47-72bffc1234e1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c534def0-b8f3-4be1-8f47-72bffc1234e1-workload-certs\") pod \"c534def0-b8f3-4be1-8f47-72bffc1234e1\" (UID: \"c534def0-b8f3-4be1-8f47-72bffc1234e1\") "
Oct 30 00:56:43.120941 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:43.120914    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c534def0-b8f3-4be1-8f47-72bffc1234e1-workload-certs" pod "c534def0-b8f3-4be1-8f47-72bffc1234e1" (UID: "c534def0-b8f3-4be1-8f47-72bffc1234e1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:56:43.121049 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:43.120940    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c534def0-b8f3-4be1-8f47-72bffc1234e1-istiod-ca-cert" pod "c534def0-b8f3-4be1-8f47-72bffc1234e1" (UID: "c534def0-b8f3-4be1-8f47-72bffc1234e1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:56:43.121418 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:43.121365    5002 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c534def0-b8f3-4be1-8f47-72bffc1234e1-istiod-ca-cert\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:56:43.121418 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:43.121412    5002 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c534def0-b8f3-4be1-8f47-72bffc1234e1-workload-certs\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:56:43.121571 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:43.121427    5002 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c534def0-b8f3-4be1-8f47-72bffc1234e1-workload-socket\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:56:43.121571 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:43.121440    5002 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c534def0-b8f3-4be1-8f47-72bffc1234e1-credential-socket\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:56:43.121571 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:43.121451    5002 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c534def0-b8f3-4be1-8f47-72bffc1234e1-istio-data\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:56:43.123795 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:43.123749    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c534def0-b8f3-4be1-8f47-72bffc1234e1-istio-podinfo" pod "c534def0-b8f3-4be1-8f47-72bffc1234e1" (UID: "c534def0-b8f3-4be1-8f47-72bffc1234e1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 00:56:43.124151 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:43.124125    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c534def0-b8f3-4be1-8f47-72bffc1234e1-kube-api-access-fgxbc" pod "c534def0-b8f3-4be1-8f47-72bffc1234e1" (UID: "c534def0-b8f3-4be1-8f47-72bffc1234e1"). InnerVolumeSpecName "kube-api-access-fgxbc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:56:43.124245 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:43.124166    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c534def0-b8f3-4be1-8f47-72bffc1234e1-istio-token" pod "c534def0-b8f3-4be1-8f47-72bffc1234e1" (UID: "c534def0-b8f3-4be1-8f47-72bffc1234e1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:56:43.124245 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:43.124187    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c534def0-b8f3-4be1-8f47-72bffc1234e1-istio-envoy" pod "c534def0-b8f3-4be1-8f47-72bffc1234e1" (UID: "c534def0-b8f3-4be1-8f47-72bffc1234e1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:56:43.222646 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:43.222611    5002 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c534def0-b8f3-4be1-8f47-72bffc1234e1-istio-podinfo\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:56:43.222646 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:43.222643    5002 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c534def0-b8f3-4be1-8f47-72bffc1234e1-istio-envoy\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:56:43.222646 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:43.222651    5002 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c534def0-b8f3-4be1-8f47-72bffc1234e1-istio-token\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:56:43.222646 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:43.222659    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fgxbc\" (UniqueName: \"kubernetes.io/projected/c534def0-b8f3-4be1-8f47-72bffc1234e1-kube-api-access-fgxbc\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:56:44.017522 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:44.017491    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-x8wgq"
Oct 30 00:56:44.018104 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:44.017534    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-x8wgq" event={"ID":"c534def0-b8f3-4be1-8f47-72bffc1234e1","Type":"ContainerDied","Data":"e0317a5d45bc42c47adbc0b2b854d0ce5d28c912eaa770656245b1acd14ed083"}
Oct 30 00:56:44.018104 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:44.017608    5002 scope.go:122] "RemoveContainer" containerID="1f4486a5b827e8159aab1d7ebd1bac12a4071091b2a303abb7a71ed51131269d"
Oct 30 00:56:44.135649 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:44.135612    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-x8wgq"]
Oct 30 00:56:44.144901 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:44.144865    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-x8wgq"]
Oct 30 00:56:44.318123 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:44.317983    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-8dbe73-788cc7d8c9-qnxcz"]
Oct 30 00:56:44.324430 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:44.321017    5002 cpu_manager.go:414] "containerMap: removing container" podUID="c534def0-b8f3-4be1-8f47-72bffc1234e1" containerName="istio-proxy"
Oct 30 00:56:44.324430 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:44.321047    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c534def0-b8f3-4be1-8f47-72bffc1234e1" containerName="istio-proxy"
Oct 30 00:56:44.324430 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:44.321080    5002 cpu_manager.go:414] "containerMap: removing container" podUID="bcdd7ce4-a94f-4e6d-aebc-e7ca25d36b4a" containerName="istio-proxy"
Oct 30 00:56:44.324430 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:44.321089    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bcdd7ce4-a94f-4e6d-aebc-e7ca25d36b4a" containerName="istio-proxy"
Oct 30 00:56:44.324430 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:44.321122    5002 cpu_manager.go:414] "containerMap: removing container" podUID="bcdd7ce4-a94f-4e6d-aebc-e7ca25d36b4a" containerName="istio-proxy"
Oct 30 00:56:44.324430 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:44.321131    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bcdd7ce4-a94f-4e6d-aebc-e7ca25d36b4a" containerName="istio-proxy"
Oct 30 00:56:44.324430 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:44.321169    5002 cpu_manager.go:414] "containerMap: removing container" podUID="51a535ea-8c8a-46e1-a624-3d8f324e97fc" containerName="revoke-keys"
Oct 30 00:56:44.324430 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:44.321179    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51a535ea-8c8a-46e1-a624-3d8f324e97fc" containerName="revoke-keys"
Oct 30 00:56:44.324430 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:44.321364    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="c534def0-b8f3-4be1-8f47-72bffc1234e1" containerName="istio-proxy"
Oct 30 00:56:44.324430 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:44.321395    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="bcdd7ce4-a94f-4e6d-aebc-e7ca25d36b4a" containerName="istio-proxy"
Oct 30 00:56:44.324430 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:44.321410    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="bcdd7ce4-a94f-4e6d-aebc-e7ca25d36b4a" containerName="istio-proxy"
Oct 30 00:56:44.324430 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:44.321423    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="51a535ea-8c8a-46e1-a624-3d8f324e97fc" containerName="revoke-keys"
Oct 30 00:56:44.329423 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:44.329351    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8dbe73-788cc7d8c9-qnxcz"
Oct 30 00:56:44.339015 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:44.338257    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-8dbe73-788cc7d8c9-qnxcz"]
Oct 30 00:56:44.347638 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:44.347612    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-8dbe73\""
Oct 30 00:56:44.397361 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:44.397329    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-8dbe73-dockercfg-8kwrd\""
Oct 30 00:56:44.416491 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:44.416446    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-8dbe73-698f74c77d-l7tpx"]
Oct 30 00:56:44.480439 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:44.458954    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlkk9\" (UniqueName: \"kubernetes.io/projected/4521f1a9-13c3-409b-b0f3-d636722108a5-kube-api-access-zlkk9\") pod \"payload-pre-processing-e2e-inf-a-8dbe73-788cc7d8c9-qnxcz\" (UID: \"4521f1a9-13c3-409b-b0f3-d636722108a5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8dbe73-788cc7d8c9-qnxcz"
Oct 30 00:56:44.480439 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:44.468621    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4521f1a9-13c3-409b-b0f3-d636722108a5-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-8dbe73-788cc7d8c9-qnxcz\" (UID: \"4521f1a9-13c3-409b-b0f3-d636722108a5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8dbe73-788cc7d8c9-qnxcz"
Oct 30 00:56:44.568889 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:44.568782    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-8dbe73-698f74c77d-l7tpx"
Oct 30 00:56:44.571229 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:44.570281    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4521f1a9-13c3-409b-b0f3-d636722108a5-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-8dbe73-788cc7d8c9-qnxcz\" (UID: \"4521f1a9-13c3-409b-b0f3-d636722108a5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8dbe73-788cc7d8c9-qnxcz"
Oct 30 00:56:44.575256 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:44.575219    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlkk9\" (UniqueName: \"kubernetes.io/projected/4521f1a9-13c3-409b-b0f3-d636722108a5-kube-api-access-zlkk9\") pod \"payload-pre-processing-e2e-inf-a-8dbe73-788cc7d8c9-qnxcz\" (UID: \"4521f1a9-13c3-409b-b0f3-d636722108a5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8dbe73-788cc7d8c9-qnxcz"
Oct 30 00:56:44.606369 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:44.606338    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-8dbe73\""
Oct 30 00:56:44.619823 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:44.614680    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4521f1a9-13c3-409b-b0f3-d636722108a5-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-8dbe73-788cc7d8c9-qnxcz\" (UID: \"4521f1a9-13c3-409b-b0f3-d636722108a5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8dbe73-788cc7d8c9-qnxcz"
Oct 30 00:56:44.651846 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:44.651455    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c534def0-b8f3-4be1-8f47-72bffc1234e1" path="/var/lib/kubelet/pods/c534def0-b8f3-4be1-8f47-72bffc1234e1/volumes"
Oct 30 00:56:44.678774 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:44.677756    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd1c2b8c-8b70-4583-bdec-d1901c0bd640-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-8dbe73-698f74c77d-l7tpx\" (UID: \"dd1c2b8c-8b70-4583-bdec-d1901c0bd640\") " pod="openshift-ingress/payload-processing-e2e-inf-a-8dbe73-698f74c77d-l7tpx"
Oct 30 00:56:44.678774 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:44.677980    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fg2tp\" (UniqueName: \"kubernetes.io/projected/dd1c2b8c-8b70-4583-bdec-d1901c0bd640-kube-api-access-fg2tp\") pod \"payload-processing-e2e-inf-a-8dbe73-698f74c77d-l7tpx\" (UID: \"dd1c2b8c-8b70-4583-bdec-d1901c0bd640\") " pod="openshift-ingress/payload-processing-e2e-inf-a-8dbe73-698f74c77d-l7tpx"
Oct 30 00:56:44.686083 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:44.686053    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-8dbe73-698f74c77d-l7tpx"]
Oct 30 00:56:44.731460 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:44.730924    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlkk9\" (UniqueName: \"kubernetes.io/projected/4521f1a9-13c3-409b-b0f3-d636722108a5-kube-api-access-zlkk9\") pod \"payload-pre-processing-e2e-inf-a-8dbe73-788cc7d8c9-qnxcz\" (UID: \"4521f1a9-13c3-409b-b0f3-d636722108a5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8dbe73-788cc7d8c9-qnxcz"
Oct 30 00:56:44.778884 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:44.778854    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd1c2b8c-8b70-4583-bdec-d1901c0bd640-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-8dbe73-698f74c77d-l7tpx\" (UID: \"dd1c2b8c-8b70-4583-bdec-d1901c0bd640\") " pod="openshift-ingress/payload-processing-e2e-inf-a-8dbe73-698f74c77d-l7tpx"
Oct 30 00:56:44.779500 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:44.779481    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fg2tp\" (UniqueName: \"kubernetes.io/projected/dd1c2b8c-8b70-4583-bdec-d1901c0bd640-kube-api-access-fg2tp\") pod \"payload-processing-e2e-inf-a-8dbe73-698f74c77d-l7tpx\" (UID: \"dd1c2b8c-8b70-4583-bdec-d1901c0bd640\") " pod="openshift-ingress/payload-processing-e2e-inf-a-8dbe73-698f74c77d-l7tpx"
Oct 30 00:56:44.780231 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:44.780206    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd1c2b8c-8b70-4583-bdec-d1901c0bd640-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-8dbe73-698f74c77d-l7tpx\" (UID: \"dd1c2b8c-8b70-4583-bdec-d1901c0bd640\") " pod="openshift-ingress/payload-processing-e2e-inf-a-8dbe73-698f74c77d-l7tpx"
Oct 30 00:56:44.793890 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:44.793868    5002 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-g7c2w_5dc56e7e-f3c1-43eb-a21b-8be6bb4706cf/console-operator/1.log"
Oct 30 00:56:44.800624 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:44.800602    5002 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-g7c2w_5dc56e7e-f3c1-43eb-a21b-8be6bb4706cf/console-operator/1.log"
Oct 30 00:56:44.805045 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:44.805026    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fg2tp\" (UniqueName: \"kubernetes.io/projected/dd1c2b8c-8b70-4583-bdec-d1901c0bd640-kube-api-access-fg2tp\") pod \"payload-processing-e2e-inf-a-8dbe73-698f74c77d-l7tpx\" (UID: \"dd1c2b8c-8b70-4583-bdec-d1901c0bd640\") " pod="openshift-ingress/payload-processing-e2e-inf-a-8dbe73-698f74c77d-l7tpx"
Oct 30 00:56:44.963858 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:44.963824    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-8dbe73-dockercfg-8kwrd\""
Oct 30 00:56:44.969234 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:44.969207    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8dbe73-788cc7d8c9-qnxcz"
Oct 30 00:56:45.059398 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:45.057627    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-8dbe73-698f74c77d-l7tpx"
Oct 30 00:56:45.289815 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:56:45.289782    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4521f1a9_13c3_409b_b0f3_d636722108a5.slice/crio-bf238e6eb8f275f4f6e6cf5094c885e61d6e59d9c09aeddd631c71ffef20ea4d WatchSource:0}: Error finding container bf238e6eb8f275f4f6e6cf5094c885e61d6e59d9c09aeddd631c71ffef20ea4d: Status 404 returned error can't find the container with id bf238e6eb8f275f4f6e6cf5094c885e61d6e59d9c09aeddd631c71ffef20ea4d
Oct 30 00:56:45.292810 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:45.292788    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-8dbe73-788cc7d8c9-qnxcz"]
Oct 30 00:56:45.394481 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:56:45.394441    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd1c2b8c_8b70_4583_bdec_d1901c0bd640.slice/crio-b94b9099aac02494ab45dda93600ada102d2e788743824f8dfd4cf32a46f0919 WatchSource:0}: Error finding container b94b9099aac02494ab45dda93600ada102d2e788743824f8dfd4cf32a46f0919: Status 404 returned error can't find the container with id b94b9099aac02494ab45dda93600ada102d2e788743824f8dfd4cf32a46f0919
Oct 30 00:56:45.396761 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:45.396730    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-8dbe73-698f74c77d-l7tpx"]
Oct 30 00:56:46.112402 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:46.103980    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-8dbe73-698f74c77d-l7tpx" event={"ID":"dd1c2b8c-8b70-4583-bdec-d1901c0bd640","Type":"ContainerStarted","Data":"a21e598f815033054af077d05ebe0bea9f35846b54db470069f55dece4d9daa4"}
Oct 30 00:56:46.112402 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:46.104049    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-8dbe73-698f74c77d-l7tpx" event={"ID":"dd1c2b8c-8b70-4583-bdec-d1901c0bd640","Type":"ContainerStarted","Data":"b94b9099aac02494ab45dda93600ada102d2e788743824f8dfd4cf32a46f0919"}
Oct 30 00:56:46.112402 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:46.105208    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-8dbe73-698f74c77d-l7tpx"
Oct 30 00:56:46.141405 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:46.130997    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8dbe73-788cc7d8c9-qnxcz" event={"ID":"4521f1a9-13c3-409b-b0f3-d636722108a5","Type":"ContainerStarted","Data":"40402337a9a3a2dd5bcc1186f43b22c64efbb7956c12a4eddbfc2a1381888920"}
Oct 30 00:56:46.141405 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:46.131062    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8dbe73-788cc7d8c9-qnxcz" event={"ID":"4521f1a9-13c3-409b-b0f3-d636722108a5","Type":"ContainerStarted","Data":"bf238e6eb8f275f4f6e6cf5094c885e61d6e59d9c09aeddd631c71ffef20ea4d"}
Oct 30 00:56:46.141405 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:46.131944    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8dbe73-788cc7d8c9-qnxcz"
Oct 30 00:56:46.231578 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:46.226237    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-8dbe73-698f74c77d-l7tpx" podStartSLOduration=2.226215366 podStartE2EDuration="2.226215366s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:56:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:56:46.169691606 +0000 UTC m=+1501.786124507" watchObservedRunningTime="2026-10-30 00:56:46.226215366 +0000 UTC m=+1501.842648247"
Oct 30 00:56:46.245448 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:46.243762    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8dbe73-788cc7d8c9-qnxcz" podStartSLOduration=2.243746097 podStartE2EDuration="2.243746097s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:56:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:56:46.223773161 +0000 UTC m=+1501.840206061" watchObservedRunningTime="2026-10-30 00:56:46.243746097 +0000 UTC m=+1501.860178980"
Oct 30 00:56:46.435785 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:46.435754    5002 scope.go:122] "RemoveContainer" containerID="b7c5ed48c9a8360bc67c944348411ec45c2747e6629e5a2e7ca6a0417e81d457"
Oct 30 00:56:47.887550 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:47.887508    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-d7a6db25-openshift-default-68896bcdc7-pgwjg"]
Oct 30 00:56:47.888051 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:47.887766    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-d7a6db25-openshift-default-68896bcdc7-pgwjg" podUID="fabdd87d-54a8-4fae-adf9-f2e23ff299cf" containerName="istio-proxy" containerID="cri-o://b58568d8c76e3febbab0fa823c357c99c840fbb59378f57b8f45eeb3a827274b" gracePeriod=30
Oct 30 00:56:49.593713 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:49.593677    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4d83951-wxdv6"]
Oct 30 00:56:49.601920 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:49.599912    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4d83951-wxdv6"
Oct 30 00:56:49.618274 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:49.618242    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4d83951-wxdv6"]
Oct 30 00:56:49.628398 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:49.627898    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 00:56:49.628398 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:49.628153    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 00:56:49.662792 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:49.662750    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5phsh\""
Oct 30 00:56:49.662987 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:49.662839    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 00:56:49.679906 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:49.679861    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4n9w\" (UniqueName: \"kubernetes.io/projected/25e53f18-a85a-4601-874d-e58d0704f6df-kube-api-access-m4n9w\") pod \"maas-api-revoke-sub-d4d83951-wxdv6\" (UID: \"25e53f18-a85a-4601-874d-e58d0704f6df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4d83951-wxdv6"
Oct 30 00:56:49.680183 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:49.680155    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25e53f18-a85a-4601-874d-e58d0704f6df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d4d83951-wxdv6\" (UID: \"25e53f18-a85a-4601-874d-e58d0704f6df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4d83951-wxdv6"
Oct 30 00:56:49.680271 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:49.680252    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25e53f18-a85a-4601-874d-e58d0704f6df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d4d83951-wxdv6\" (UID: \"25e53f18-a85a-4601-874d-e58d0704f6df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4d83951-wxdv6"
Oct 30 00:56:49.781652 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:49.781620    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25e53f18-a85a-4601-874d-e58d0704f6df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d4d83951-wxdv6\" (UID: \"25e53f18-a85a-4601-874d-e58d0704f6df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4d83951-wxdv6"
Oct 30 00:56:49.781817 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:49.781750    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4n9w\" (UniqueName: \"kubernetes.io/projected/25e53f18-a85a-4601-874d-e58d0704f6df-kube-api-access-m4n9w\") pod \"maas-api-revoke-sub-d4d83951-wxdv6\" (UID: \"25e53f18-a85a-4601-874d-e58d0704f6df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4d83951-wxdv6"
Oct 30 00:56:49.781966 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:49.781951    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25e53f18-a85a-4601-874d-e58d0704f6df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d4d83951-wxdv6\" (UID: \"25e53f18-a85a-4601-874d-e58d0704f6df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4d83951-wxdv6"
Oct 30 00:56:49.782646 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:49.782619    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25e53f18-a85a-4601-874d-e58d0704f6df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d4d83951-wxdv6\" (UID: \"25e53f18-a85a-4601-874d-e58d0704f6df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4d83951-wxdv6"
Oct 30 00:56:49.784178 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:49.784154    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25e53f18-a85a-4601-874d-e58d0704f6df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d4d83951-wxdv6\" (UID: \"25e53f18-a85a-4601-874d-e58d0704f6df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4d83951-wxdv6"
Oct 30 00:56:49.806518 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:49.806490    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4n9w\" (UniqueName: \"kubernetes.io/projected/25e53f18-a85a-4601-874d-e58d0704f6df-kube-api-access-m4n9w\") pod \"maas-api-revoke-sub-d4d83951-wxdv6\" (UID: \"25e53f18-a85a-4601-874d-e58d0704f6df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4d83951-wxdv6"
Oct 30 00:56:49.919773 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:49.919696    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4d83951-wxdv6"
Oct 30 00:56:50.232449 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:56:50.232418    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25e53f18_a85a_4601_874d_e58d0704f6df.slice/crio-1b047991badbc5d6366cf16b12a7d5b07da718d09bfe8e13a8192c58ae3775df WatchSource:0}: Error finding container 1b047991badbc5d6366cf16b12a7d5b07da718d09bfe8e13a8192c58ae3775df: Status 404 returned error can't find the container with id 1b047991badbc5d6366cf16b12a7d5b07da718d09bfe8e13a8192c58ae3775df
Oct 30 00:56:50.233828 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:50.233801    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4d83951-wxdv6"]
Oct 30 00:56:51.220873 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:51.220835    5002 generic.go:356] "Generic (PLEG): container finished" podID="25e53f18-a85a-4601-874d-e58d0704f6df" containerID="924f68c091f4b8ac7157a073873b7ba1d194ba0e4eb23252c60a9ff8c07d5fc9" exitCode=0
Oct 30 00:56:51.221301 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:51.220979    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4d83951-wxdv6" event={"ID":"25e53f18-a85a-4601-874d-e58d0704f6df","Type":"ContainerDied","Data":"924f68c091f4b8ac7157a073873b7ba1d194ba0e4eb23252c60a9ff8c07d5fc9"}
Oct 30 00:56:51.221301 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:51.221068    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4d83951-wxdv6" event={"ID":"25e53f18-a85a-4601-874d-e58d0704f6df","Type":"ContainerStarted","Data":"1b047991badbc5d6366cf16b12a7d5b07da718d09bfe8e13a8192c58ae3775df"}
Oct 30 00:56:52.441534 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:52.441501    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4d83951-wxdv6"
Oct 30 00:56:52.528988 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:52.528904    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/25e53f18-a85a-4601-874d-e58d0704f6df-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25e53f18-a85a-4601-874d-e58d0704f6df-maas-api-xks-ca\") pod \"25e53f18-a85a-4601-874d-e58d0704f6df\" (UID: \"25e53f18-a85a-4601-874d-e58d0704f6df\") "
Oct 30 00:56:52.529267 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:52.529242    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25e53f18-a85a-4601-874d-e58d0704f6df-kube-api-access-m4n9w\" (UniqueName: \"kubernetes.io/projected/25e53f18-a85a-4601-874d-e58d0704f6df-kube-api-access-m4n9w\") pod \"25e53f18-a85a-4601-874d-e58d0704f6df\" (UID: \"25e53f18-a85a-4601-874d-e58d0704f6df\") "
Oct 30 00:56:52.529413 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:52.529318    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25e53f18-a85a-4601-874d-e58d0704f6df-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25e53f18-a85a-4601-874d-e58d0704f6df-maas-api-openshift-ca\") pod \"25e53f18-a85a-4601-874d-e58d0704f6df\" (UID: \"25e53f18-a85a-4601-874d-e58d0704f6df\") "
Oct 30 00:56:52.529743 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:52.529716    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25e53f18-a85a-4601-874d-e58d0704f6df-maas-api-openshift-ca" pod "25e53f18-a85a-4601-874d-e58d0704f6df" (UID: "25e53f18-a85a-4601-874d-e58d0704f6df"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:56:52.529982 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:52.529954    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25e53f18-a85a-4601-874d-e58d0704f6df-maas-api-openshift-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:56:52.531555 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:52.531528    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25e53f18-a85a-4601-874d-e58d0704f6df-maas-api-xks-ca" pod "25e53f18-a85a-4601-874d-e58d0704f6df" (UID: "25e53f18-a85a-4601-874d-e58d0704f6df"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:56:52.531969 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:52.531950    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25e53f18-a85a-4601-874d-e58d0704f6df-kube-api-access-m4n9w" pod "25e53f18-a85a-4601-874d-e58d0704f6df" (UID: "25e53f18-a85a-4601-874d-e58d0704f6df"). InnerVolumeSpecName "kube-api-access-m4n9w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:56:52.631990 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:52.631955    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4n9w\" (UniqueName: \"kubernetes.io/projected/25e53f18-a85a-4601-874d-e58d0704f6df-kube-api-access-m4n9w\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:56:52.631990 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:52.631984    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25e53f18-a85a-4601-874d-e58d0704f6df-maas-api-xks-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:56:53.238521 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:53.238476    5002 generic.go:356] "Generic (PLEG): container finished" podID="fabdd87d-54a8-4fae-adf9-f2e23ff299cf" containerID="b58568d8c76e3febbab0fa823c357c99c840fbb59378f57b8f45eeb3a827274b" exitCode=0
Oct 30 00:56:53.238691 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:53.238543    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d7a6db25-openshift-default-68896bcdc7-pgwjg" event={"ID":"fabdd87d-54a8-4fae-adf9-f2e23ff299cf","Type":"ContainerDied","Data":"b58568d8c76e3febbab0fa823c357c99c840fbb59378f57b8f45eeb3a827274b"}
Oct 30 00:56:53.240256 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:53.240231    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4d83951-wxdv6"
Oct 30 00:56:53.240256 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:53.240247    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4d83951-wxdv6" event={"ID":"25e53f18-a85a-4601-874d-e58d0704f6df","Type":"ContainerDied","Data":"1b047991badbc5d6366cf16b12a7d5b07da718d09bfe8e13a8192c58ae3775df"}
Oct 30 00:56:53.240463 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:53.240292    5002 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1b047991badbc5d6366cf16b12a7d5b07da718d09bfe8e13a8192c58ae3775df"
Oct 30 00:56:53.592230 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:53.592201    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d7a6db25-openshift-default-68896bcdc7-pgwjg"
Oct 30 00:56:53.747708 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:53.747668    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fabdd87d-54a8-4fae-adf9-f2e23ff299cf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fabdd87d-54a8-4fae-adf9-f2e23ff299cf-workload-certs\") pod \"fabdd87d-54a8-4fae-adf9-f2e23ff299cf\" (UID: \"fabdd87d-54a8-4fae-adf9-f2e23ff299cf\") "
Oct 30 00:56:53.747893 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:53.747753    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fabdd87d-54a8-4fae-adf9-f2e23ff299cf-istio-token\" (UniqueName: \"kubernetes.io/projected/fabdd87d-54a8-4fae-adf9-f2e23ff299cf-istio-token\") pod \"fabdd87d-54a8-4fae-adf9-f2e23ff299cf\" (UID: \"fabdd87d-54a8-4fae-adf9-f2e23ff299cf\") "
Oct 30 00:56:53.747893 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:53.747783    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fabdd87d-54a8-4fae-adf9-f2e23ff299cf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fabdd87d-54a8-4fae-adf9-f2e23ff299cf-istiod-ca-cert\") pod \"fabdd87d-54a8-4fae-adf9-f2e23ff299cf\" (UID: \"fabdd87d-54a8-4fae-adf9-f2e23ff299cf\") "
Oct 30 00:56:53.748020 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:53.747958    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fabdd87d-54a8-4fae-adf9-f2e23ff299cf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fabdd87d-54a8-4fae-adf9-f2e23ff299cf-istio-data\") pod \"fabdd87d-54a8-4fae-adf9-f2e23ff299cf\" (UID: \"fabdd87d-54a8-4fae-adf9-f2e23ff299cf\") "
Oct 30 00:56:53.748020 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:53.747959    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fabdd87d-54a8-4fae-adf9-f2e23ff299cf-workload-certs" pod "fabdd87d-54a8-4fae-adf9-f2e23ff299cf" (UID: "fabdd87d-54a8-4fae-adf9-f2e23ff299cf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:56:53.748122 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:53.748024    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fabdd87d-54a8-4fae-adf9-f2e23ff299cf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fabdd87d-54a8-4fae-adf9-f2e23ff299cf-istio-envoy\") pod \"fabdd87d-54a8-4fae-adf9-f2e23ff299cf\" (UID: \"fabdd87d-54a8-4fae-adf9-f2e23ff299cf\") "
Oct 30 00:56:53.748122 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:53.748057    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fabdd87d-54a8-4fae-adf9-f2e23ff299cf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fabdd87d-54a8-4fae-adf9-f2e23ff299cf-credential-socket\") pod \"fabdd87d-54a8-4fae-adf9-f2e23ff299cf\" (UID: \"fabdd87d-54a8-4fae-adf9-f2e23ff299cf\") "
Oct 30 00:56:53.748122 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:53.748097    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fabdd87d-54a8-4fae-adf9-f2e23ff299cf-kube-api-access-kkvqh\" (UniqueName: \"kubernetes.io/projected/fabdd87d-54a8-4fae-adf9-f2e23ff299cf-kube-api-access-kkvqh\") pod \"fabdd87d-54a8-4fae-adf9-f2e23ff299cf\" (UID: \"fabdd87d-54a8-4fae-adf9-f2e23ff299cf\") "
Oct 30 00:56:53.748291 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:53.748136    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fabdd87d-54a8-4fae-adf9-f2e23ff299cf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fabdd87d-54a8-4fae-adf9-f2e23ff299cf-istio-podinfo\") pod \"fabdd87d-54a8-4fae-adf9-f2e23ff299cf\" (UID: \"fabdd87d-54a8-4fae-adf9-f2e23ff299cf\") "
Oct 30 00:56:53.748291 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:53.748202    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fabdd87d-54a8-4fae-adf9-f2e23ff299cf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fabdd87d-54a8-4fae-adf9-f2e23ff299cf-workload-socket\") pod \"fabdd87d-54a8-4fae-adf9-f2e23ff299cf\" (UID: \"fabdd87d-54a8-4fae-adf9-f2e23ff299cf\") "
Oct 30 00:56:53.748291 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:53.748200    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fabdd87d-54a8-4fae-adf9-f2e23ff299cf-istio-data" pod "fabdd87d-54a8-4fae-adf9-f2e23ff299cf" (UID: "fabdd87d-54a8-4fae-adf9-f2e23ff299cf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:56:53.748291 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:53.748243    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fabdd87d-54a8-4fae-adf9-f2e23ff299cf-istiod-ca-cert" pod "fabdd87d-54a8-4fae-adf9-f2e23ff299cf" (UID: "fabdd87d-54a8-4fae-adf9-f2e23ff299cf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:56:53.748536 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:53.748353    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fabdd87d-54a8-4fae-adf9-f2e23ff299cf-credential-socket" pod "fabdd87d-54a8-4fae-adf9-f2e23ff299cf" (UID: "fabdd87d-54a8-4fae-adf9-f2e23ff299cf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:56:53.748592 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:53.748568    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fabdd87d-54a8-4fae-adf9-f2e23ff299cf-workload-socket" pod "fabdd87d-54a8-4fae-adf9-f2e23ff299cf" (UID: "fabdd87d-54a8-4fae-adf9-f2e23ff299cf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:56:53.748957 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:53.748936    5002 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fabdd87d-54a8-4fae-adf9-f2e23ff299cf-istio-data\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:56:53.749062 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:53.748962    5002 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fabdd87d-54a8-4fae-adf9-f2e23ff299cf-credential-socket\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:56:53.749795 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:53.749734    5002 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fabdd87d-54a8-4fae-adf9-f2e23ff299cf-workload-socket\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:56:53.749795 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:53.749758    5002 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fabdd87d-54a8-4fae-adf9-f2e23ff299cf-workload-certs\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:56:53.749795 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:53.749775    5002 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fabdd87d-54a8-4fae-adf9-f2e23ff299cf-istiod-ca-cert\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:56:53.751465 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:53.751429    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fabdd87d-54a8-4fae-adf9-f2e23ff299cf-istio-envoy" pod "fabdd87d-54a8-4fae-adf9-f2e23ff299cf" (UID: "fabdd87d-54a8-4fae-adf9-f2e23ff299cf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:56:53.751583 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:53.751460    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fabdd87d-54a8-4fae-adf9-f2e23ff299cf-istio-token" pod "fabdd87d-54a8-4fae-adf9-f2e23ff299cf" (UID: "fabdd87d-54a8-4fae-adf9-f2e23ff299cf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:56:53.751583 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:53.751499    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fabdd87d-54a8-4fae-adf9-f2e23ff299cf-kube-api-access-kkvqh" pod "fabdd87d-54a8-4fae-adf9-f2e23ff299cf" (UID: "fabdd87d-54a8-4fae-adf9-f2e23ff299cf"). InnerVolumeSpecName "kube-api-access-kkvqh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:56:53.751785 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:53.751761    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fabdd87d-54a8-4fae-adf9-f2e23ff299cf-istio-podinfo" pod "fabdd87d-54a8-4fae-adf9-f2e23ff299cf" (UID: "fabdd87d-54a8-4fae-adf9-f2e23ff299cf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 00:56:53.852088 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:53.851999    5002 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fabdd87d-54a8-4fae-adf9-f2e23ff299cf-istio-token\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:56:53.852088 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:53.852034    5002 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fabdd87d-54a8-4fae-adf9-f2e23ff299cf-istio-envoy\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:56:53.852088 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:53.852049    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kkvqh\" (UniqueName: \"kubernetes.io/projected/fabdd87d-54a8-4fae-adf9-f2e23ff299cf-kube-api-access-kkvqh\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:56:53.852088 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:53.852065    5002 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fabdd87d-54a8-4fae-adf9-f2e23ff299cf-istio-podinfo\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:56:54.248180 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:54.248132    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d7a6db25-openshift-default-68896bcdc7-pgwjg" event={"ID":"fabdd87d-54a8-4fae-adf9-f2e23ff299cf","Type":"ContainerDied","Data":"6fc710c6235483b913ee37bfbfe64cd4cd07adb1c6d42987681a061fae666151"}
Oct 30 00:56:54.248180 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:54.248173    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d7a6db25-openshift-default-68896bcdc7-pgwjg"
Oct 30 00:56:54.248401 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:54.248245    5002 scope.go:122] "RemoveContainer" containerID="b58568d8c76e3febbab0fa823c357c99c840fbb59378f57b8f45eeb3a827274b"
Oct 30 00:56:54.309534 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:54.309505    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-d7a6db25-openshift-default-68896bcdc7-pgwjg"]
Oct 30 00:56:54.315057 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:54.315023    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-d7a6db25-openshift-default-68896bcdc7-pgwjg"]
Oct 30 00:56:54.499257 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:54.499149    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fabdd87d-54a8-4fae-adf9-f2e23ff299cf" path="/var/lib/kubelet/pods/fabdd87d-54a8-4fae-adf9-f2e23ff299cf/volumes"
Oct 30 00:56:55.972401 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:55.970866    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7d8z4"]
Oct 30 00:56:58.173430 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:58.173400    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8dbe73-788cc7d8c9-qnxcz"
Oct 30 00:56:58.173825 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:56:58.173586    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-8dbe73-698f74c77d-l7tpx"
Oct 30 00:57:08.713358 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:57:08.713319    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-52785f5e-9fgf7"]
Oct 30 00:57:08.726962 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:57:08.726933    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-52785f5e-9fgf7"]
Oct 30 00:57:10.497777 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:57:10.497736    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51a535ea-8c8a-46e1-a624-3d8f324e97fc" path="/var/lib/kubelet/pods/51a535ea-8c8a-46e1-a624-3d8f324e97fc/volumes"
Oct 30 00:57:12.914257 ip-10-0-128-7 kubenswrapper[5002]: E1030 00:57:12.912260    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbjdq" containerName="extractor"
Oct 30 00:57:16.936415 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:57:16.936366    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-51b95e-openshift-default-74bf6d5cb5-swcm7"]
Oct 30 00:57:16.937304 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:57:16.937270    5002 cpu_manager.go:414] "containerMap: removing container" podUID="25e53f18-a85a-4601-874d-e58d0704f6df" containerName="revoke-keys"
Oct 30 00:57:16.937304 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:57:16.937296    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25e53f18-a85a-4601-874d-e58d0704f6df" containerName="revoke-keys"
Oct 30 00:57:16.937546 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:57:16.937340    5002 cpu_manager.go:414] "containerMap: removing container" podUID="fabdd87d-54a8-4fae-adf9-f2e23ff299cf" containerName="istio-proxy"
Oct 30 00:57:16.937546 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:57:16.937348    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fabdd87d-54a8-4fae-adf9-f2e23ff299cf" containerName="istio-proxy"
Oct 30 00:57:16.937546 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:57:16.937526    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="25e53f18-a85a-4601-874d-e58d0704f6df" containerName="revoke-keys"
Oct 30 00:57:16.937546 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:57:16.937543    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="fabdd87d-54a8-4fae-adf9-f2e23ff299cf" containerName="istio-proxy"
Oct 30 00:57:16.943782 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:57:16.943756    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-51b95e-openshift-default-74bf6d5cb5-swcm7"
Oct 30 00:57:16.984116 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:57:16.984085    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-51b95e-openshift-default-74bf6d5cb5-swcm7"]
Oct 30 00:57:16.986602 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:57:16.986571    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-51b95e-openshift-default-dockercfg-vtc5s\""
Oct 30 00:57:17.084372 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:57:17.084331    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bb71dc1-0795-464e-9887-a6b5ce483890-workload-certs\") pod \"e2e-inf-a-51b95e-openshift-default-74bf6d5cb5-swcm7\" (UID: \"2bb71dc1-0795-464e-9887-a6b5ce483890\") " pod="openshift-ingress/e2e-inf-a-51b95e-openshift-default-74bf6d5cb5-swcm7"
Oct 30 00:57:17.084606 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:57:17.084566    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bb71dc1-0795-464e-9887-a6b5ce483890-credential-socket\") pod \"e2e-inf-a-51b95e-openshift-default-74bf6d5cb5-swcm7\" (UID: \"2bb71dc1-0795-464e-9887-a6b5ce483890\") " pod="openshift-ingress/e2e-inf-a-51b95e-openshift-default-74bf6d5cb5-swcm7"
Oct 30 00:57:17.084672 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:57:17.084604    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bb71dc1-0795-464e-9887-a6b5ce483890-istiod-ca-cert\") pod \"e2e-inf-a-51b95e-openshift-default-74bf6d5cb5-swcm7\" (UID: \"2bb71dc1-0795-464e-9887-a6b5ce483890\") " pod="openshift-ingress/e2e-inf-a-51b95e-openshift-default-74bf6d5cb5-swcm7"
Oct 30 00:57:17.084672 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:57:17.084640    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2bb71dc1-0795-464e-9887-a6b5ce483890-istio-token\") pod \"e2e-inf-a-51b95e-openshift-default-74bf6d5cb5-swcm7\" (UID: \"2bb71dc1-0795-464e-9887-a6b5ce483890\") " pod="openshift-ingress/e2e-inf-a-51b95e-openshift-default-74bf6d5cb5-swcm7"
Oct 30 00:57:17.084770 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:57:17.084737    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bb71dc1-0795-464e-9887-a6b5ce483890-workload-socket\") pod \"e2e-inf-a-51b95e-openshift-default-74bf6d5cb5-swcm7\" (UID: \"2bb71dc1-0795-464e-9887-a6b5ce483890\") " pod="openshift-ingress/e2e-inf-a-51b95e-openshift-default-74bf6d5cb5-swcm7"
Oct 30 00:57:17.084831 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:57:17.084778    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bb71dc1-0795-464e-9887-a6b5ce483890-istio-envoy\") pod \"e2e-inf-a-51b95e-openshift-default-74bf6d5cb5-swcm7\" (UID: \"2bb71dc1-0795-464e-9887-a6b5ce483890\") " pod="openshift-ingress/e2e-inf-a-51b95e-openshift-default-74bf6d5cb5-swcm7"
Oct 30 00:57:17.084831 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:57:17.084794    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bb71dc1-0795-464e-9887-a6b5ce483890-istio-podinfo\") pod \"e2e-inf-a-51b95e-openshift-default-74bf6d5cb5-swcm7\" (UID: \"2bb71dc1-0795-464e-9887-a6b5ce483890\") " pod="openshift-ingress/e2e-inf-a-51b95e-openshift-default-74bf6d5cb5-swcm7"
Oct 30 00:57:17.084831 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:57:17.084821    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vf72\" (UniqueName: \"kubernetes.io/projected/2bb71dc1-0795-464e-9887-a6b5ce483890-kube-api-access-4vf72\") pod \"e2e-inf-a-51b95e-openshift-default-74bf6d5cb5-swcm7\" (UID: \"2bb71dc1-0795-464e-9887-a6b5ce483890\") " pod="openshift-ingress/e2e-inf-a-51b95e-openshift-default-74bf6d5cb5-swcm7"
Oct 30 00:57:17.084974 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:57:17.084837    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bb71dc1-0795-464e-9887-a6b5ce483890-istio-data\") pod \"e2e-inf-a-51b95e-openshift-default-74bf6d5cb5-swcm7\" (UID: \"2bb71dc1-0795-464e-9887-a6b5ce483890\") " pod="openshift-ingress/e2e-inf-a-51b95e-openshift-default-74bf6d5cb5-swcm7"
Oct 30 00:57:17.209464 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:57:17.189757    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bb71dc1-0795-464e-9887-a6b5ce483890-workload-socket\") pod \"e2e-inf-a-51b95e-openshift-default-74bf6d5cb5-swcm7\" (UID: \"2bb71dc1-0795-464e-9887-a6b5ce483890\") " pod="openshift-ingress/e2e-inf-a-51b95e-openshift-default-74bf6d5cb5-swcm7"
Oct 30 00:57:17.210339 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:57:17.210208    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bb71dc1-0795-464e-9887-a6b5ce483890-workload-socket\") pod \"e2e-inf-a-51b95e-openshift-default-74bf6d5cb5-swcm7\" (UID: \"2bb71dc1-0795-464e-9887-a6b5ce483890\") " pod="openshift-ingress/e2e-inf-a-51b95e-openshift-default-74bf6d5cb5-swcm7"
Oct 30 00:57:17.211259 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:57:17.210590    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bb71dc1-0795-464e-9887-a6b5ce483890-istio-envoy\") pod \"e2e-inf-a-51b95e-openshift-default-74bf6d5cb5-swcm7\" (UID: \"2bb71dc1-0795-464e-9887-a6b5ce483890\") " pod="openshift-ingress/e2e-inf-a-51b95e-openshift-default-74bf6d5cb5-swcm7"
Oct 30 00:57:17.211434 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:57:17.211417    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bb71dc1-0795-464e-9887-a6b5ce483890-istio-podinfo\") pod \"e2e-inf-a-51b95e-openshift-default-74bf6d5cb5-swcm7\" (UID: \"2bb71dc1-0795-464e-9887-a6b5ce483890\") " pod="openshift-ingress/e2e-inf-a-51b95e-openshift-default-74bf6d5cb5-swcm7"
Oct 30 00:57:17.211851 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:57:17.211822    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vf72\" (UniqueName: \"kubernetes.io/projected/2bb71dc1-0795-464e-9887-a6b5ce483890-kube-api-access-4vf72\") pod \"e2e-inf-a-51b95e-openshift-default-74bf6d5cb5-swcm7\" (UID: \"2bb71dc1-0795-464e-9887-a6b5ce483890\") " pod="openshift-ingress/e2e-inf-a-51b95e-openshift-default-74bf6d5cb5-swcm7"
Oct 30 00:57:17.212536 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:57:17.212517    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bb71dc1-0795-464e-9887-a6b5ce483890-istio-data\") pod \"e2e-inf-a-51b95e-openshift-default-74bf6d5cb5-swcm7\" (UID: \"2bb71dc1-0795-464e-9887-a6b5ce483890\") " pod="openshift-ingress/e2e-inf-a-51b95e-openshift-default-74bf6d5cb5-swcm7"
Oct 30 00:57:17.213269 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:57:17.213003    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bb71dc1-0795-464e-9887-a6b5ce483890-istio-data\") pod \"e2e-inf-a-51b95e-openshift-default-74bf6d5cb5-swcm7\" (UID: \"2bb71dc1-0795-464e-9887-a6b5ce483890\") " pod="openshift-ingress/e2e-inf-a-51b95e-openshift-default-74bf6d5cb5-swcm7"
Oct 30 00:57:17.213730 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:57:17.213711    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bb71dc1-0795-464e-9887-a6b5ce483890-workload-certs\") pod \"e2e-inf-a-51b95e-openshift-default-74bf6d5cb5-swcm7\" (UID: \"2bb71dc1-0795-464e-9887-a6b5ce483890\") " pod="openshift-ingress/e2e-inf-a-51b95e-openshift-default-74bf6d5cb5-swcm7"
Oct 30 00:57:17.216942 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:57:17.216818    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bb71dc1-0795-464e-9887-a6b5ce483890-workload-certs\") pod \"e2e-inf-a-51b95e-openshift-default-74bf6d5cb5-swcm7\" (UID: \"2bb71dc1-0795-464e-9887-a6b5ce483890\") " pod="openshift-ingress/e2e-inf-a-51b95e-openshift-default-74bf6d5cb5-swcm7"
Oct 30 00:57:17.217232 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:57:17.217192    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bb71dc1-0795-464e-9887-a6b5ce483890-credential-socket\") pod \"e2e-inf-a-51b95e-openshift-default-74bf6d5cb5-swcm7\" (UID: \"2bb71dc1-0795-464e-9887-a6b5ce483890\") " pod="openshift-ingress/e2e-inf-a-51b95e-openshift-default-74bf6d5cb5-swcm7"
Oct 30 00:57:17.217794 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:57:17.217759    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bb71dc1-0795-464e-9887-a6b5ce483890-istiod-ca-cert\") pod \"e2e-inf-a-51b95e-openshift-default-74bf6d5cb5-swcm7\" (UID: \"2bb71dc1-0795-464e-9887-a6b5ce483890\") " pod="openshift-ingress/e2e-inf-a-51b95e-openshift-default-74bf6d5cb5-swcm7"
Oct 30 00:57:17.224829 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:57:17.224810    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2bb71dc1-0795-464e-9887-a6b5ce483890-istio-token\") pod \"e2e-inf-a-51b95e-openshift-default-74bf6d5cb5-swcm7\" (UID: \"2bb71dc1-0795-464e-9887-a6b5ce483890\") " pod="openshift-ingress/e2e-inf-a-51b95e-openshift-default-74bf6d5cb5-swcm7"
Oct 30 00:57:17.239722 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:57:17.224692    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bb71dc1-0795-464e-9887-a6b5ce483890-istiod-ca-cert\") pod \"e2e-inf-a-51b95e-openshift-default-74bf6d5cb5-swcm7\" (UID: \"2bb71dc1-0795-464e-9887-a6b5ce483890\") " pod="openshift-ingress/e2e-inf-a-51b95e-openshift-default-74bf6d5cb5-swcm7"
Oct 30 00:57:17.239951 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:57:17.217661    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bb71dc1-0795-464e-9887-a6b5ce483890-credential-socket\") pod \"e2e-inf-a-51b95e-openshift-default-74bf6d5cb5-swcm7\" (UID: \"2bb71dc1-0795-464e-9887-a6b5ce483890\") " pod="openshift-ingress/e2e-inf-a-51b95e-openshift-default-74bf6d5cb5-swcm7"
Oct 30 00:57:17.258452 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:57:17.245543    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vf72\" (UniqueName: \"kubernetes.io/projected/2bb71dc1-0795-464e-9887-a6b5ce483890-kube-api-access-4vf72\") pod \"e2e-inf-a-51b95e-openshift-default-74bf6d5cb5-swcm7\" (UID: \"2bb71dc1-0795-464e-9887-a6b5ce483890\") " pod="openshift-ingress/e2e-inf-a-51b95e-openshift-default-74bf6d5cb5-swcm7"
Oct 30 00:57:17.258691 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:57:17.252435    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bb71dc1-0795-464e-9887-a6b5ce483890-istio-envoy\") pod \"e2e-inf-a-51b95e-openshift-default-74bf6d5cb5-swcm7\" (UID: \"2bb71dc1-0795-464e-9887-a6b5ce483890\") " pod="openshift-ingress/e2e-inf-a-51b95e-openshift-default-74bf6d5cb5-swcm7"
Oct 30 00:57:17.258805 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:57:17.255948    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bb71dc1-0795-464e-9887-a6b5ce483890-istio-podinfo\") pod \"e2e-inf-a-51b95e-openshift-default-74bf6d5cb5-swcm7\" (UID: \"2bb71dc1-0795-464e-9887-a6b5ce483890\") " pod="openshift-ingress/e2e-inf-a-51b95e-openshift-default-74bf6d5cb5-swcm7"
Oct 30 00:57:17.282407 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:57:17.281912    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2bb71dc1-0795-464e-9887-a6b5ce483890-istio-token\") pod \"e2e-inf-a-51b95e-openshift-default-74bf6d5cb5-swcm7\" (UID: \"2bb71dc1-0795-464e-9887-a6b5ce483890\") " pod="openshift-ingress/e2e-inf-a-51b95e-openshift-default-74bf6d5cb5-swcm7"
Oct 30 00:57:17.298987 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:57:17.298755    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-51b95e-openshift-default-74bf6d5cb5-swcm7"
Oct 30 00:57:17.813930 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:57:17.813896    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bb71dc1_0795_464e_9887_a6b5ce483890.slice/crio-517d8e5d7594b3f5f60089dcb0d2c615e7903aacad5141f4b664c8def9364167 WatchSource:0}: Error finding container 517d8e5d7594b3f5f60089dcb0d2c615e7903aacad5141f4b664c8def9364167: Status 404 returned error can't find the container with id 517d8e5d7594b3f5f60089dcb0d2c615e7903aacad5141f4b664c8def9364167
Oct 30 00:57:17.815146 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:57:17.815122    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-51b95e-openshift-default-74bf6d5cb5-swcm7"]
Oct 30 00:57:17.817614 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:57:17.817575    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 00:57:17.817708 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:57:17.817667    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 00:57:17.817752 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:57:17.817717    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 00:57:18.655371 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:57:18.655333    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-51b95e-openshift-default-74bf6d5cb5-swcm7" event={"ID":"2bb71dc1-0795-464e-9887-a6b5ce483890","Type":"ContainerStarted","Data":"525f9d8e4f3147eacee1dd4e21683a0a0d59733dec2dde54440f3169f819f90a"}
Oct 30 00:57:18.656968 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:57:18.656940    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-51b95e-openshift-default-74bf6d5cb5-swcm7" event={"ID":"2bb71dc1-0795-464e-9887-a6b5ce483890","Type":"ContainerStarted","Data":"517d8e5d7594b3f5f60089dcb0d2c615e7903aacad5141f4b664c8def9364167"}
Oct 30 00:57:18.712007 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:57:18.711948    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-51b95e-openshift-default-74bf6d5cb5-swcm7" podStartSLOduration=2.71193319 podStartE2EDuration="2.71193319s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:57:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:57:18.709825156 +0000 UTC m=+1534.326258046" watchObservedRunningTime="2026-10-30 00:57:18.71193319 +0000 UTC m=+1534.328366068"
Oct 30 00:57:19.299895 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:57:19.299857    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-51b95e-openshift-default-74bf6d5cb5-swcm7"
Oct 30 00:57:19.306263 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:57:19.306232    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-51b95e-openshift-default-74bf6d5cb5-swcm7"
Oct 30 00:57:19.663637 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:57:19.663542    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-51b95e-openshift-default-74bf6d5cb5-swcm7"
Oct 30 00:57:19.664704 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:57:19.664680    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-51b95e-openshift-default-74bf6d5cb5-swcm7"
Oct 30 00:57:21.884404 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:57:21.876428    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-02fbf5-w477t"]
Oct 30 00:57:21.893409 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:57:21.891482    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-02fbf5-w477t"]
Oct 30 00:57:22.500337 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:57:22.500287    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1178ba7-6655-4ecd-b2f5-79477a445322" path="/var/lib/kubelet/pods/b1178ba7-6655-4ecd-b2f5-79477a445322/volumes"
Oct 30 00:57:29.941876 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:57:29.941840    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-02fbf5-openshift-default-64fc59d485-jhdtc"]
Oct 30 00:57:29.942427 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:57:29.942106    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-02fbf5-openshift-default-64fc59d485-jhdtc" podUID="58608e31-3cf2-4a5c-9d19-c8c72b774f51" containerName="istio-proxy" containerID="cri-o://2c078af2f907835699d23134075d192736e875a49f894125f413bb89a83d2870" gracePeriod=30
Oct 30 00:57:35.316261 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:57:35.316229    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-02fbf5-openshift-default-64fc59d485-jhdtc"
Oct 30 00:57:35.443175 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:57:35.443142    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58608e31-3cf2-4a5c-9d19-c8c72b774f51-istio-token\" (UniqueName: \"kubernetes.io/projected/58608e31-3cf2-4a5c-9d19-c8c72b774f51-istio-token\") pod \"58608e31-3cf2-4a5c-9d19-c8c72b774f51\" (UID: \"58608e31-3cf2-4a5c-9d19-c8c72b774f51\") "
Oct 30 00:57:35.443395 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:57:35.443204    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58608e31-3cf2-4a5c-9d19-c8c72b774f51-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58608e31-3cf2-4a5c-9d19-c8c72b774f51-istio-data\") pod \"58608e31-3cf2-4a5c-9d19-c8c72b774f51\" (UID: \"58608e31-3cf2-4a5c-9d19-c8c72b774f51\") "
Oct 30 00:57:35.443395 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:57:35.443229    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58608e31-3cf2-4a5c-9d19-c8c72b774f51-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58608e31-3cf2-4a5c-9d19-c8c72b774f51-istiod-ca-cert\") pod \"58608e31-3cf2-4a5c-9d19-c8c72b774f51\" (UID: \"58608e31-3cf2-4a5c-9d19-c8c72b774f51\") "
Oct 30 00:57:35.443395 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:57:35.443332    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/58608e31-3cf2-4a5c-9d19-c8c72b774f51-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58608e31-3cf2-4a5c-9d19-c8c72b774f51-istio-podinfo\") pod \"58608e31-3cf2-4a5c-9d19-c8c72b774f51\" (UID: \"58608e31-3cf2-4a5c-9d19-c8c72b774f51\") "
Oct 30 00:57:35.443610 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:57:35.443415    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58608e31-3cf2-4a5c-9d19-c8c72b774f51-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58608e31-3cf2-4a5c-9d19-c8c72b774f51-credential-socket\") pod \"58608e31-3cf2-4a5c-9d19-c8c72b774f51\" (UID: \"58608e31-3cf2-4a5c-9d19-c8c72b774f51\") "
Oct 30 00:57:35.443610 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:57:35.443453    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58608e31-3cf2-4a5c-9d19-c8c72b774f51-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58608e31-3cf2-4a5c-9d19-c8c72b774f51-workload-certs\") pod \"58608e31-3cf2-4a5c-9d19-c8c72b774f51\" (UID: \"58608e31-3cf2-4a5c-9d19-c8c72b774f51\") "
Oct 30 00:57:35.443610 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:57:35.443515    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58608e31-3cf2-4a5c-9d19-c8c72b774f51-kube-api-access-p4fkl\" (UniqueName: \"kubernetes.io/projected/58608e31-3cf2-4a5c-9d19-c8c72b774f51-kube-api-access-p4fkl\") pod \"58608e31-3cf2-4a5c-9d19-c8c72b774f51\" (UID: \"58608e31-3cf2-4a5c-9d19-c8c72b774f51\") "
Oct 30 00:57:35.443610 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:57:35.443534    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58608e31-3cf2-4a5c-9d19-c8c72b774f51-istio-data" pod "58608e31-3cf2-4a5c-9d19-c8c72b774f51" (UID: "58608e31-3cf2-4a5c-9d19-c8c72b774f51"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:57:35.443829 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:57:35.443610    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58608e31-3cf2-4a5c-9d19-c8c72b774f51-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58608e31-3cf2-4a5c-9d19-c8c72b774f51-workload-socket\") pod \"58608e31-3cf2-4a5c-9d19-c8c72b774f51\" (UID: \"58608e31-3cf2-4a5c-9d19-c8c72b774f51\") "
Oct 30 00:57:35.443829 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:57:35.443645    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58608e31-3cf2-4a5c-9d19-c8c72b774f51-istiod-ca-cert" pod "58608e31-3cf2-4a5c-9d19-c8c72b774f51" (UID: "58608e31-3cf2-4a5c-9d19-c8c72b774f51"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:57:35.443829 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:57:35.443739    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58608e31-3cf2-4a5c-9d19-c8c72b774f51-credential-socket" pod "58608e31-3cf2-4a5c-9d19-c8c72b774f51" (UID: "58608e31-3cf2-4a5c-9d19-c8c72b774f51"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:57:35.443829 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:57:35.443801    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58608e31-3cf2-4a5c-9d19-c8c72b774f51-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58608e31-3cf2-4a5c-9d19-c8c72b774f51-istio-envoy\") pod \"58608e31-3cf2-4a5c-9d19-c8c72b774f51\" (UID: \"58608e31-3cf2-4a5c-9d19-c8c72b774f51\") "
Oct 30 00:57:35.444038 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:57:35.443875    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58608e31-3cf2-4a5c-9d19-c8c72b774f51-workload-certs" pod "58608e31-3cf2-4a5c-9d19-c8c72b774f51" (UID: "58608e31-3cf2-4a5c-9d19-c8c72b774f51"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:57:35.444212 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:57:35.444191    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58608e31-3cf2-4a5c-9d19-c8c72b774f51-workload-socket" pod "58608e31-3cf2-4a5c-9d19-c8c72b774f51" (UID: "58608e31-3cf2-4a5c-9d19-c8c72b774f51"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:57:35.444457 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:57:35.444353    5002 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58608e31-3cf2-4a5c-9d19-c8c72b774f51-credential-socket\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:57:35.444587 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:57:35.444572    5002 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58608e31-3cf2-4a5c-9d19-c8c72b774f51-workload-certs\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:57:35.444689 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:57:35.444679    5002 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58608e31-3cf2-4a5c-9d19-c8c72b774f51-istio-data\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:57:35.444820 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:57:35.444803    5002 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58608e31-3cf2-4a5c-9d19-c8c72b774f51-istiod-ca-cert\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:57:35.446233 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:57:35.446212    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58608e31-3cf2-4a5c-9d19-c8c72b774f51-kube-api-access-p4fkl" pod "58608e31-3cf2-4a5c-9d19-c8c72b774f51" (UID: "58608e31-3cf2-4a5c-9d19-c8c72b774f51"). InnerVolumeSpecName "kube-api-access-p4fkl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:57:35.446422 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:57:35.446357    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/58608e31-3cf2-4a5c-9d19-c8c72b774f51-istio-podinfo" pod "58608e31-3cf2-4a5c-9d19-c8c72b774f51" (UID: "58608e31-3cf2-4a5c-9d19-c8c72b774f51"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 00:57:35.446522 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:57:35.446452    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58608e31-3cf2-4a5c-9d19-c8c72b774f51-istio-token" pod "58608e31-3cf2-4a5c-9d19-c8c72b774f51" (UID: "58608e31-3cf2-4a5c-9d19-c8c72b774f51"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:57:35.446890 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:57:35.446874    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58608e31-3cf2-4a5c-9d19-c8c72b774f51-istio-envoy" pod "58608e31-3cf2-4a5c-9d19-c8c72b774f51" (UID: "58608e31-3cf2-4a5c-9d19-c8c72b774f51"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:57:35.546112 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:57:35.546032    5002 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58608e31-3cf2-4a5c-9d19-c8c72b774f51-istio-podinfo\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:57:35.546112 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:57:35.546062    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p4fkl\" (UniqueName: \"kubernetes.io/projected/58608e31-3cf2-4a5c-9d19-c8c72b774f51-kube-api-access-p4fkl\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:57:35.546112 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:57:35.546072    5002 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58608e31-3cf2-4a5c-9d19-c8c72b774f51-workload-socket\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:57:35.546112 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:57:35.546082    5002 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58608e31-3cf2-4a5c-9d19-c8c72b774f51-istio-envoy\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:57:35.546112 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:57:35.546089    5002 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58608e31-3cf2-4a5c-9d19-c8c72b774f51-istio-token\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:57:35.904141 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:57:35.904040    5002 generic.go:356] "Generic (PLEG): container finished" podID="58608e31-3cf2-4a5c-9d19-c8c72b774f51" containerID="2c078af2f907835699d23134075d192736e875a49f894125f413bb89a83d2870" exitCode=0
Oct 30 00:57:35.904311 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:57:35.904133    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-02fbf5-openshift-default-64fc59d485-jhdtc"
Oct 30 00:57:35.904311 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:57:35.904129    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-02fbf5-openshift-default-64fc59d485-jhdtc" event={"ID":"58608e31-3cf2-4a5c-9d19-c8c72b774f51","Type":"ContainerDied","Data":"2c078af2f907835699d23134075d192736e875a49f894125f413bb89a83d2870"}
Oct 30 00:57:35.904311 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:57:35.904270    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-02fbf5-openshift-default-64fc59d485-jhdtc" event={"ID":"58608e31-3cf2-4a5c-9d19-c8c72b774f51","Type":"ContainerDied","Data":"4b1608c79c2227804d1079325de4c888039a28c1569ed65f1edbe6a5ce241796"}
Oct 30 00:57:35.904311 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:57:35.904292    5002 scope.go:122] "RemoveContainer" containerID="2c078af2f907835699d23134075d192736e875a49f894125f413bb89a83d2870"
Oct 30 00:57:35.916878 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:57:35.916866    5002 scope.go:122] "RemoveContainer" containerID="0cf4e79fe5ba7762ac6dae465a00bf85949ae7159a9c4ad0b2992c31d13e23aa"
Oct 30 00:57:35.927431 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:57:35.927418    5002 scope.go:122] "RemoveContainer" containerID="2c078af2f907835699d23134075d192736e875a49f894125f413bb89a83d2870"
Oct 30 00:57:35.927698 ip-10-0-128-7 kubenswrapper[5002]: E1030 00:57:35.927679    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c078af2f907835699d23134075d192736e875a49f894125f413bb89a83d2870\": container with ID starting with 2c078af2f907835699d23134075d192736e875a49f894125f413bb89a83d2870 not found: ID does not exist" containerID="2c078af2f907835699d23134075d192736e875a49f894125f413bb89a83d2870"
Oct 30 00:57:35.927768 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:57:35.927705    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c078af2f907835699d23134075d192736e875a49f894125f413bb89a83d2870"} err="failed to get container status \"2c078af2f907835699d23134075d192736e875a49f894125f413bb89a83d2870\": rpc error: code = NotFound desc = could not find container \"2c078af2f907835699d23134075d192736e875a49f894125f413bb89a83d2870\": container with ID starting with 2c078af2f907835699d23134075d192736e875a49f894125f413bb89a83d2870 not found: ID does not exist"
Oct 30 00:57:35.927768 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:57:35.927727    5002 scope.go:122] "RemoveContainer" containerID="0cf4e79fe5ba7762ac6dae465a00bf85949ae7159a9c4ad0b2992c31d13e23aa"
Oct 30 00:57:35.927950 ip-10-0-128-7 kubenswrapper[5002]: E1030 00:57:35.927933    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0cf4e79fe5ba7762ac6dae465a00bf85949ae7159a9c4ad0b2992c31d13e23aa\": container with ID starting with 0cf4e79fe5ba7762ac6dae465a00bf85949ae7159a9c4ad0b2992c31d13e23aa not found: ID does not exist" containerID="0cf4e79fe5ba7762ac6dae465a00bf85949ae7159a9c4ad0b2992c31d13e23aa"
Oct 30 00:57:35.928007 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:57:35.927964    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0cf4e79fe5ba7762ac6dae465a00bf85949ae7159a9c4ad0b2992c31d13e23aa"} err="failed to get container status \"0cf4e79fe5ba7762ac6dae465a00bf85949ae7159a9c4ad0b2992c31d13e23aa\": rpc error: code = NotFound desc = could not find container \"0cf4e79fe5ba7762ac6dae465a00bf85949ae7159a9c4ad0b2992c31d13e23aa\": container with ID starting with 0cf4e79fe5ba7762ac6dae465a00bf85949ae7159a9c4ad0b2992c31d13e23aa not found: ID does not exist"
Oct 30 00:57:35.980534 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:57:35.980496    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-02fbf5-openshift-default-64fc59d485-jhdtc"]
Oct 30 00:57:35.994072 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:57:35.994044    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-02fbf5-openshift-default-64fc59d485-jhdtc"]
Oct 30 00:57:36.497085 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:57:36.497053    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58608e31-3cf2-4a5c-9d19-c8c72b774f51" path="/var/lib/kubelet/pods/58608e31-3cf2-4a5c-9d19-c8c72b774f51/volumes"
Oct 30 00:57:42.867962 ip-10-0-128-7 kubenswrapper[5002]: E1030 00:57:42.867924    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbjdq" containerName="extractor"
Oct 30 00:57:46.352692 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:57:46.352656    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fa8dd0-mrmmc"]
Oct 30 00:57:46.353336 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:57:46.353318    5002 cpu_manager.go:414] "containerMap: removing container" podUID="58608e31-3cf2-4a5c-9d19-c8c72b774f51" containerName="istio-proxy"
Oct 30 00:57:46.353391 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:57:46.353337    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58608e31-3cf2-4a5c-9d19-c8c72b774f51" containerName="istio-proxy"
Oct 30 00:57:46.353391 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:57:46.353356    5002 cpu_manager.go:414] "containerMap: removing container" podUID="58608e31-3cf2-4a5c-9d19-c8c72b774f51" containerName="istio-proxy"
Oct 30 00:57:46.353391 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:57:46.353361    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58608e31-3cf2-4a5c-9d19-c8c72b774f51" containerName="istio-proxy"
Oct 30 00:57:46.353520 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:57:46.353495    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="58608e31-3cf2-4a5c-9d19-c8c72b774f51" containerName="istio-proxy"
Oct 30 00:57:46.353520 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:57:46.353509    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="58608e31-3cf2-4a5c-9d19-c8c72b774f51" containerName="istio-proxy"
Oct 30 00:57:46.358629 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:57:46.358609    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fa8dd0-mrmmc"
Oct 30 00:57:46.376133 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:57:46.376105    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fa8dd0-mrmmc"]
Oct 30 00:57:46.385778 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:57:46.385746    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 00:57:46.385962 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:57:46.385860    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5phsh\""
Oct 30 00:57:46.385962 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:57:46.385908    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 00:57:46.385962 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:57:46.385949    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 00:57:46.454294 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:57:46.454256    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15b4eff7-8879-4a7d-8a30-19016d0d0b35-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-fa8dd0-mrmmc\" (UID: \"15b4eff7-8879-4a7d-8a30-19016d0d0b35\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fa8dd0-mrmmc"
Oct 30 00:57:46.454513 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:57:46.454321    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15b4eff7-8879-4a7d-8a30-19016d0d0b35-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-fa8dd0-mrmmc\" (UID: \"15b4eff7-8879-4a7d-8a30-19016d0d0b35\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fa8dd0-mrmmc"
Oct 30 00:57:46.454513 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:57:46.454492    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzrb8\" (UniqueName: \"kubernetes.io/projected/15b4eff7-8879-4a7d-8a30-19016d0d0b35-kube-api-access-bzrb8\") pod \"maas-api-revoke-keys-e2e-api-a-fa8dd0-mrmmc\" (UID: \"15b4eff7-8879-4a7d-8a30-19016d0d0b35\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fa8dd0-mrmmc"
Oct 30 00:57:46.510332 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:57:46.510301    5002 scope.go:122] "RemoveContainer" containerID="b29ab1d27affa01ab294bcedff8fc6dcbdae7c06be30fd3267f99cd921c51f4d"
Oct 30 00:57:46.556576 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:57:46.556546    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15b4eff7-8879-4a7d-8a30-19016d0d0b35-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-fa8dd0-mrmmc\" (UID: \"15b4eff7-8879-4a7d-8a30-19016d0d0b35\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fa8dd0-mrmmc"
Oct 30 00:57:46.556690 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:57:46.556610    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzrb8\" (UniqueName: \"kubernetes.io/projected/15b4eff7-8879-4a7d-8a30-19016d0d0b35-kube-api-access-bzrb8\") pod \"maas-api-revoke-keys-e2e-api-a-fa8dd0-mrmmc\" (UID: \"15b4eff7-8879-4a7d-8a30-19016d0d0b35\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fa8dd0-mrmmc"
Oct 30 00:57:46.556865 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:57:46.556849    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15b4eff7-8879-4a7d-8a30-19016d0d0b35-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-fa8dd0-mrmmc\" (UID: \"15b4eff7-8879-4a7d-8a30-19016d0d0b35\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fa8dd0-mrmmc"
Oct 30 00:57:46.557204 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:57:46.557186    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15b4eff7-8879-4a7d-8a30-19016d0d0b35-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-fa8dd0-mrmmc\" (UID: \"15b4eff7-8879-4a7d-8a30-19016d0d0b35\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fa8dd0-mrmmc"
Oct 30 00:57:46.559244 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:57:46.559220    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15b4eff7-8879-4a7d-8a30-19016d0d0b35-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-fa8dd0-mrmmc\" (UID: \"15b4eff7-8879-4a7d-8a30-19016d0d0b35\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fa8dd0-mrmmc"
Oct 30 00:57:46.581276 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:57:46.581250    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzrb8\" (UniqueName: \"kubernetes.io/projected/15b4eff7-8879-4a7d-8a30-19016d0d0b35-kube-api-access-bzrb8\") pod \"maas-api-revoke-keys-e2e-api-a-fa8dd0-mrmmc\" (UID: \"15b4eff7-8879-4a7d-8a30-19016d0d0b35\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fa8dd0-mrmmc"
Oct 30 00:57:46.673539 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:57:46.673462    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fa8dd0-mrmmc"
Oct 30 00:57:46.983519 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:57:46.983481    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15b4eff7_8879_4a7d_8a30_19016d0d0b35.slice/crio-7af16483ed083de26a50de5e9c30803f333214100f679aac5654c8f508d0ad4a WatchSource:0}: Error finding container 7af16483ed083de26a50de5e9c30803f333214100f679aac5654c8f508d0ad4a: Status 404 returned error can't find the container with id 7af16483ed083de26a50de5e9c30803f333214100f679aac5654c8f508d0ad4a
Oct 30 00:57:46.983896 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:57:46.983870    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fa8dd0-mrmmc"]
Oct 30 00:57:48.001692 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:57:48.001657    5002 generic.go:356] "Generic (PLEG): container finished" podID="15b4eff7-8879-4a7d-8a30-19016d0d0b35" containerID="2d20eef6aad7b011766f73825c68964c489c598cbcf672010ff9475ac569a9de" exitCode=0
Oct 30 00:57:48.002110 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:57:48.001723    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fa8dd0-mrmmc" event={"ID":"15b4eff7-8879-4a7d-8a30-19016d0d0b35","Type":"ContainerDied","Data":"2d20eef6aad7b011766f73825c68964c489c598cbcf672010ff9475ac569a9de"}
Oct 30 00:57:48.002110 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:57:48.001782    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fa8dd0-mrmmc" event={"ID":"15b4eff7-8879-4a7d-8a30-19016d0d0b35","Type":"ContainerStarted","Data":"7af16483ed083de26a50de5e9c30803f333214100f679aac5654c8f508d0ad4a"}
Oct 30 00:57:49.278940 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:57:49.278870    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fa8dd0-mrmmc"
Oct 30 00:57:49.315579 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:57:49.315546    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15b4eff7-8879-4a7d-8a30-19016d0d0b35-kube-api-access-bzrb8\" (UniqueName: \"kubernetes.io/projected/15b4eff7-8879-4a7d-8a30-19016d0d0b35-kube-api-access-bzrb8\") pod \"15b4eff7-8879-4a7d-8a30-19016d0d0b35\" (UID: \"15b4eff7-8879-4a7d-8a30-19016d0d0b35\") "
Oct 30 00:57:49.315752 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:57:49.315737    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15b4eff7-8879-4a7d-8a30-19016d0d0b35-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15b4eff7-8879-4a7d-8a30-19016d0d0b35-maas-api-openshift-ca\") pod \"15b4eff7-8879-4a7d-8a30-19016d0d0b35\" (UID: \"15b4eff7-8879-4a7d-8a30-19016d0d0b35\") "
Oct 30 00:57:49.315813 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:57:49.315780    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/15b4eff7-8879-4a7d-8a30-19016d0d0b35-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15b4eff7-8879-4a7d-8a30-19016d0d0b35-maas-api-xks-ca\") pod \"15b4eff7-8879-4a7d-8a30-19016d0d0b35\" (UID: \"15b4eff7-8879-4a7d-8a30-19016d0d0b35\") "
Oct 30 00:57:49.316116 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:57:49.316076    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15b4eff7-8879-4a7d-8a30-19016d0d0b35-maas-api-openshift-ca" pod "15b4eff7-8879-4a7d-8a30-19016d0d0b35" (UID: "15b4eff7-8879-4a7d-8a30-19016d0d0b35"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:57:49.316269 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:57:49.316253    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15b4eff7-8879-4a7d-8a30-19016d0d0b35-maas-api-openshift-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:57:49.317661 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:57:49.317636    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15b4eff7-8879-4a7d-8a30-19016d0d0b35-kube-api-access-bzrb8" pod "15b4eff7-8879-4a7d-8a30-19016d0d0b35" (UID: "15b4eff7-8879-4a7d-8a30-19016d0d0b35"). InnerVolumeSpecName "kube-api-access-bzrb8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:57:49.317758 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:57:49.317683    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15b4eff7-8879-4a7d-8a30-19016d0d0b35-maas-api-xks-ca" pod "15b4eff7-8879-4a7d-8a30-19016d0d0b35" (UID: "15b4eff7-8879-4a7d-8a30-19016d0d0b35"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:57:49.417103 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:57:49.417065    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15b4eff7-8879-4a7d-8a30-19016d0d0b35-maas-api-xks-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:57:49.417103 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:57:49.417098    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bzrb8\" (UniqueName: \"kubernetes.io/projected/15b4eff7-8879-4a7d-8a30-19016d0d0b35-kube-api-access-bzrb8\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:57:50.019027 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:57:50.018985    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fa8dd0-mrmmc" event={"ID":"15b4eff7-8879-4a7d-8a30-19016d0d0b35","Type":"ContainerDied","Data":"7af16483ed083de26a50de5e9c30803f333214100f679aac5654c8f508d0ad4a"}
Oct 30 00:57:50.019223 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:57:50.019055    5002 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7af16483ed083de26a50de5e9c30803f333214100f679aac5654c8f508d0ad4a"
Oct 30 00:57:50.019223 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:57:50.019001    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fa8dd0-mrmmc"
Oct 30 00:57:57.862461 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:57:57.846804    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7d8z4"]
Oct 30 00:58:00.220299 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:00.220260    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-84b89f-openshift-default-f7d86d855-mddrw"]
Oct 30 00:58:00.221783 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:00.221754    5002 cpu_manager.go:414] "containerMap: removing container" podUID="15b4eff7-8879-4a7d-8a30-19016d0d0b35" containerName="revoke-keys"
Oct 30 00:58:00.221947 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:00.221934    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15b4eff7-8879-4a7d-8a30-19016d0d0b35" containerName="revoke-keys"
Oct 30 00:58:00.222352 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:00.222333    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="15b4eff7-8879-4a7d-8a30-19016d0d0b35" containerName="revoke-keys"
Oct 30 00:58:00.312330 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:00.312298    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-84b89f-openshift-default-f7d86d855-mddrw"]
Oct 30 00:58:00.313049 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:00.312907    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-84b89f-openshift-default-f7d86d855-mddrw"
Oct 30 00:58:00.351429 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:00.347980    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-84b89f-openshift-default-dockercfg-x86x8\""
Oct 30 00:58:00.512458 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:00.510946    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b88fa1c1-bbb8-4f74-bc2a-3cff7b647082-istio-podinfo\") pod \"e2e-inf-b-84b89f-openshift-default-f7d86d855-mddrw\" (UID: \"b88fa1c1-bbb8-4f74-bc2a-3cff7b647082\") " pod="openshift-ingress/e2e-inf-b-84b89f-openshift-default-f7d86d855-mddrw"
Oct 30 00:58:00.512458 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:00.511044    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pgz4\" (UniqueName: \"kubernetes.io/projected/b88fa1c1-bbb8-4f74-bc2a-3cff7b647082-kube-api-access-5pgz4\") pod \"e2e-inf-b-84b89f-openshift-default-f7d86d855-mddrw\" (UID: \"b88fa1c1-bbb8-4f74-bc2a-3cff7b647082\") " pod="openshift-ingress/e2e-inf-b-84b89f-openshift-default-f7d86d855-mddrw"
Oct 30 00:58:00.512458 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:00.511138    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b88fa1c1-bbb8-4f74-bc2a-3cff7b647082-istio-data\") pod \"e2e-inf-b-84b89f-openshift-default-f7d86d855-mddrw\" (UID: \"b88fa1c1-bbb8-4f74-bc2a-3cff7b647082\") " pod="openshift-ingress/e2e-inf-b-84b89f-openshift-default-f7d86d855-mddrw"
Oct 30 00:58:00.512458 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:00.511192    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b88fa1c1-bbb8-4f74-bc2a-3cff7b647082-istio-token\") pod \"e2e-inf-b-84b89f-openshift-default-f7d86d855-mddrw\" (UID: \"b88fa1c1-bbb8-4f74-bc2a-3cff7b647082\") " pod="openshift-ingress/e2e-inf-b-84b89f-openshift-default-f7d86d855-mddrw"
Oct 30 00:58:00.512458 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:00.511219    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b88fa1c1-bbb8-4f74-bc2a-3cff7b647082-istio-envoy\") pod \"e2e-inf-b-84b89f-openshift-default-f7d86d855-mddrw\" (UID: \"b88fa1c1-bbb8-4f74-bc2a-3cff7b647082\") " pod="openshift-ingress/e2e-inf-b-84b89f-openshift-default-f7d86d855-mddrw"
Oct 30 00:58:00.512458 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:00.511264    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b88fa1c1-bbb8-4f74-bc2a-3cff7b647082-credential-socket\") pod \"e2e-inf-b-84b89f-openshift-default-f7d86d855-mddrw\" (UID: \"b88fa1c1-bbb8-4f74-bc2a-3cff7b647082\") " pod="openshift-ingress/e2e-inf-b-84b89f-openshift-default-f7d86d855-mddrw"
Oct 30 00:58:00.526436 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:00.525425    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b88fa1c1-bbb8-4f74-bc2a-3cff7b647082-workload-certs\") pod \"e2e-inf-b-84b89f-openshift-default-f7d86d855-mddrw\" (UID: \"b88fa1c1-bbb8-4f74-bc2a-3cff7b647082\") " pod="openshift-ingress/e2e-inf-b-84b89f-openshift-default-f7d86d855-mddrw"
Oct 30 00:58:00.526436 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:00.525490    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b88fa1c1-bbb8-4f74-bc2a-3cff7b647082-istiod-ca-cert\") pod \"e2e-inf-b-84b89f-openshift-default-f7d86d855-mddrw\" (UID: \"b88fa1c1-bbb8-4f74-bc2a-3cff7b647082\") " pod="openshift-ingress/e2e-inf-b-84b89f-openshift-default-f7d86d855-mddrw"
Oct 30 00:58:00.526436 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:00.525527    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b88fa1c1-bbb8-4f74-bc2a-3cff7b647082-workload-socket\") pod \"e2e-inf-b-84b89f-openshift-default-f7d86d855-mddrw\" (UID: \"b88fa1c1-bbb8-4f74-bc2a-3cff7b647082\") " pod="openshift-ingress/e2e-inf-b-84b89f-openshift-default-f7d86d855-mddrw"
Oct 30 00:58:00.629160 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:00.629119    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b88fa1c1-bbb8-4f74-bc2a-3cff7b647082-istio-token\") pod \"e2e-inf-b-84b89f-openshift-default-f7d86d855-mddrw\" (UID: \"b88fa1c1-bbb8-4f74-bc2a-3cff7b647082\") " pod="openshift-ingress/e2e-inf-b-84b89f-openshift-default-f7d86d855-mddrw"
Oct 30 00:58:00.634116 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:00.634085    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b88fa1c1-bbb8-4f74-bc2a-3cff7b647082-istio-envoy\") pod \"e2e-inf-b-84b89f-openshift-default-f7d86d855-mddrw\" (UID: \"b88fa1c1-bbb8-4f74-bc2a-3cff7b647082\") " pod="openshift-ingress/e2e-inf-b-84b89f-openshift-default-f7d86d855-mddrw"
Oct 30 00:58:00.646404 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:00.634396    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b88fa1c1-bbb8-4f74-bc2a-3cff7b647082-credential-socket\") pod \"e2e-inf-b-84b89f-openshift-default-f7d86d855-mddrw\" (UID: \"b88fa1c1-bbb8-4f74-bc2a-3cff7b647082\") " pod="openshift-ingress/e2e-inf-b-84b89f-openshift-default-f7d86d855-mddrw"
Oct 30 00:58:00.647929 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:00.647899    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b88fa1c1-bbb8-4f74-bc2a-3cff7b647082-workload-certs\") pod \"e2e-inf-b-84b89f-openshift-default-f7d86d855-mddrw\" (UID: \"b88fa1c1-bbb8-4f74-bc2a-3cff7b647082\") " pod="openshift-ingress/e2e-inf-b-84b89f-openshift-default-f7d86d855-mddrw"
Oct 30 00:58:00.648116 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:00.648099    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b88fa1c1-bbb8-4f74-bc2a-3cff7b647082-istiod-ca-cert\") pod \"e2e-inf-b-84b89f-openshift-default-f7d86d855-mddrw\" (UID: \"b88fa1c1-bbb8-4f74-bc2a-3cff7b647082\") " pod="openshift-ingress/e2e-inf-b-84b89f-openshift-default-f7d86d855-mddrw"
Oct 30 00:58:00.648231 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:00.648217    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b88fa1c1-bbb8-4f74-bc2a-3cff7b647082-workload-socket\") pod \"e2e-inf-b-84b89f-openshift-default-f7d86d855-mddrw\" (UID: \"b88fa1c1-bbb8-4f74-bc2a-3cff7b647082\") " pod="openshift-ingress/e2e-inf-b-84b89f-openshift-default-f7d86d855-mddrw"
Oct 30 00:58:00.648508 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:00.648487    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b88fa1c1-bbb8-4f74-bc2a-3cff7b647082-istio-podinfo\") pod \"e2e-inf-b-84b89f-openshift-default-f7d86d855-mddrw\" (UID: \"b88fa1c1-bbb8-4f74-bc2a-3cff7b647082\") " pod="openshift-ingress/e2e-inf-b-84b89f-openshift-default-f7d86d855-mddrw"
Oct 30 00:58:00.648739 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:00.648719    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pgz4\" (UniqueName: \"kubernetes.io/projected/b88fa1c1-bbb8-4f74-bc2a-3cff7b647082-kube-api-access-5pgz4\") pod \"e2e-inf-b-84b89f-openshift-default-f7d86d855-mddrw\" (UID: \"b88fa1c1-bbb8-4f74-bc2a-3cff7b647082\") " pod="openshift-ingress/e2e-inf-b-84b89f-openshift-default-f7d86d855-mddrw"
Oct 30 00:58:00.648990 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:00.648971    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b88fa1c1-bbb8-4f74-bc2a-3cff7b647082-istio-data\") pod \"e2e-inf-b-84b89f-openshift-default-f7d86d855-mddrw\" (UID: \"b88fa1c1-bbb8-4f74-bc2a-3cff7b647082\") " pod="openshift-ingress/e2e-inf-b-84b89f-openshift-default-f7d86d855-mddrw"
Oct 30 00:58:00.662573 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:00.649518    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b88fa1c1-bbb8-4f74-bc2a-3cff7b647082-istio-data\") pod \"e2e-inf-b-84b89f-openshift-default-f7d86d855-mddrw\" (UID: \"b88fa1c1-bbb8-4f74-bc2a-3cff7b647082\") " pod="openshift-ingress/e2e-inf-b-84b89f-openshift-default-f7d86d855-mddrw"
Oct 30 00:58:00.664562 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:00.647018    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b88fa1c1-bbb8-4f74-bc2a-3cff7b647082-credential-socket\") pod \"e2e-inf-b-84b89f-openshift-default-f7d86d855-mddrw\" (UID: \"b88fa1c1-bbb8-4f74-bc2a-3cff7b647082\") " pod="openshift-ingress/e2e-inf-b-84b89f-openshift-default-f7d86d855-mddrw"
Oct 30 00:58:00.665322 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:00.665288    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b88fa1c1-bbb8-4f74-bc2a-3cff7b647082-workload-socket\") pod \"e2e-inf-b-84b89f-openshift-default-f7d86d855-mddrw\" (UID: \"b88fa1c1-bbb8-4f74-bc2a-3cff7b647082\") " pod="openshift-ingress/e2e-inf-b-84b89f-openshift-default-f7d86d855-mddrw"
Oct 30 00:58:00.665757 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:00.665734    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b88fa1c1-bbb8-4f74-bc2a-3cff7b647082-workload-certs\") pod \"e2e-inf-b-84b89f-openshift-default-f7d86d855-mddrw\" (UID: \"b88fa1c1-bbb8-4f74-bc2a-3cff7b647082\") " pod="openshift-ingress/e2e-inf-b-84b89f-openshift-default-f7d86d855-mddrw"
Oct 30 00:58:00.675640 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:00.675609    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b88fa1c1-bbb8-4f74-bc2a-3cff7b647082-istiod-ca-cert\") pod \"e2e-inf-b-84b89f-openshift-default-f7d86d855-mddrw\" (UID: \"b88fa1c1-bbb8-4f74-bc2a-3cff7b647082\") " pod="openshift-ingress/e2e-inf-b-84b89f-openshift-default-f7d86d855-mddrw"
Oct 30 00:58:00.687399 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:00.687225    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b88fa1c1-bbb8-4f74-bc2a-3cff7b647082-istio-token\") pod \"e2e-inf-b-84b89f-openshift-default-f7d86d855-mddrw\" (UID: \"b88fa1c1-bbb8-4f74-bc2a-3cff7b647082\") " pod="openshift-ingress/e2e-inf-b-84b89f-openshift-default-f7d86d855-mddrw"
Oct 30 00:58:00.692330 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:00.692300    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b88fa1c1-bbb8-4f74-bc2a-3cff7b647082-istio-podinfo\") pod \"e2e-inf-b-84b89f-openshift-default-f7d86d855-mddrw\" (UID: \"b88fa1c1-bbb8-4f74-bc2a-3cff7b647082\") " pod="openshift-ingress/e2e-inf-b-84b89f-openshift-default-f7d86d855-mddrw"
Oct 30 00:58:00.724225 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:00.724150    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pgz4\" (UniqueName: \"kubernetes.io/projected/b88fa1c1-bbb8-4f74-bc2a-3cff7b647082-kube-api-access-5pgz4\") pod \"e2e-inf-b-84b89f-openshift-default-f7d86d855-mddrw\" (UID: \"b88fa1c1-bbb8-4f74-bc2a-3cff7b647082\") " pod="openshift-ingress/e2e-inf-b-84b89f-openshift-default-f7d86d855-mddrw"
Oct 30 00:58:00.747402 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:00.743682    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b88fa1c1-bbb8-4f74-bc2a-3cff7b647082-istio-envoy\") pod \"e2e-inf-b-84b89f-openshift-default-f7d86d855-mddrw\" (UID: \"b88fa1c1-bbb8-4f74-bc2a-3cff7b647082\") " pod="openshift-ingress/e2e-inf-b-84b89f-openshift-default-f7d86d855-mddrw"
Oct 30 00:58:00.962561 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:00.962523    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-84b89f-openshift-default-f7d86d855-mddrw"
Oct 30 00:58:01.324942 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:01.324886    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-84b89f-openshift-default-f7d86d855-mddrw"]
Oct 30 00:58:01.329807 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:58:01.329776    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb88fa1c1_bbb8_4f74_bc2a_3cff7b647082.slice/crio-86cf8a2d24ddfadc3c0f8e564c1a0ece855be88972a81cfe40521c7b0164f927 WatchSource:0}: Error finding container 86cf8a2d24ddfadc3c0f8e564c1a0ece855be88972a81cfe40521c7b0164f927: Status 404 returned error can't find the container with id 86cf8a2d24ddfadc3c0f8e564c1a0ece855be88972a81cfe40521c7b0164f927
Oct 30 00:58:01.332691 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:01.332654    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 00:58:01.332793 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:01.332739    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 00:58:01.332793 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:01.332770    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 00:58:02.155389 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:02.155333    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-84b89f-openshift-default-f7d86d855-mddrw" event={"ID":"b88fa1c1-bbb8-4f74-bc2a-3cff7b647082","Type":"ContainerStarted","Data":"b2b29a7f9905299ed60ad98fe7132fd97f68645f1333a030245fae1aa7c2c409"}
Oct 30 00:58:02.155616 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:02.155435    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-84b89f-openshift-default-f7d86d855-mddrw" event={"ID":"b88fa1c1-bbb8-4f74-bc2a-3cff7b647082","Type":"ContainerStarted","Data":"86cf8a2d24ddfadc3c0f8e564c1a0ece855be88972a81cfe40521c7b0164f927"}
Oct 30 00:58:02.220949 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:02.220864    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-84b89f-openshift-default-f7d86d855-mddrw" podStartSLOduration=2.220847952 podStartE2EDuration="2.220847952s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:58:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:58:02.21869365 +0000 UTC m=+1577.835126564" watchObservedRunningTime="2026-10-30 00:58:02.220847952 +0000 UTC m=+1577.837280904"
Oct 30 00:58:02.963155 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:02.963119    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-84b89f-openshift-default-f7d86d855-mddrw"
Oct 30 00:58:02.970938 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:02.970907    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-84b89f-openshift-default-f7d86d855-mddrw"
Oct 30 00:58:03.177021 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:03.176591    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-84b89f-openshift-default-f7d86d855-mddrw"
Oct 30 00:58:03.181411 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:03.177843    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-84b89f-openshift-default-f7d86d855-mddrw"
Oct 30 00:58:12.864574 ip-10-0-128-7 kubenswrapper[5002]: E1030 00:58:12.864530    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbjdq" containerName="extractor"
Oct 30 00:58:19.030143 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:19.030108    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e026345b-r64w7"]
Oct 30 00:58:19.043714 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:19.043684    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e026345b-r64w7"]
Oct 30 00:58:20.497500 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:20.497462    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="717cdfa3-84f9-47b2-bc60-0aee1b494a82" path="/var/lib/kubelet/pods/717cdfa3-84f9-47b2-bc60-0aee1b494a82/volumes"
Oct 30 00:58:22.597977 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:22.597923    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-b3bdf9-openshift-default-755bccddf-cb9sz"]
Oct 30 00:58:22.603203 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:22.603182    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b3bdf9-openshift-default-755bccddf-cb9sz"
Oct 30 00:58:22.628424 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:22.628391    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-b3bdf9-openshift-default-755bccddf-cb9sz"]
Oct 30 00:58:22.637109 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:22.637080    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-b3bdf9-openshift-default-dockercfg-fqtc6\""
Oct 30 00:58:22.737096 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:22.737056    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6dcd8f60-f8b4-476e-951c-21736a8d5255-istio-envoy\") pod \"e2e-inf-b-b3bdf9-openshift-default-755bccddf-cb9sz\" (UID: \"6dcd8f60-f8b4-476e-951c-21736a8d5255\") " pod="openshift-ingress/e2e-inf-b-b3bdf9-openshift-default-755bccddf-cb9sz"
Oct 30 00:58:22.737284 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:22.737126    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75g24\" (UniqueName: \"kubernetes.io/projected/6dcd8f60-f8b4-476e-951c-21736a8d5255-kube-api-access-75g24\") pod \"e2e-inf-b-b3bdf9-openshift-default-755bccddf-cb9sz\" (UID: \"6dcd8f60-f8b4-476e-951c-21736a8d5255\") " pod="openshift-ingress/e2e-inf-b-b3bdf9-openshift-default-755bccddf-cb9sz"
Oct 30 00:58:22.737284 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:22.737226    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6dcd8f60-f8b4-476e-951c-21736a8d5255-istiod-ca-cert\") pod \"e2e-inf-b-b3bdf9-openshift-default-755bccddf-cb9sz\" (UID: \"6dcd8f60-f8b4-476e-951c-21736a8d5255\") " pod="openshift-ingress/e2e-inf-b-b3bdf9-openshift-default-755bccddf-cb9sz"
Oct 30 00:58:22.737421 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:22.737287    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6dcd8f60-f8b4-476e-951c-21736a8d5255-istio-token\") pod \"e2e-inf-b-b3bdf9-openshift-default-755bccddf-cb9sz\" (UID: \"6dcd8f60-f8b4-476e-951c-21736a8d5255\") " pod="openshift-ingress/e2e-inf-b-b3bdf9-openshift-default-755bccddf-cb9sz"
Oct 30 00:58:22.737486 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:22.737419    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dcd8f60-f8b4-476e-951c-21736a8d5255-credential-socket\") pod \"e2e-inf-b-b3bdf9-openshift-default-755bccddf-cb9sz\" (UID: \"6dcd8f60-f8b4-476e-951c-21736a8d5255\") " pod="openshift-ingress/e2e-inf-b-b3bdf9-openshift-default-755bccddf-cb9sz"
Oct 30 00:58:22.737486 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:22.737461    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6dcd8f60-f8b4-476e-951c-21736a8d5255-istio-data\") pod \"e2e-inf-b-b3bdf9-openshift-default-755bccddf-cb9sz\" (UID: \"6dcd8f60-f8b4-476e-951c-21736a8d5255\") " pod="openshift-ingress/e2e-inf-b-b3bdf9-openshift-default-755bccddf-cb9sz"
Oct 30 00:58:22.737619 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:22.737604    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dcd8f60-f8b4-476e-951c-21736a8d5255-workload-socket\") pod \"e2e-inf-b-b3bdf9-openshift-default-755bccddf-cb9sz\" (UID: \"6dcd8f60-f8b4-476e-951c-21736a8d5255\") " pod="openshift-ingress/e2e-inf-b-b3bdf9-openshift-default-755bccddf-cb9sz"
Oct 30 00:58:22.737716 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:22.737698    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6dcd8f60-f8b4-476e-951c-21736a8d5255-workload-certs\") pod \"e2e-inf-b-b3bdf9-openshift-default-755bccddf-cb9sz\" (UID: \"6dcd8f60-f8b4-476e-951c-21736a8d5255\") " pod="openshift-ingress/e2e-inf-b-b3bdf9-openshift-default-755bccddf-cb9sz"
Oct 30 00:58:22.737770 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:22.737730    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6dcd8f60-f8b4-476e-951c-21736a8d5255-istio-podinfo\") pod \"e2e-inf-b-b3bdf9-openshift-default-755bccddf-cb9sz\" (UID: \"6dcd8f60-f8b4-476e-951c-21736a8d5255\") " pod="openshift-ingress/e2e-inf-b-b3bdf9-openshift-default-755bccddf-cb9sz"
Oct 30 00:58:22.839124 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:22.839086    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6dcd8f60-f8b4-476e-951c-21736a8d5255-istio-envoy\") pod \"e2e-inf-b-b3bdf9-openshift-default-755bccddf-cb9sz\" (UID: \"6dcd8f60-f8b4-476e-951c-21736a8d5255\") " pod="openshift-ingress/e2e-inf-b-b3bdf9-openshift-default-755bccddf-cb9sz"
Oct 30 00:58:22.839317 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:22.839168    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75g24\" (UniqueName: \"kubernetes.io/projected/6dcd8f60-f8b4-476e-951c-21736a8d5255-kube-api-access-75g24\") pod \"e2e-inf-b-b3bdf9-openshift-default-755bccddf-cb9sz\" (UID: \"6dcd8f60-f8b4-476e-951c-21736a8d5255\") " pod="openshift-ingress/e2e-inf-b-b3bdf9-openshift-default-755bccddf-cb9sz"
Oct 30 00:58:22.839317 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:22.839219    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6dcd8f60-f8b4-476e-951c-21736a8d5255-istiod-ca-cert\") pod \"e2e-inf-b-b3bdf9-openshift-default-755bccddf-cb9sz\" (UID: \"6dcd8f60-f8b4-476e-951c-21736a8d5255\") " pod="openshift-ingress/e2e-inf-b-b3bdf9-openshift-default-755bccddf-cb9sz"
Oct 30 00:58:22.839317 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:22.839287    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6dcd8f60-f8b4-476e-951c-21736a8d5255-istio-token\") pod \"e2e-inf-b-b3bdf9-openshift-default-755bccddf-cb9sz\" (UID: \"6dcd8f60-f8b4-476e-951c-21736a8d5255\") " pod="openshift-ingress/e2e-inf-b-b3bdf9-openshift-default-755bccddf-cb9sz"
Oct 30 00:58:22.839511 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:22.839342    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dcd8f60-f8b4-476e-951c-21736a8d5255-credential-socket\") pod \"e2e-inf-b-b3bdf9-openshift-default-755bccddf-cb9sz\" (UID: \"6dcd8f60-f8b4-476e-951c-21736a8d5255\") " pod="openshift-ingress/e2e-inf-b-b3bdf9-openshift-default-755bccddf-cb9sz"
Oct 30 00:58:22.839511 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:22.839372    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6dcd8f60-f8b4-476e-951c-21736a8d5255-istio-data\") pod \"e2e-inf-b-b3bdf9-openshift-default-755bccddf-cb9sz\" (UID: \"6dcd8f60-f8b4-476e-951c-21736a8d5255\") " pod="openshift-ingress/e2e-inf-b-b3bdf9-openshift-default-755bccddf-cb9sz"
Oct 30 00:58:22.839511 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:22.839499    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dcd8f60-f8b4-476e-951c-21736a8d5255-workload-socket\") pod \"e2e-inf-b-b3bdf9-openshift-default-755bccddf-cb9sz\" (UID: \"6dcd8f60-f8b4-476e-951c-21736a8d5255\") " pod="openshift-ingress/e2e-inf-b-b3bdf9-openshift-default-755bccddf-cb9sz"
Oct 30 00:58:22.839676 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:22.839573    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6dcd8f60-f8b4-476e-951c-21736a8d5255-workload-certs\") pod \"e2e-inf-b-b3bdf9-openshift-default-755bccddf-cb9sz\" (UID: \"6dcd8f60-f8b4-476e-951c-21736a8d5255\") " pod="openshift-ingress/e2e-inf-b-b3bdf9-openshift-default-755bccddf-cb9sz"
Oct 30 00:58:22.839676 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:22.839606    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6dcd8f60-f8b4-476e-951c-21736a8d5255-istio-podinfo\") pod \"e2e-inf-b-b3bdf9-openshift-default-755bccddf-cb9sz\" (UID: \"6dcd8f60-f8b4-476e-951c-21736a8d5255\") " pod="openshift-ingress/e2e-inf-b-b3bdf9-openshift-default-755bccddf-cb9sz"
Oct 30 00:58:22.839782 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:22.839730    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dcd8f60-f8b4-476e-951c-21736a8d5255-credential-socket\") pod \"e2e-inf-b-b3bdf9-openshift-default-755bccddf-cb9sz\" (UID: \"6dcd8f60-f8b4-476e-951c-21736a8d5255\") " pod="openshift-ingress/e2e-inf-b-b3bdf9-openshift-default-755bccddf-cb9sz"
Oct 30 00:58:22.839961 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:22.839841    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6dcd8f60-f8b4-476e-951c-21736a8d5255-istio-data\") pod \"e2e-inf-b-b3bdf9-openshift-default-755bccddf-cb9sz\" (UID: \"6dcd8f60-f8b4-476e-951c-21736a8d5255\") " pod="openshift-ingress/e2e-inf-b-b3bdf9-openshift-default-755bccddf-cb9sz"
Oct 30 00:58:22.839961 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:22.839932    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6dcd8f60-f8b4-476e-951c-21736a8d5255-workload-certs\") pod \"e2e-inf-b-b3bdf9-openshift-default-755bccddf-cb9sz\" (UID: \"6dcd8f60-f8b4-476e-951c-21736a8d5255\") " pod="openshift-ingress/e2e-inf-b-b3bdf9-openshift-default-755bccddf-cb9sz"
Oct 30 00:58:22.839961 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:22.839949    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dcd8f60-f8b4-476e-951c-21736a8d5255-workload-socket\") pod \"e2e-inf-b-b3bdf9-openshift-default-755bccddf-cb9sz\" (UID: \"6dcd8f60-f8b4-476e-951c-21736a8d5255\") " pod="openshift-ingress/e2e-inf-b-b3bdf9-openshift-default-755bccddf-cb9sz"
Oct 30 00:58:22.840845 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:22.840824    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6dcd8f60-f8b4-476e-951c-21736a8d5255-istiod-ca-cert\") pod \"e2e-inf-b-b3bdf9-openshift-default-755bccddf-cb9sz\" (UID: \"6dcd8f60-f8b4-476e-951c-21736a8d5255\") " pod="openshift-ingress/e2e-inf-b-b3bdf9-openshift-default-755bccddf-cb9sz"
Oct 30 00:58:22.841971 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:22.841945    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6dcd8f60-f8b4-476e-951c-21736a8d5255-istio-envoy\") pod \"e2e-inf-b-b3bdf9-openshift-default-755bccddf-cb9sz\" (UID: \"6dcd8f60-f8b4-476e-951c-21736a8d5255\") " pod="openshift-ingress/e2e-inf-b-b3bdf9-openshift-default-755bccddf-cb9sz"
Oct 30 00:58:22.842471 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:22.842454    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6dcd8f60-f8b4-476e-951c-21736a8d5255-istio-podinfo\") pod \"e2e-inf-b-b3bdf9-openshift-default-755bccddf-cb9sz\" (UID: \"6dcd8f60-f8b4-476e-951c-21736a8d5255\") " pod="openshift-ingress/e2e-inf-b-b3bdf9-openshift-default-755bccddf-cb9sz"
Oct 30 00:58:22.863996 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:22.863931    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6dcd8f60-f8b4-476e-951c-21736a8d5255-istio-token\") pod \"e2e-inf-b-b3bdf9-openshift-default-755bccddf-cb9sz\" (UID: \"6dcd8f60-f8b4-476e-951c-21736a8d5255\") " pod="openshift-ingress/e2e-inf-b-b3bdf9-openshift-default-755bccddf-cb9sz"
Oct 30 00:58:22.863996 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:22.863976    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-75g24\" (UniqueName: \"kubernetes.io/projected/6dcd8f60-f8b4-476e-951c-21736a8d5255-kube-api-access-75g24\") pod \"e2e-inf-b-b3bdf9-openshift-default-755bccddf-cb9sz\" (UID: \"6dcd8f60-f8b4-476e-951c-21736a8d5255\") " pod="openshift-ingress/e2e-inf-b-b3bdf9-openshift-default-755bccddf-cb9sz"
Oct 30 00:58:22.918447 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:22.918413    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b3bdf9-openshift-default-755bccddf-cb9sz"
Oct 30 00:58:23.240763 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:58:23.240735    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6dcd8f60_f8b4_476e_951c_21736a8d5255.slice/crio-3101ab809de26e58178acad2959242a0ed2826d42552d27a40d6b0c1f18d0715 WatchSource:0}: Error finding container 3101ab809de26e58178acad2959242a0ed2826d42552d27a40d6b0c1f18d0715: Status 404 returned error can't find the container with id 3101ab809de26e58178acad2959242a0ed2826d42552d27a40d6b0c1f18d0715
Oct 30 00:58:23.243943 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:23.243917    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-b3bdf9-openshift-default-755bccddf-cb9sz"]
Oct 30 00:58:23.244144 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:23.244113    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 00:58:23.244196 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:23.244181    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 00:58:23.244233 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:23.244210    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 00:58:23.400009 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:23.399967    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b3bdf9-openshift-default-755bccddf-cb9sz" event={"ID":"6dcd8f60-f8b4-476e-951c-21736a8d5255","Type":"ContainerStarted","Data":"d5313e0432da2b710ceb81de77388b9eb5f08959c5cc92670745085758d46302"}
Oct 30 00:58:23.400212 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:23.400034    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b3bdf9-openshift-default-755bccddf-cb9sz" event={"ID":"6dcd8f60-f8b4-476e-951c-21736a8d5255","Type":"ContainerStarted","Data":"3101ab809de26e58178acad2959242a0ed2826d42552d27a40d6b0c1f18d0715"}
Oct 30 00:58:23.451147 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:23.451097    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-b3bdf9-openshift-default-755bccddf-cb9sz" podStartSLOduration=1.4510837460000001 podStartE2EDuration="1.451083746s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:58:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:58:23.448553742 +0000 UTC m=+1599.064986622" watchObservedRunningTime="2026-10-30 00:58:23.451083746 +0000 UTC m=+1599.067516625"
Oct 30 00:58:23.918989 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:23.918956    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-b3bdf9-openshift-default-755bccddf-cb9sz"
Oct 30 00:58:24.921005 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:24.920967    5002 patch_prober.go:29] interesting pod/e2e-inf-b-b3bdf9-openshift-default-755bccddf-cb9sz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body=
Oct 30 00:58:24.921518 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:24.921027    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b3bdf9-openshift-default-755bccddf-cb9sz" podUID="6dcd8f60-f8b4-476e-951c-21736a8d5255" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused"
Oct 30 00:58:25.919405 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:25.919077    5002 patch_prober.go:29] interesting pod/e2e-inf-b-b3bdf9-openshift-default-755bccddf-cb9sz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body=
Oct 30 00:58:25.919405 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:25.919152    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b3bdf9-openshift-default-755bccddf-cb9sz" podUID="6dcd8f60-f8b4-476e-951c-21736a8d5255" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused"
Oct 30 00:58:26.919267 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:26.919226    5002 patch_prober.go:29] interesting pod/e2e-inf-b-b3bdf9-openshift-default-755bccddf-cb9sz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body=
Oct 30 00:58:26.919755 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:26.919296    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b3bdf9-openshift-default-755bccddf-cb9sz" podUID="6dcd8f60-f8b4-476e-951c-21736a8d5255" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused"
Oct 30 00:58:27.925403 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:27.924767    5002 patch_prober.go:29] interesting pod/e2e-inf-b-b3bdf9-openshift-default-755bccddf-cb9sz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body=
Oct 30 00:58:27.925403 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:27.924831    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b3bdf9-openshift-default-755bccddf-cb9sz" podUID="6dcd8f60-f8b4-476e-951c-21736a8d5255" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused"
Oct 30 00:58:28.919351 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:28.919313    5002 patch_prober.go:29] interesting pod/e2e-inf-b-b3bdf9-openshift-default-755bccddf-cb9sz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body=
Oct 30 00:58:28.919554 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:28.919366    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b3bdf9-openshift-default-755bccddf-cb9sz" podUID="6dcd8f60-f8b4-476e-951c-21736a8d5255" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused"
Oct 30 00:58:29.919439 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:29.919405    5002 patch_prober.go:29] interesting pod/e2e-inf-b-b3bdf9-openshift-default-755bccddf-cb9sz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body=
Oct 30 00:58:29.919852 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:29.919481    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b3bdf9-openshift-default-755bccddf-cb9sz" podUID="6dcd8f60-f8b4-476e-951c-21736a8d5255" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused"
Oct 30 00:58:30.924446 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:30.924375    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-b3bdf9-openshift-default-755bccddf-cb9sz"
Oct 30 00:58:30.924846 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:30.924825    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-b3bdf9-openshift-default-755bccddf-cb9sz"
Oct 30 00:58:30.925642 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:30.925618    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-b3bdf9-openshift-default-755bccddf-cb9sz"
Oct 30 00:58:37.633465 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:37.633425    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7d8z4"]
Oct 30 00:58:42.887359 ip-10-0-128-7 kubenswrapper[5002]: E1030 00:58:42.887322    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbjdq" containerName="extractor"
Oct 30 00:58:44.189402 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:44.169614    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-b1f6ed-openshift-default-767786fdd5-5gvj4"]
Oct 30 00:58:44.203637 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:44.203610    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-b1f6ed-openshift-default-767786fdd5-5gvj4"]
Oct 30 00:58:44.203943 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:44.203923    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b1f6ed-openshift-default-767786fdd5-5gvj4"
Oct 30 00:58:44.259416 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:44.258849    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a0578e8-4726-4c93-9c21-d09fac94c229-istio-envoy\") pod \"e2e-inf-b-b1f6ed-openshift-default-767786fdd5-5gvj4\" (UID: \"4a0578e8-4726-4c93-9c21-d09fac94c229\") " pod="openshift-ingress/e2e-inf-b-b1f6ed-openshift-default-767786fdd5-5gvj4"
Oct 30 00:58:44.259416 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:44.258918    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a0578e8-4726-4c93-9c21-d09fac94c229-istio-podinfo\") pod \"e2e-inf-b-b1f6ed-openshift-default-767786fdd5-5gvj4\" (UID: \"4a0578e8-4726-4c93-9c21-d09fac94c229\") " pod="openshift-ingress/e2e-inf-b-b1f6ed-openshift-default-767786fdd5-5gvj4"
Oct 30 00:58:44.259416 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:44.258996    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ql2v\" (UniqueName: \"kubernetes.io/projected/4a0578e8-4726-4c93-9c21-d09fac94c229-kube-api-access-9ql2v\") pod \"e2e-inf-b-b1f6ed-openshift-default-767786fdd5-5gvj4\" (UID: \"4a0578e8-4726-4c93-9c21-d09fac94c229\") " pod="openshift-ingress/e2e-inf-b-b1f6ed-openshift-default-767786fdd5-5gvj4"
Oct 30 00:58:44.259416 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:44.259129    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a0578e8-4726-4c93-9c21-d09fac94c229-istio-token\") pod \"e2e-inf-b-b1f6ed-openshift-default-767786fdd5-5gvj4\" (UID: \"4a0578e8-4726-4c93-9c21-d09fac94c229\") " pod="openshift-ingress/e2e-inf-b-b1f6ed-openshift-default-767786fdd5-5gvj4"
Oct 30 00:58:44.259416 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:44.259215    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a0578e8-4726-4c93-9c21-d09fac94c229-workload-certs\") pod \"e2e-inf-b-b1f6ed-openshift-default-767786fdd5-5gvj4\" (UID: \"4a0578e8-4726-4c93-9c21-d09fac94c229\") " pod="openshift-ingress/e2e-inf-b-b1f6ed-openshift-default-767786fdd5-5gvj4"
Oct 30 00:58:44.259416 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:44.259261    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a0578e8-4726-4c93-9c21-d09fac94c229-credential-socket\") pod \"e2e-inf-b-b1f6ed-openshift-default-767786fdd5-5gvj4\" (UID: \"4a0578e8-4726-4c93-9c21-d09fac94c229\") " pod="openshift-ingress/e2e-inf-b-b1f6ed-openshift-default-767786fdd5-5gvj4"
Oct 30 00:58:44.259416 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:44.259325    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a0578e8-4726-4c93-9c21-d09fac94c229-istio-data\") pod \"e2e-inf-b-b1f6ed-openshift-default-767786fdd5-5gvj4\" (UID: \"4a0578e8-4726-4c93-9c21-d09fac94c229\") " pod="openshift-ingress/e2e-inf-b-b1f6ed-openshift-default-767786fdd5-5gvj4"
Oct 30 00:58:44.259416 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:44.259361    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a0578e8-4726-4c93-9c21-d09fac94c229-workload-socket\") pod \"e2e-inf-b-b1f6ed-openshift-default-767786fdd5-5gvj4\" (UID: \"4a0578e8-4726-4c93-9c21-d09fac94c229\") " pod="openshift-ingress/e2e-inf-b-b1f6ed-openshift-default-767786fdd5-5gvj4"
Oct 30 00:58:44.259416 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:44.259424    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a0578e8-4726-4c93-9c21-d09fac94c229-istiod-ca-cert\") pod \"e2e-inf-b-b1f6ed-openshift-default-767786fdd5-5gvj4\" (UID: \"4a0578e8-4726-4c93-9c21-d09fac94c229\") " pod="openshift-ingress/e2e-inf-b-b1f6ed-openshift-default-767786fdd5-5gvj4"
Oct 30 00:58:44.309789 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:44.309755    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-b1f6ed-openshift-default-dockercfg-wjgfj\""
Oct 30 00:58:44.495877 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:44.450817    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a0578e8-4726-4c93-9c21-d09fac94c229-workload-certs\") pod \"e2e-inf-b-b1f6ed-openshift-default-767786fdd5-5gvj4\" (UID: \"4a0578e8-4726-4c93-9c21-d09fac94c229\") " pod="openshift-ingress/e2e-inf-b-b1f6ed-openshift-default-767786fdd5-5gvj4"
Oct 30 00:58:44.495877 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:44.450885    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a0578e8-4726-4c93-9c21-d09fac94c229-credential-socket\") pod \"e2e-inf-b-b1f6ed-openshift-default-767786fdd5-5gvj4\" (UID: \"4a0578e8-4726-4c93-9c21-d09fac94c229\") " pod="openshift-ingress/e2e-inf-b-b1f6ed-openshift-default-767786fdd5-5gvj4"
Oct 30 00:58:44.495877 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:44.450940    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a0578e8-4726-4c93-9c21-d09fac94c229-istio-data\") pod \"e2e-inf-b-b1f6ed-openshift-default-767786fdd5-5gvj4\" (UID: \"4a0578e8-4726-4c93-9c21-d09fac94c229\") " pod="openshift-ingress/e2e-inf-b-b1f6ed-openshift-default-767786fdd5-5gvj4"
Oct 30 00:58:44.495877 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:44.450971    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a0578e8-4726-4c93-9c21-d09fac94c229-workload-socket\") pod \"e2e-inf-b-b1f6ed-openshift-default-767786fdd5-5gvj4\" (UID: \"4a0578e8-4726-4c93-9c21-d09fac94c229\") " pod="openshift-ingress/e2e-inf-b-b1f6ed-openshift-default-767786fdd5-5gvj4"
Oct 30 00:58:44.495877 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:44.451009    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a0578e8-4726-4c93-9c21-d09fac94c229-istiod-ca-cert\") pod \"e2e-inf-b-b1f6ed-openshift-default-767786fdd5-5gvj4\" (UID: \"4a0578e8-4726-4c93-9c21-d09fac94c229\") " pod="openshift-ingress/e2e-inf-b-b1f6ed-openshift-default-767786fdd5-5gvj4"
Oct 30 00:58:44.495877 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:44.451146    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a0578e8-4726-4c93-9c21-d09fac94c229-istio-envoy\") pod \"e2e-inf-b-b1f6ed-openshift-default-767786fdd5-5gvj4\" (UID: \"4a0578e8-4726-4c93-9c21-d09fac94c229\") " pod="openshift-ingress/e2e-inf-b-b1f6ed-openshift-default-767786fdd5-5gvj4"
Oct 30 00:58:44.495877 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:44.451178    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a0578e8-4726-4c93-9c21-d09fac94c229-istio-podinfo\") pod \"e2e-inf-b-b1f6ed-openshift-default-767786fdd5-5gvj4\" (UID: \"4a0578e8-4726-4c93-9c21-d09fac94c229\") " pod="openshift-ingress/e2e-inf-b-b1f6ed-openshift-default-767786fdd5-5gvj4"
Oct 30 00:58:44.495877 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:44.451245    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ql2v\" (UniqueName: \"kubernetes.io/projected/4a0578e8-4726-4c93-9c21-d09fac94c229-kube-api-access-9ql2v\") pod \"e2e-inf-b-b1f6ed-openshift-default-767786fdd5-5gvj4\" (UID: \"4a0578e8-4726-4c93-9c21-d09fac94c229\") " pod="openshift-ingress/e2e-inf-b-b1f6ed-openshift-default-767786fdd5-5gvj4"
Oct 30 00:58:44.495877 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:44.451356    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a0578e8-4726-4c93-9c21-d09fac94c229-istio-token\") pod \"e2e-inf-b-b1f6ed-openshift-default-767786fdd5-5gvj4\" (UID: \"4a0578e8-4726-4c93-9c21-d09fac94c229\") " pod="openshift-ingress/e2e-inf-b-b1f6ed-openshift-default-767786fdd5-5gvj4"
Oct 30 00:58:44.495877 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:44.452110    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a0578e8-4726-4c93-9c21-d09fac94c229-workload-certs\") pod \"e2e-inf-b-b1f6ed-openshift-default-767786fdd5-5gvj4\" (UID: \"4a0578e8-4726-4c93-9c21-d09fac94c229\") " pod="openshift-ingress/e2e-inf-b-b1f6ed-openshift-default-767786fdd5-5gvj4"
Oct 30 00:58:44.495877 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:44.452331    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a0578e8-4726-4c93-9c21-d09fac94c229-credential-socket\") pod \"e2e-inf-b-b1f6ed-openshift-default-767786fdd5-5gvj4\" (UID: \"4a0578e8-4726-4c93-9c21-d09fac94c229\") " pod="openshift-ingress/e2e-inf-b-b1f6ed-openshift-default-767786fdd5-5gvj4"
Oct 30 00:58:44.495877 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:44.452554    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a0578e8-4726-4c93-9c21-d09fac94c229-istio-data\") pod \"e2e-inf-b-b1f6ed-openshift-default-767786fdd5-5gvj4\" (UID: \"4a0578e8-4726-4c93-9c21-d09fac94c229\") " pod="openshift-ingress/e2e-inf-b-b1f6ed-openshift-default-767786fdd5-5gvj4"
Oct 30 00:58:44.495877 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:44.452753    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a0578e8-4726-4c93-9c21-d09fac94c229-workload-socket\") pod \"e2e-inf-b-b1f6ed-openshift-default-767786fdd5-5gvj4\" (UID: \"4a0578e8-4726-4c93-9c21-d09fac94c229\") " pod="openshift-ingress/e2e-inf-b-b1f6ed-openshift-default-767786fdd5-5gvj4"
Oct 30 00:58:44.495877 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:44.453293    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a0578e8-4726-4c93-9c21-d09fac94c229-istiod-ca-cert\") pod \"e2e-inf-b-b1f6ed-openshift-default-767786fdd5-5gvj4\" (UID: \"4a0578e8-4726-4c93-9c21-d09fac94c229\") " pod="openshift-ingress/e2e-inf-b-b1f6ed-openshift-default-767786fdd5-5gvj4"
Oct 30 00:58:44.516412 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:44.501978    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a0578e8-4726-4c93-9c21-d09fac94c229-istio-podinfo\") pod \"e2e-inf-b-b1f6ed-openshift-default-767786fdd5-5gvj4\" (UID: \"4a0578e8-4726-4c93-9c21-d09fac94c229\") " pod="openshift-ingress/e2e-inf-b-b1f6ed-openshift-default-767786fdd5-5gvj4"
Oct 30 00:58:44.537847 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:44.537784    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a0578e8-4726-4c93-9c21-d09fac94c229-istio-envoy\") pod \"e2e-inf-b-b1f6ed-openshift-default-767786fdd5-5gvj4\" (UID: \"4a0578e8-4726-4c93-9c21-d09fac94c229\") " pod="openshift-ingress/e2e-inf-b-b1f6ed-openshift-default-767786fdd5-5gvj4"
Oct 30 00:58:44.558415 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:44.542897    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a0578e8-4726-4c93-9c21-d09fac94c229-istio-token\") pod \"e2e-inf-b-b1f6ed-openshift-default-767786fdd5-5gvj4\" (UID: \"4a0578e8-4726-4c93-9c21-d09fac94c229\") " pod="openshift-ingress/e2e-inf-b-b1f6ed-openshift-default-767786fdd5-5gvj4"
Oct 30 00:58:44.643725 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:44.643655    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ql2v\" (UniqueName: \"kubernetes.io/projected/4a0578e8-4726-4c93-9c21-d09fac94c229-kube-api-access-9ql2v\") pod \"e2e-inf-b-b1f6ed-openshift-default-767786fdd5-5gvj4\" (UID: \"4a0578e8-4726-4c93-9c21-d09fac94c229\") " pod="openshift-ingress/e2e-inf-b-b1f6ed-openshift-default-767786fdd5-5gvj4"
Oct 30 00:58:44.845172 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:44.845097    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-b1f6ed-openshift-default-dockercfg-wjgfj\""
Oct 30 00:58:44.854493 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:44.854460    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b1f6ed-openshift-default-767786fdd5-5gvj4"
Oct 30 00:58:45.490361 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:45.489798    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 00:58:45.490361 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:45.489897    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 00:58:45.490361 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:45.489940    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 00:58:45.494162 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:45.494138    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-b1f6ed-openshift-default-767786fdd5-5gvj4"]
Oct 30 00:58:45.772462 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:45.772049    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b1f6ed-openshift-default-767786fdd5-5gvj4" event={"ID":"4a0578e8-4726-4c93-9c21-d09fac94c229","Type":"ContainerStarted","Data":"3ffeb3906c89d012b7dd3d63ff485e9872aa3b6d2cbb6723d33e9b567a928133"}
Oct 30 00:58:45.772462 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:45.772119    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b1f6ed-openshift-default-767786fdd5-5gvj4" event={"ID":"4a0578e8-4726-4c93-9c21-d09fac94c229","Type":"ContainerStarted","Data":"66a100254b44781aee4d9c1537692fcbe49ebdb31d9338f4166016bfb0432e2b"}
Oct 30 00:58:45.856425 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:45.855593    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-b1f6ed-openshift-default-767786fdd5-5gvj4"
Oct 30 00:58:45.900933 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:45.900870    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-b1f6ed-openshift-default-767786fdd5-5gvj4" podStartSLOduration=1.900846783 podStartE2EDuration="1.900846783s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:58:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:58:45.877058088 +0000 UTC m=+1621.493490948" watchObservedRunningTime="2026-10-30 00:58:45.900846783 +0000 UTC m=+1621.517279664"
Oct 30 00:58:46.697064 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:46.697023    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-4989x"]
Oct 30 00:58:46.708708 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:46.708681    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-4989x"
Oct 30 00:58:46.721855 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:46.721823    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-4989x"]
Oct 30 00:58:46.730226 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:46.730198    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-8d2a70-openshift-default-dockercfg-vpjnj\""
Oct 30 00:58:46.795941 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:46.795887    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf2f68d2-28a3-4488-b3c5-82d2e278c311-istiod-ca-cert\") pod \"e2e-inf-a-8d2a70-openshift-default-677474fcd7-4989x\" (UID: \"bf2f68d2-28a3-4488-b3c5-82d2e278c311\") " pod="openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-4989x"
Oct 30 00:58:46.796136 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:46.795999    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf2f68d2-28a3-4488-b3c5-82d2e278c311-istio-token\") pod \"e2e-inf-a-8d2a70-openshift-default-677474fcd7-4989x\" (UID: \"bf2f68d2-28a3-4488-b3c5-82d2e278c311\") " pod="openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-4989x"
Oct 30 00:58:46.796136 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:46.796041    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncxg4\" (UniqueName: \"kubernetes.io/projected/bf2f68d2-28a3-4488-b3c5-82d2e278c311-kube-api-access-ncxg4\") pod \"e2e-inf-a-8d2a70-openshift-default-677474fcd7-4989x\" (UID: \"bf2f68d2-28a3-4488-b3c5-82d2e278c311\") " pod="openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-4989x"
Oct 30 00:58:46.796136 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:46.796087    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf2f68d2-28a3-4488-b3c5-82d2e278c311-workload-certs\") pod \"e2e-inf-a-8d2a70-openshift-default-677474fcd7-4989x\" (UID: \"bf2f68d2-28a3-4488-b3c5-82d2e278c311\") " pod="openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-4989x"
Oct 30 00:58:46.796343 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:46.796251    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf2f68d2-28a3-4488-b3c5-82d2e278c311-istio-data\") pod \"e2e-inf-a-8d2a70-openshift-default-677474fcd7-4989x\" (UID: \"bf2f68d2-28a3-4488-b3c5-82d2e278c311\") " pod="openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-4989x"
Oct 30 00:58:46.796343 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:46.796293    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf2f68d2-28a3-4488-b3c5-82d2e278c311-credential-socket\") pod \"e2e-inf-a-8d2a70-openshift-default-677474fcd7-4989x\" (UID: \"bf2f68d2-28a3-4488-b3c5-82d2e278c311\") " pod="openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-4989x"
Oct 30 00:58:46.796343 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:46.796321    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf2f68d2-28a3-4488-b3c5-82d2e278c311-istio-envoy\") pod \"e2e-inf-a-8d2a70-openshift-default-677474fcd7-4989x\" (UID: \"bf2f68d2-28a3-4488-b3c5-82d2e278c311\") " pod="openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-4989x"
Oct 30 00:58:46.796569 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:46.796453    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf2f68d2-28a3-4488-b3c5-82d2e278c311-workload-socket\") pod \"e2e-inf-a-8d2a70-openshift-default-677474fcd7-4989x\" (UID: \"bf2f68d2-28a3-4488-b3c5-82d2e278c311\") " pod="openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-4989x"
Oct 30 00:58:46.796569 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:46.796526    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf2f68d2-28a3-4488-b3c5-82d2e278c311-istio-podinfo\") pod \"e2e-inf-a-8d2a70-openshift-default-677474fcd7-4989x\" (UID: \"bf2f68d2-28a3-4488-b3c5-82d2e278c311\") " pod="openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-4989x"
Oct 30 00:58:46.863056 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:46.863022    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-b1f6ed-openshift-default-767786fdd5-5gvj4"
Oct 30 00:58:46.897413 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:46.897347    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf2f68d2-28a3-4488-b3c5-82d2e278c311-istio-data\") pod \"e2e-inf-a-8d2a70-openshift-default-677474fcd7-4989x\" (UID: \"bf2f68d2-28a3-4488-b3c5-82d2e278c311\") " pod="openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-4989x"
Oct 30 00:58:46.897592 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:46.897510    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf2f68d2-28a3-4488-b3c5-82d2e278c311-credential-socket\") pod \"e2e-inf-a-8d2a70-openshift-default-677474fcd7-4989x\" (UID: \"bf2f68d2-28a3-4488-b3c5-82d2e278c311\") " pod="openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-4989x"
Oct 30 00:58:46.897592 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:46.897558    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf2f68d2-28a3-4488-b3c5-82d2e278c311-istio-envoy\") pod \"e2e-inf-a-8d2a70-openshift-default-677474fcd7-4989x\" (UID: \"bf2f68d2-28a3-4488-b3c5-82d2e278c311\") " pod="openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-4989x"
Oct 30 00:58:46.897717 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:46.897643    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf2f68d2-28a3-4488-b3c5-82d2e278c311-workload-socket\") pod \"e2e-inf-a-8d2a70-openshift-default-677474fcd7-4989x\" (UID: \"bf2f68d2-28a3-4488-b3c5-82d2e278c311\") " pod="openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-4989x"
Oct 30 00:58:46.897772 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:46.897741    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf2f68d2-28a3-4488-b3c5-82d2e278c311-istio-podinfo\") pod \"e2e-inf-a-8d2a70-openshift-default-677474fcd7-4989x\" (UID: \"bf2f68d2-28a3-4488-b3c5-82d2e278c311\") " pod="openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-4989x"
Oct 30 00:58:46.897830 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:46.897764    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf2f68d2-28a3-4488-b3c5-82d2e278c311-istio-data\") pod \"e2e-inf-a-8d2a70-openshift-default-677474fcd7-4989x\" (UID: \"bf2f68d2-28a3-4488-b3c5-82d2e278c311\") " pod="openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-4989x"
Oct 30 00:58:46.897830 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:46.897803    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf2f68d2-28a3-4488-b3c5-82d2e278c311-credential-socket\") pod \"e2e-inf-a-8d2a70-openshift-default-677474fcd7-4989x\" (UID: \"bf2f68d2-28a3-4488-b3c5-82d2e278c311\") " pod="openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-4989x"
Oct 30 00:58:46.897942 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:46.897882    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf2f68d2-28a3-4488-b3c5-82d2e278c311-istiod-ca-cert\") pod \"e2e-inf-a-8d2a70-openshift-default-677474fcd7-4989x\" (UID: \"bf2f68d2-28a3-4488-b3c5-82d2e278c311\") " pod="openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-4989x"
Oct 30 00:58:46.898000 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:46.897940    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf2f68d2-28a3-4488-b3c5-82d2e278c311-istio-token\") pod \"e2e-inf-a-8d2a70-openshift-default-677474fcd7-4989x\" (UID: \"bf2f68d2-28a3-4488-b3c5-82d2e278c311\") " pod="openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-4989x"
Oct 30 00:58:46.898000 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:46.897978    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncxg4\" (UniqueName: \"kubernetes.io/projected/bf2f68d2-28a3-4488-b3c5-82d2e278c311-kube-api-access-ncxg4\") pod \"e2e-inf-a-8d2a70-openshift-default-677474fcd7-4989x\" (UID: \"bf2f68d2-28a3-4488-b3c5-82d2e278c311\") " pod="openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-4989x"
Oct 30 00:58:46.898114 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:46.898020    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf2f68d2-28a3-4488-b3c5-82d2e278c311-workload-certs\") pod \"e2e-inf-a-8d2a70-openshift-default-677474fcd7-4989x\" (UID: \"bf2f68d2-28a3-4488-b3c5-82d2e278c311\") " pod="openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-4989x"
Oct 30 00:58:46.898616 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:46.898579    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf2f68d2-28a3-4488-b3c5-82d2e278c311-workload-socket\") pod \"e2e-inf-a-8d2a70-openshift-default-677474fcd7-4989x\" (UID: \"bf2f68d2-28a3-4488-b3c5-82d2e278c311\") " pod="openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-4989x"
Oct 30 00:58:46.898746 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:46.898638    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf2f68d2-28a3-4488-b3c5-82d2e278c311-istiod-ca-cert\") pod \"e2e-inf-a-8d2a70-openshift-default-677474fcd7-4989x\" (UID: \"bf2f68d2-28a3-4488-b3c5-82d2e278c311\") " pod="openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-4989x"
Oct 30 00:58:46.898811 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:46.898783    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf2f68d2-28a3-4488-b3c5-82d2e278c311-workload-certs\") pod \"e2e-inf-a-8d2a70-openshift-default-677474fcd7-4989x\" (UID: \"bf2f68d2-28a3-4488-b3c5-82d2e278c311\") " pod="openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-4989x"
Oct 30 00:58:46.900481 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:46.900459    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf2f68d2-28a3-4488-b3c5-82d2e278c311-istio-podinfo\") pod \"e2e-inf-a-8d2a70-openshift-default-677474fcd7-4989x\" (UID: \"bf2f68d2-28a3-4488-b3c5-82d2e278c311\") " pod="openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-4989x"
Oct 30 00:58:46.900867 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:46.900843    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf2f68d2-28a3-4488-b3c5-82d2e278c311-istio-envoy\") pod \"e2e-inf-a-8d2a70-openshift-default-677474fcd7-4989x\" (UID: \"bf2f68d2-28a3-4488-b3c5-82d2e278c311\") " pod="openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-4989x"
Oct 30 00:58:46.922929 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:46.922899    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf2f68d2-28a3-4488-b3c5-82d2e278c311-istio-token\") pod \"e2e-inf-a-8d2a70-openshift-default-677474fcd7-4989x\" (UID: \"bf2f68d2-28a3-4488-b3c5-82d2e278c311\") " pod="openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-4989x"
Oct 30 00:58:46.923518 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:46.923493    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncxg4\" (UniqueName: \"kubernetes.io/projected/bf2f68d2-28a3-4488-b3c5-82d2e278c311-kube-api-access-ncxg4\") pod \"e2e-inf-a-8d2a70-openshift-default-677474fcd7-4989x\" (UID: \"bf2f68d2-28a3-4488-b3c5-82d2e278c311\") " pod="openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-4989x"
Oct 30 00:58:47.032756 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:47.032655    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-4989x"
Oct 30 00:58:47.462730 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:58:47.462701    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf2f68d2_28a3_4488_b3c5_82d2e278c311.slice/crio-ddaeffcdaad623856514cb3f81997d6166dec591ea445808af9e832374dfaee0 WatchSource:0}: Error finding container ddaeffcdaad623856514cb3f81997d6166dec591ea445808af9e832374dfaee0: Status 404 returned error can't find the container with id ddaeffcdaad623856514cb3f81997d6166dec591ea445808af9e832374dfaee0
Oct 30 00:58:47.466130 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:47.466092    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 00:58:47.466262 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:47.466175    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 00:58:47.466262 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:47.466222    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 00:58:47.466468 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:47.466441    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-4989x"]
Oct 30 00:58:47.796705 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:47.796607    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-4989x" event={"ID":"bf2f68d2-28a3-4488-b3c5-82d2e278c311","Type":"ContainerStarted","Data":"fb68e95a1d2cd04cf1310ef73c36ee324858282514e4098464b09faced7c7239"}
Oct 30 00:58:47.797413 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:47.797335    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-b1f6ed-openshift-default-767786fdd5-5gvj4"
Oct 30 00:58:47.797413 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:47.797400    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-4989x" event={"ID":"bf2f68d2-28a3-4488-b3c5-82d2e278c311","Type":"ContainerStarted","Data":"ddaeffcdaad623856514cb3f81997d6166dec591ea445808af9e832374dfaee0"}
Oct 30 00:58:47.797749 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:47.797725    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-b1f6ed-openshift-default-767786fdd5-5gvj4"
Oct 30 00:58:47.847508 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:47.847435    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-4989x" podStartSLOduration=1.847415354 podStartE2EDuration="1.847415354s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:58:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:58:47.84554533 +0000 UTC m=+1623.461978236" watchObservedRunningTime="2026-10-30 00:58:47.847415354 +0000 UTC m=+1623.463848233"
Oct 30 00:58:48.033351 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:48.033312    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-4989x"
Oct 30 00:58:48.034896 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:48.034873    5002 patch_prober.go:29] interesting pod/e2e-inf-a-8d2a70-openshift-default-677474fcd7-4989x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body=
Oct 30 00:58:48.034997 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:48.034925    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-4989x" podUID="bf2f68d2-28a3-4488-b3c5-82d2e278c311" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused"
Oct 30 00:58:49.033320 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:49.033259    5002 patch_prober.go:29] interesting pod/e2e-inf-a-8d2a70-openshift-default-677474fcd7-4989x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body=
Oct 30 00:58:49.033981 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:49.033950    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-4989x" podUID="bf2f68d2-28a3-4488-b3c5-82d2e278c311" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused"
Oct 30 00:58:50.033224 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:50.033183    5002 patch_prober.go:29] interesting pod/e2e-inf-a-8d2a70-openshift-default-677474fcd7-4989x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body=
Oct 30 00:58:50.033439 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:50.033249    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-4989x" podUID="bf2f68d2-28a3-4488-b3c5-82d2e278c311" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused"
Oct 30 00:58:50.120835 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:50.120797    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fa8dd0-mrmmc"]
Oct 30 00:58:50.126830 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:50.126793    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fa8dd0-mrmmc"]
Oct 30 00:58:50.500515 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:50.500477    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15b4eff7-8879-4a7d-8a30-19016d0d0b35" path="/var/lib/kubelet/pods/15b4eff7-8879-4a7d-8a30-19016d0d0b35/volumes"
Oct 30 00:58:50.711545 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:50.711507    5002 patch_prober.go:29] interesting pod/e2e-inf-a-51b95e-openshift-default-74bf6d5cb5-swcm7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 00:58:50.711710 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:50.711576    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-51b95e-openshift-default-74bf6d5cb5-swcm7" podUID="2bb71dc1-0795-464e-9887-a6b5ce483890" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:58:51.033547 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:51.033511    5002 patch_prober.go:29] interesting pod/e2e-inf-a-8d2a70-openshift-default-677474fcd7-4989x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body=
Oct 30 00:58:51.033547 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:51.033576    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-4989x" podUID="bf2f68d2-28a3-4488-b3c5-82d2e278c311" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused"
Oct 30 00:58:52.034527 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:52.034486    5002 patch_prober.go:29] interesting pod/e2e-inf-a-8d2a70-openshift-default-677474fcd7-4989x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body=
Oct 30 00:58:52.034949 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:52.034553    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-4989x" podUID="bf2f68d2-28a3-4488-b3c5-82d2e278c311" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused"
Oct 30 00:58:52.914833 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:52.914798    5002 generic.go:356] "Generic (PLEG): container finished" podID="2bb71dc1-0795-464e-9887-a6b5ce483890" containerID="525f9d8e4f3147eacee1dd4e21683a0a0d59733dec2dde54440f3169f819f90a" exitCode=0
Oct 30 00:58:52.915058 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:52.914877    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-51b95e-openshift-default-74bf6d5cb5-swcm7" event={"ID":"2bb71dc1-0795-464e-9887-a6b5ce483890","Type":"ContainerDied","Data":"525f9d8e4f3147eacee1dd4e21683a0a0d59733dec2dde54440f3169f819f90a"}
Oct 30 00:58:52.915499 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:52.915477    5002 scope.go:122] "RemoveContainer" containerID="525f9d8e4f3147eacee1dd4e21683a0a0d59733dec2dde54440f3169f819f90a"
Oct 30 00:58:52.917485 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:52.917460    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 00:58:52.917564 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:52.917529    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 00:58:52.917600 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:52.917562    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 00:58:53.034068 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:53.034043    5002 patch_prober.go:29] interesting pod/e2e-inf-a-8d2a70-openshift-default-677474fcd7-4989x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body=
Oct 30 00:58:53.034228 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:53.034092    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-4989x" podUID="bf2f68d2-28a3-4488-b3c5-82d2e278c311" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused"
Oct 30 00:58:53.926796 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:53.926762    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-51b95e-openshift-default-74bf6d5cb5-swcm7" event={"ID":"2bb71dc1-0795-464e-9887-a6b5ce483890","Type":"ContainerStarted","Data":"287ceb9a066879306acc32029703e4c5c72d27e5212a8150a5cbf3e5368df071"}
Oct 30 00:58:54.033190 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:54.033147    5002 patch_prober.go:29] interesting pod/e2e-inf-a-8d2a70-openshift-default-677474fcd7-4989x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body=
Oct 30 00:58:54.033364 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:54.033208    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-4989x" podUID="bf2f68d2-28a3-4488-b3c5-82d2e278c311" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused"
Oct 30 00:58:54.321450 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:54.312712    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-51b95e-openshift-default-74bf6d5cb5-swcm7"
Oct 30 00:58:54.321450 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:54.313464    5002 patch_prober.go:29] interesting pod/e2e-inf-a-51b95e-openshift-default-74bf6d5cb5-swcm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body=
Oct 30 00:58:54.321450 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:54.318418    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-51b95e-openshift-default-74bf6d5cb5-swcm7" podUID="2bb71dc1-0795-464e-9887-a6b5ce483890" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused"
Oct 30 00:58:55.037873 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:55.037844    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-4989x"
Oct 30 00:58:55.038365 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:55.038332    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-4989x"
Oct 30 00:58:55.039139 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:55.039119    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-4989x"
Oct 30 00:58:55.300776 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:55.300690    5002 patch_prober.go:29] interesting pod/e2e-inf-a-51b95e-openshift-default-74bf6d5cb5-swcm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body=
Oct 30 00:58:55.301061 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:55.301030    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-51b95e-openshift-default-74bf6d5cb5-swcm7" podUID="2bb71dc1-0795-464e-9887-a6b5ce483890" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused"
Oct 30 00:58:56.303406 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:56.302758    5002 patch_prober.go:29] interesting pod/e2e-inf-a-51b95e-openshift-default-74bf6d5cb5-swcm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body=
Oct 30 00:58:56.303406 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:56.302875    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-51b95e-openshift-default-74bf6d5cb5-swcm7" podUID="2bb71dc1-0795-464e-9887-a6b5ce483890" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused"
Oct 30 00:58:57.300521 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:57.300484    5002 patch_prober.go:29] interesting pod/e2e-inf-a-51b95e-openshift-default-74bf6d5cb5-swcm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body=
Oct 30 00:58:57.300697 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:57.300550    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-51b95e-openshift-default-74bf6d5cb5-swcm7" podUID="2bb71dc1-0795-464e-9887-a6b5ce483890" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused"
Oct 30 00:58:58.305403 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:58.304962    5002 patch_prober.go:29] interesting pod/e2e-inf-a-51b95e-openshift-default-74bf6d5cb5-swcm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body=
Oct 30 00:58:58.305403 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:58.305021    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-51b95e-openshift-default-74bf6d5cb5-swcm7" podUID="2bb71dc1-0795-464e-9887-a6b5ce483890" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused"
Oct 30 00:58:58.620236 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:58.620144    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bab19a-nrvmd"]
Oct 30 00:58:58.645081 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:58.644051    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bab19a-nrvmd"]
Oct 30 00:58:58.645081 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:58.644177    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bab19a-nrvmd"
Oct 30 00:58:58.710744 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:58.710602    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 00:58:58.715682 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:58.715654    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 00:58:58.715843 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:58.715827    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 00:58:58.715912 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:58.715896    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5phsh\""
Oct 30 00:58:58.821004 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:58.820968    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xh4zd\" (UniqueName: \"kubernetes.io/projected/b780a91d-c4f6-4c5a-834b-ec5a686bae51-kube-api-access-xh4zd\") pod \"maas-api-revoke-keys-e2e-api-b-bab19a-nrvmd\" (UID: \"b780a91d-c4f6-4c5a-834b-ec5a686bae51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bab19a-nrvmd"
Oct 30 00:58:58.821170 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:58.821079    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b780a91d-c4f6-4c5a-834b-ec5a686bae51-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-bab19a-nrvmd\" (UID: \"b780a91d-c4f6-4c5a-834b-ec5a686bae51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bab19a-nrvmd"
Oct 30 00:58:58.821267 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:58.821250    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b780a91d-c4f6-4c5a-834b-ec5a686bae51-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-bab19a-nrvmd\" (UID: \"b780a91d-c4f6-4c5a-834b-ec5a686bae51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bab19a-nrvmd"
Oct 30 00:58:58.923028 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:58.922941    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b780a91d-c4f6-4c5a-834b-ec5a686bae51-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-bab19a-nrvmd\" (UID: \"b780a91d-c4f6-4c5a-834b-ec5a686bae51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bab19a-nrvmd"
Oct 30 00:58:58.923167 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:58.923102    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b780a91d-c4f6-4c5a-834b-ec5a686bae51-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-bab19a-nrvmd\" (UID: \"b780a91d-c4f6-4c5a-834b-ec5a686bae51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bab19a-nrvmd"
Oct 30 00:58:58.923276 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:58.923259    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xh4zd\" (UniqueName: \"kubernetes.io/projected/b780a91d-c4f6-4c5a-834b-ec5a686bae51-kube-api-access-xh4zd\") pod \"maas-api-revoke-keys-e2e-api-b-bab19a-nrvmd\" (UID: \"b780a91d-c4f6-4c5a-834b-ec5a686bae51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bab19a-nrvmd"
Oct 30 00:58:58.923691 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:58.923661    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b780a91d-c4f6-4c5a-834b-ec5a686bae51-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-bab19a-nrvmd\" (UID: \"b780a91d-c4f6-4c5a-834b-ec5a686bae51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bab19a-nrvmd"
Oct 30 00:58:58.926919 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:58.926896    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b780a91d-c4f6-4c5a-834b-ec5a686bae51-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-bab19a-nrvmd\" (UID: \"b780a91d-c4f6-4c5a-834b-ec5a686bae51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bab19a-nrvmd"
Oct 30 00:58:58.947929 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:58.947899    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xh4zd\" (UniqueName: \"kubernetes.io/projected/b780a91d-c4f6-4c5a-834b-ec5a686bae51-kube-api-access-xh4zd\") pod \"maas-api-revoke-keys-e2e-api-b-bab19a-nrvmd\" (UID: \"b780a91d-c4f6-4c5a-834b-ec5a686bae51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bab19a-nrvmd"
Oct 30 00:58:59.023062 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:59.023027    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bab19a-nrvmd"
Oct 30 00:58:59.299672 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:59.299630    5002 patch_prober.go:29] interesting pod/e2e-inf-a-51b95e-openshift-default-74bf6d5cb5-swcm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body=
Oct 30 00:58:59.299877 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:59.299693    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-51b95e-openshift-default-74bf6d5cb5-swcm7" podUID="2bb71dc1-0795-464e-9887-a6b5ce483890" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused"
Oct 30 00:58:59.389202 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:58:59.389140    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bab19a-nrvmd"]
Oct 30 00:59:00.245982 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:00.245941    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bab19a-nrvmd" event={"ID":"b780a91d-c4f6-4c5a-834b-ec5a686bae51","Type":"ContainerStarted","Data":"16bd7ee14049867db2b6e6436c1b25286ba975787ec688d92c4b3b67f856edbe"}
Oct 30 00:59:00.246154 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:00.246029    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bab19a-nrvmd" event={"ID":"b780a91d-c4f6-4c5a-834b-ec5a686bae51","Type":"ContainerStarted","Data":"70ac7d581f02ff190cc8133b92c93b96e147a952c08d7167eb9c33880fcf9336"}
Oct 30 00:59:00.290898 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:00.290835    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bab19a-nrvmd" podStartSLOduration=2.290820723 podStartE2EDuration="2.290820723s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:58:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:59:00.289482935 +0000 UTC m=+1635.905915832" watchObservedRunningTime="2026-10-30 00:59:00.290820723 +0000 UTC m=+1635.907253625"
Oct 30 00:59:00.305663 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:00.305630    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-51b95e-openshift-default-74bf6d5cb5-swcm7"
Oct 30 00:59:00.306030 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:00.306005    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-51b95e-openshift-default-74bf6d5cb5-swcm7"
Oct 30 00:59:00.306806 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:00.306787    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-51b95e-openshift-default-74bf6d5cb5-swcm7"
Oct 30 00:59:01.254882 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:01.254846    5002 generic.go:356] "Generic (PLEG): container finished" podID="b780a91d-c4f6-4c5a-834b-ec5a686bae51" containerID="16bd7ee14049867db2b6e6436c1b25286ba975787ec688d92c4b3b67f856edbe" exitCode=0
Oct 30 00:59:01.255367 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:01.254929    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bab19a-nrvmd" event={"ID":"b780a91d-c4f6-4c5a-834b-ec5a686bae51","Type":"ContainerDied","Data":"16bd7ee14049867db2b6e6436c1b25286ba975787ec688d92c4b3b67f856edbe"}
Oct 30 00:59:02.525435 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:02.525363    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bab19a-nrvmd"
Oct 30 00:59:02.608749 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:02.608711    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b780a91d-c4f6-4c5a-834b-ec5a686bae51-kube-api-access-xh4zd\" (UniqueName: \"kubernetes.io/projected/b780a91d-c4f6-4c5a-834b-ec5a686bae51-kube-api-access-xh4zd\") pod \"b780a91d-c4f6-4c5a-834b-ec5a686bae51\" (UID: \"b780a91d-c4f6-4c5a-834b-ec5a686bae51\") "
Oct 30 00:59:02.608933 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:02.608764    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b780a91d-c4f6-4c5a-834b-ec5a686bae51-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b780a91d-c4f6-4c5a-834b-ec5a686bae51-maas-api-xks-ca\") pod \"b780a91d-c4f6-4c5a-834b-ec5a686bae51\" (UID: \"b780a91d-c4f6-4c5a-834b-ec5a686bae51\") "
Oct 30 00:59:02.608933 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:02.608839    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b780a91d-c4f6-4c5a-834b-ec5a686bae51-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b780a91d-c4f6-4c5a-834b-ec5a686bae51-maas-api-openshift-ca\") pod \"b780a91d-c4f6-4c5a-834b-ec5a686bae51\" (UID: \"b780a91d-c4f6-4c5a-834b-ec5a686bae51\") "
Oct 30 00:59:02.609283 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:02.609254    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b780a91d-c4f6-4c5a-834b-ec5a686bae51-maas-api-openshift-ca" pod "b780a91d-c4f6-4c5a-834b-ec5a686bae51" (UID: "b780a91d-c4f6-4c5a-834b-ec5a686bae51"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:59:02.609896 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:02.609873    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b780a91d-c4f6-4c5a-834b-ec5a686bae51-maas-api-openshift-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:59:02.611005 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:02.610981    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b780a91d-c4f6-4c5a-834b-ec5a686bae51-maas-api-xks-ca" pod "b780a91d-c4f6-4c5a-834b-ec5a686bae51" (UID: "b780a91d-c4f6-4c5a-834b-ec5a686bae51"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:59:02.611309 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:02.611287    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b780a91d-c4f6-4c5a-834b-ec5a686bae51-kube-api-access-xh4zd" pod "b780a91d-c4f6-4c5a-834b-ec5a686bae51" (UID: "b780a91d-c4f6-4c5a-834b-ec5a686bae51"). InnerVolumeSpecName "kube-api-access-xh4zd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:59:02.711369 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:02.711335    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xh4zd\" (UniqueName: \"kubernetes.io/projected/b780a91d-c4f6-4c5a-834b-ec5a686bae51-kube-api-access-xh4zd\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:59:02.711369 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:02.711366    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b780a91d-c4f6-4c5a-834b-ec5a686bae51-maas-api-xks-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:59:03.325370 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:03.325331    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bab19a-nrvmd" event={"ID":"b780a91d-c4f6-4c5a-834b-ec5a686bae51","Type":"ContainerDied","Data":"70ac7d581f02ff190cc8133b92c93b96e147a952c08d7167eb9c33880fcf9336"}
Oct 30 00:59:03.325655 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:03.325634    5002 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="70ac7d581f02ff190cc8133b92c93b96e147a952c08d7167eb9c33880fcf9336"
Oct 30 00:59:03.325822 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:03.325810    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bab19a-nrvmd"
Oct 30 00:59:09.271362 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:09.271318    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-5007e1-d7bb8bb4c-xbqmg"]
Oct 30 00:59:09.272234 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:09.272210    5002 cpu_manager.go:414] "containerMap: removing container" podUID="b780a91d-c4f6-4c5a-834b-ec5a686bae51" containerName="revoke-keys"
Oct 30 00:59:09.272342 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:09.272235    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b780a91d-c4f6-4c5a-834b-ec5a686bae51" containerName="revoke-keys"
Oct 30 00:59:09.272444 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:09.272428    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="b780a91d-c4f6-4c5a-834b-ec5a686bae51" containerName="revoke-keys"
Oct 30 00:59:09.274840 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:09.274819    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5007e1-d7bb8bb4c-xbqmg"
Oct 30 00:59:09.306864 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:09.306830    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-5007e1\""
Oct 30 00:59:09.310507 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:09.310477    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-5007e1-dockercfg-wdbz8\""
Oct 30 00:59:09.313223 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:09.313198    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-5007e1-d7bb8bb4c-xbqmg"]
Oct 30 00:59:09.478407 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:09.475190    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tj54b\" (UniqueName: \"kubernetes.io/projected/fa5f1440-5fa9-40e6-b85f-073f31d24d01-kube-api-access-tj54b\") pod \"payload-pre-processing-e2e-inf-b-5007e1-d7bb8bb4c-xbqmg\" (UID: \"fa5f1440-5fa9-40e6-b85f-073f31d24d01\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5007e1-d7bb8bb4c-xbqmg"
Oct 30 00:59:09.478407 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:09.475320    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa5f1440-5fa9-40e6-b85f-073f31d24d01-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-5007e1-d7bb8bb4c-xbqmg\" (UID: \"fa5f1440-5fa9-40e6-b85f-073f31d24d01\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5007e1-d7bb8bb4c-xbqmg"
Oct 30 00:59:09.622494 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:09.614914    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tj54b\" (UniqueName: \"kubernetes.io/projected/fa5f1440-5fa9-40e6-b85f-073f31d24d01-kube-api-access-tj54b\") pod \"payload-pre-processing-e2e-inf-b-5007e1-d7bb8bb4c-xbqmg\" (UID: \"fa5f1440-5fa9-40e6-b85f-073f31d24d01\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5007e1-d7bb8bb4c-xbqmg"
Oct 30 00:59:09.622494 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:09.615033    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa5f1440-5fa9-40e6-b85f-073f31d24d01-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-5007e1-d7bb8bb4c-xbqmg\" (UID: \"fa5f1440-5fa9-40e6-b85f-073f31d24d01\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5007e1-d7bb8bb4c-xbqmg"
Oct 30 00:59:09.675459 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:09.640160    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa5f1440-5fa9-40e6-b85f-073f31d24d01-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-5007e1-d7bb8bb4c-xbqmg\" (UID: \"fa5f1440-5fa9-40e6-b85f-073f31d24d01\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5007e1-d7bb8bb4c-xbqmg"
Oct 30 00:59:09.684507 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:09.682208    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tj54b\" (UniqueName: \"kubernetes.io/projected/fa5f1440-5fa9-40e6-b85f-073f31d24d01-kube-api-access-tj54b\") pod \"payload-pre-processing-e2e-inf-b-5007e1-d7bb8bb4c-xbqmg\" (UID: \"fa5f1440-5fa9-40e6-b85f-073f31d24d01\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5007e1-d7bb8bb4c-xbqmg"
Oct 30 00:59:09.888094 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:09.888006    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5007e1-d7bb8bb4c-xbqmg"
Oct 30 00:59:10.198076 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:59:10.197891    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa5f1440_5fa9_40e6_b85f_073f31d24d01.slice/crio-9e0af7b0e8ab4b936b6c4a09db18d8d3369655b1c2fee911f736036936095360 WatchSource:0}: Error finding container 9e0af7b0e8ab4b936b6c4a09db18d8d3369655b1c2fee911f736036936095360: Status 404 returned error can't find the container with id 9e0af7b0e8ab4b936b6c4a09db18d8d3369655b1c2fee911f736036936095360
Oct 30 00:59:10.200393 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:10.200355    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-5007e1-d7bb8bb4c-xbqmg"]
Oct 30 00:59:10.755403 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:10.754980    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5007e1-d7bb8bb4c-xbqmg" event={"ID":"fa5f1440-5fa9-40e6-b85f-073f31d24d01","Type":"ContainerStarted","Data":"246310ce3175bbeb48494f66550d34f30d161f99fe59d9b707fb69a6a7e502fe"}
Oct 30 00:59:10.755403 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:10.755119    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5007e1-d7bb8bb4c-xbqmg" event={"ID":"fa5f1440-5fa9-40e6-b85f-073f31d24d01","Type":"ContainerStarted","Data":"9e0af7b0e8ab4b936b6c4a09db18d8d3369655b1c2fee911f736036936095360"}
Oct 30 00:59:10.756093 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:10.756062    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5007e1-d7bb8bb4c-xbqmg"
Oct 30 00:59:10.874268 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:10.874203    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5007e1-d7bb8bb4c-xbqmg" podStartSLOduration=1.874178653 podStartE2EDuration="1.874178653s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:59:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:59:10.868108063 +0000 UTC m=+1646.484540973" watchObservedRunningTime="2026-10-30 00:59:10.874178653 +0000 UTC m=+1646.490611533"
Oct 30 00:59:12.863506 ip-10-0-128-7 kubenswrapper[5002]: E1030 00:59:12.863466    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbjdq" containerName="extractor"
Oct 30 00:59:21.767563 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:21.767530    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5007e1-d7bb8bb4c-xbqmg"
Oct 30 00:59:31.426694 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:31.426297    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-481ab7/test-model-481ab7-kserve-59c6875d8f-wz4bj"]
Oct 30 00:59:31.431558 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:31.431530    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-481ab7/test-model-481ab7-kserve-59c6875d8f-wz4bj"
Oct 30 00:59:31.463411 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:31.462981    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-481ab7\"/\"test-model-481ab7-kserve-self-signed-certs\""
Oct 30 00:59:31.463411 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:31.463008    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-481ab7\"/\"default-dockercfg-dq2ff\""
Oct 30 00:59:31.463411 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:31.463020    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-481ab7\"/\"openshift-service-ca.crt\""
Oct 30 00:59:31.463411 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:31.463129    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-481ab7\"/\"kube-root-ca.crt\""
Oct 30 00:59:31.470575 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:31.470545    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-481ab7/test-model-481ab7-kserve-59c6875d8f-wz4bj"]
Oct 30 00:59:31.533720 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:31.533686    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75c0d4c3-f369-4930-839f-25cfad1f3f5e-dshm\") pod \"test-model-481ab7-kserve-59c6875d8f-wz4bj\" (UID: \"75c0d4c3-f369-4930-839f-25cfad1f3f5e\") " pod="ai-tenant-e2e-inf-a-481ab7/test-model-481ab7-kserve-59c6875d8f-wz4bj"
Oct 30 00:59:31.533903 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:31.533733    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxt27\" (UniqueName: \"kubernetes.io/projected/75c0d4c3-f369-4930-839f-25cfad1f3f5e-kube-api-access-cxt27\") pod \"test-model-481ab7-kserve-59c6875d8f-wz4bj\" (UID: \"75c0d4c3-f369-4930-839f-25cfad1f3f5e\") " pod="ai-tenant-e2e-inf-a-481ab7/test-model-481ab7-kserve-59c6875d8f-wz4bj"
Oct 30 00:59:31.533903 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:31.533832    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75c0d4c3-f369-4930-839f-25cfad1f3f5e-home\") pod \"test-model-481ab7-kserve-59c6875d8f-wz4bj\" (UID: \"75c0d4c3-f369-4930-839f-25cfad1f3f5e\") " pod="ai-tenant-e2e-inf-a-481ab7/test-model-481ab7-kserve-59c6875d8f-wz4bj"
Oct 30 00:59:31.534106 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:31.533948    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75c0d4c3-f369-4930-839f-25cfad1f3f5e-tmp-dir\") pod \"test-model-481ab7-kserve-59c6875d8f-wz4bj\" (UID: \"75c0d4c3-f369-4930-839f-25cfad1f3f5e\") " pod="ai-tenant-e2e-inf-a-481ab7/test-model-481ab7-kserve-59c6875d8f-wz4bj"
Oct 30 00:59:31.534106 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:31.534086    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75c0d4c3-f369-4930-839f-25cfad1f3f5e-tls-certs\") pod \"test-model-481ab7-kserve-59c6875d8f-wz4bj\" (UID: \"75c0d4c3-f369-4930-839f-25cfad1f3f5e\") " pod="ai-tenant-e2e-inf-a-481ab7/test-model-481ab7-kserve-59c6875d8f-wz4bj"
Oct 30 00:59:31.534220 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:31.534202    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75c0d4c3-f369-4930-839f-25cfad1f3f5e-model-cache\") pod \"test-model-481ab7-kserve-59c6875d8f-wz4bj\" (UID: \"75c0d4c3-f369-4930-839f-25cfad1f3f5e\") " pod="ai-tenant-e2e-inf-a-481ab7/test-model-481ab7-kserve-59c6875d8f-wz4bj"
Oct 30 00:59:31.636283 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:31.636248    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75c0d4c3-f369-4930-839f-25cfad1f3f5e-tmp-dir\") pod \"test-model-481ab7-kserve-59c6875d8f-wz4bj\" (UID: \"75c0d4c3-f369-4930-839f-25cfad1f3f5e\") " pod="ai-tenant-e2e-inf-a-481ab7/test-model-481ab7-kserve-59c6875d8f-wz4bj"
Oct 30 00:59:31.636534 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:31.636471    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75c0d4c3-f369-4930-839f-25cfad1f3f5e-tls-certs\") pod \"test-model-481ab7-kserve-59c6875d8f-wz4bj\" (UID: \"75c0d4c3-f369-4930-839f-25cfad1f3f5e\") " pod="ai-tenant-e2e-inf-a-481ab7/test-model-481ab7-kserve-59c6875d8f-wz4bj"
Oct 30 00:59:31.636611 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:31.636565    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75c0d4c3-f369-4930-839f-25cfad1f3f5e-model-cache\") pod \"test-model-481ab7-kserve-59c6875d8f-wz4bj\" (UID: \"75c0d4c3-f369-4930-839f-25cfad1f3f5e\") " pod="ai-tenant-e2e-inf-a-481ab7/test-model-481ab7-kserve-59c6875d8f-wz4bj"
Oct 30 00:59:31.636762 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:31.636738    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75c0d4c3-f369-4930-839f-25cfad1f3f5e-tmp-dir\") pod \"test-model-481ab7-kserve-59c6875d8f-wz4bj\" (UID: \"75c0d4c3-f369-4930-839f-25cfad1f3f5e\") " pod="ai-tenant-e2e-inf-a-481ab7/test-model-481ab7-kserve-59c6875d8f-wz4bj"
Oct 30 00:59:31.636843 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:31.636741    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75c0d4c3-f369-4930-839f-25cfad1f3f5e-dshm\") pod \"test-model-481ab7-kserve-59c6875d8f-wz4bj\" (UID: \"75c0d4c3-f369-4930-839f-25cfad1f3f5e\") " pod="ai-tenant-e2e-inf-a-481ab7/test-model-481ab7-kserve-59c6875d8f-wz4bj"
Oct 30 00:59:31.636843 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:31.636827    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxt27\" (UniqueName: \"kubernetes.io/projected/75c0d4c3-f369-4930-839f-25cfad1f3f5e-kube-api-access-cxt27\") pod \"test-model-481ab7-kserve-59c6875d8f-wz4bj\" (UID: \"75c0d4c3-f369-4930-839f-25cfad1f3f5e\") " pod="ai-tenant-e2e-inf-a-481ab7/test-model-481ab7-kserve-59c6875d8f-wz4bj"
Oct 30 00:59:31.636953 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:31.636867    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75c0d4c3-f369-4930-839f-25cfad1f3f5e-home\") pod \"test-model-481ab7-kserve-59c6875d8f-wz4bj\" (UID: \"75c0d4c3-f369-4930-839f-25cfad1f3f5e\") " pod="ai-tenant-e2e-inf-a-481ab7/test-model-481ab7-kserve-59c6875d8f-wz4bj"
Oct 30 00:59:31.636953 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:31.636924    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75c0d4c3-f369-4930-839f-25cfad1f3f5e-model-cache\") pod \"test-model-481ab7-kserve-59c6875d8f-wz4bj\" (UID: \"75c0d4c3-f369-4930-839f-25cfad1f3f5e\") " pod="ai-tenant-e2e-inf-a-481ab7/test-model-481ab7-kserve-59c6875d8f-wz4bj"
Oct 30 00:59:31.637283 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:31.637257    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75c0d4c3-f369-4930-839f-25cfad1f3f5e-home\") pod \"test-model-481ab7-kserve-59c6875d8f-wz4bj\" (UID: \"75c0d4c3-f369-4930-839f-25cfad1f3f5e\") " pod="ai-tenant-e2e-inf-a-481ab7/test-model-481ab7-kserve-59c6875d8f-wz4bj"
Oct 30 00:59:31.639336 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:31.639305    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75c0d4c3-f369-4930-839f-25cfad1f3f5e-dshm\") pod \"test-model-481ab7-kserve-59c6875d8f-wz4bj\" (UID: \"75c0d4c3-f369-4930-839f-25cfad1f3f5e\") " pod="ai-tenant-e2e-inf-a-481ab7/test-model-481ab7-kserve-59c6875d8f-wz4bj"
Oct 30 00:59:31.639503 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:31.639347    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75c0d4c3-f369-4930-839f-25cfad1f3f5e-tls-certs\") pod \"test-model-481ab7-kserve-59c6875d8f-wz4bj\" (UID: \"75c0d4c3-f369-4930-839f-25cfad1f3f5e\") " pod="ai-tenant-e2e-inf-a-481ab7/test-model-481ab7-kserve-59c6875d8f-wz4bj"
Oct 30 00:59:31.663332 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:31.663283    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxt27\" (UniqueName: \"kubernetes.io/projected/75c0d4c3-f369-4930-839f-25cfad1f3f5e-kube-api-access-cxt27\") pod \"test-model-481ab7-kserve-59c6875d8f-wz4bj\" (UID: \"75c0d4c3-f369-4930-839f-25cfad1f3f5e\") " pod="ai-tenant-e2e-inf-a-481ab7/test-model-481ab7-kserve-59c6875d8f-wz4bj"
Oct 30 00:59:31.754551 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:31.754516    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-481ab7/test-model-481ab7-kserve-59c6875d8f-wz4bj"
Oct 30 00:59:32.439405 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:32.439309    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-481ab7/test-model-481ab7-kserve-59c6875d8f-wz4bj"]
Oct 30 00:59:32.554465 ip-10-0-128-7 kubenswrapper[5002]: W1030 00:59:32.537550    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod75c0d4c3_f369_4930_839f_25cfad1f3f5e.slice/crio-d411ac0b01763d0cd7ce377724fa68592a340c0c6cc00348ae1a97d07eb9d55a WatchSource:0}: Error finding container d411ac0b01763d0cd7ce377724fa68592a340c0c6cc00348ae1a97d07eb9d55a: Status 404 returned error can't find the container with id d411ac0b01763d0cd7ce377724fa68592a340c0c6cc00348ae1a97d07eb9d55a
Oct 30 00:59:32.993566 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:32.993529    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-481ab7/test-model-481ab7-kserve-59c6875d8f-wz4bj" event={"ID":"75c0d4c3-f369-4930-839f-25cfad1f3f5e","Type":"ContainerStarted","Data":"da9ce5bebe1b498d835085d541d8dce734cead76e6329fcdd00e4d5170411498"}
Oct 30 00:59:32.993775 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:32.993576    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-481ab7/test-model-481ab7-kserve-59c6875d8f-wz4bj" event={"ID":"75c0d4c3-f369-4930-839f-25cfad1f3f5e","Type":"ContainerStarted","Data":"d411ac0b01763d0cd7ce377724fa68592a340c0c6cc00348ae1a97d07eb9d55a"}
Oct 30 00:59:33.041320 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:33.041264    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-481ab7/test-model-481ab7-kserve-59c6875d8f-wz4bj" podStartSLOduration=2.041249358 podStartE2EDuration="2.041249358s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:59:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:59:33.039355435 +0000 UTC m=+1668.655788315" watchObservedRunningTime="2026-10-30 00:59:33.041249358 +0000 UTC m=+1668.657682239"
Oct 30 00:59:33.758404 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:33.757500    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-481ab7/test-model-481ab7-kserve-59c6875d8f-wz4bj"
Oct 30 00:59:37.286087 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:37.285994    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-9cjs5"]
Oct 30 00:59:37.286574 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:37.286302    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-9cjs5" podUID="34603daa-226c-4698-9bb5-774255e781a6" containerName="istio-proxy" containerID="cri-o://a5698f3ca1de1d0c16a72b38175949d2427581fb55270e1bd94653fe36131191" gracePeriod=30
Oct 30 00:59:37.317954 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:37.315334    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-znxq4"]
Oct 30 00:59:37.317954 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:37.315588    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-znxq4" podUID="c61db96f-6bcd-462f-9957-c920245070c6" containerName="istio-proxy" containerID="cri-o://6452327d99acd74fd2b200c6d4939407a52f0ef83d35586c5e35bfe98114dbe8" gracePeriod=30
Oct 30 00:59:37.616342 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:37.616256    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-ztgc9"]
Oct 30 00:59:37.616532 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:37.616492    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-ztgc9" podUID="3146dece-bcb2-4625-b507-fec96b90ba59" containerName="istio-proxy" containerID="cri-o://283945a992c2b052408456a0f9c268ce1e243fbcaff3b487e36158037732072f" gracePeriod=30
Oct 30 00:59:38.162326 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:38.162286    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-7rsqp"]
Oct 30 00:59:38.164735 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:38.164699    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-7rsqp" podUID="79f13664-8af6-4e86-bf3f-8f2dcd645788" containerName="istio-proxy" containerID="cri-o://d6e427cdce0e74b1ec0aca7c222adf7df59ae9bdaa1dbaf7b41417d71810200d" gracePeriod=30
Oct 30 00:59:38.170090 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:38.170065    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-s5rzw"]
Oct 30 00:59:38.171292 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:38.171203    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-s5rzw" podUID="cc529c8b-d888-412d-b861-f3648aaa473d" containerName="istio-proxy" containerID="cri-o://f05d8bc4d1168bc3c880bca7107be748bd8bee929866d757b6caa9787dcc0e2d" gracePeriod=30
Oct 30 00:59:41.755561 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:41.755524    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-481ab7/test-model-481ab7-kserve-59c6875d8f-wz4bj"
Oct 30 00:59:41.776001 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:41.775972    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-481ab7/test-model-481ab7-kserve-59c6875d8f-wz4bj"
Oct 30 00:59:42.110061 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:42.109971    5002 generic.go:356] "Generic (PLEG): container finished" podID="b88fa1c1-bbb8-4f74-bc2a-3cff7b647082" containerID="b2b29a7f9905299ed60ad98fe7132fd97f68645f1333a030245fae1aa7c2c409" exitCode=0
Oct 30 00:59:42.110259 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:42.110079    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-84b89f-openshift-default-f7d86d855-mddrw" event={"ID":"b88fa1c1-bbb8-4f74-bc2a-3cff7b647082","Type":"ContainerDied","Data":"b2b29a7f9905299ed60ad98fe7132fd97f68645f1333a030245fae1aa7c2c409"}
Oct 30 00:59:42.110889 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:42.110863    5002 scope.go:122] "RemoveContainer" containerID="b2b29a7f9905299ed60ad98fe7132fd97f68645f1333a030245fae1aa7c2c409"
Oct 30 00:59:42.112862 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:42.112830    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 00:59:42.113000 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:42.112916    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 00:59:42.113000 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:42.112962    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 00:59:42.122534 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:42.122514    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-481ab7/test-model-481ab7-kserve-59c6875d8f-wz4bj"
Oct 30 00:59:42.643999 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:42.643970    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-znxq4"
Oct 30 00:59:42.750897 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:42.750866    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c61db96f-6bcd-462f-9957-c920245070c6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c61db96f-6bcd-462f-9957-c920245070c6-istio-data\") pod \"c61db96f-6bcd-462f-9957-c920245070c6\" (UID: \"c61db96f-6bcd-462f-9957-c920245070c6\") "
Oct 30 00:59:42.751039 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:42.750924    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c61db96f-6bcd-462f-9957-c920245070c6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c61db96f-6bcd-462f-9957-c920245070c6-istiod-ca-cert\") pod \"c61db96f-6bcd-462f-9957-c920245070c6\" (UID: \"c61db96f-6bcd-462f-9957-c920245070c6\") "
Oct 30 00:59:42.751039 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:42.750958    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c61db96f-6bcd-462f-9957-c920245070c6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c61db96f-6bcd-462f-9957-c920245070c6-istio-envoy\") pod \"c61db96f-6bcd-462f-9957-c920245070c6\" (UID: \"c61db96f-6bcd-462f-9957-c920245070c6\") "
Oct 30 00:59:42.751160 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:42.751079    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c61db96f-6bcd-462f-9957-c920245070c6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c61db96f-6bcd-462f-9957-c920245070c6-istio-podinfo\") pod \"c61db96f-6bcd-462f-9957-c920245070c6\" (UID: \"c61db96f-6bcd-462f-9957-c920245070c6\") "
Oct 30 00:59:42.751217 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:42.751176    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c61db96f-6bcd-462f-9957-c920245070c6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c61db96f-6bcd-462f-9957-c920245070c6-workload-certs\") pod \"c61db96f-6bcd-462f-9957-c920245070c6\" (UID: \"c61db96f-6bcd-462f-9957-c920245070c6\") "
Oct 30 00:59:42.751312 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:42.751286    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c61db96f-6bcd-462f-9957-c920245070c6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c61db96f-6bcd-462f-9957-c920245070c6-credential-socket\") pod \"c61db96f-6bcd-462f-9957-c920245070c6\" (UID: \"c61db96f-6bcd-462f-9957-c920245070c6\") "
Oct 30 00:59:42.751475 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:42.751449    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c61db96f-6bcd-462f-9957-c920245070c6-istio-token\" (UniqueName: \"kubernetes.io/projected/c61db96f-6bcd-462f-9957-c920245070c6-istio-token\") pod \"c61db96f-6bcd-462f-9957-c920245070c6\" (UID: \"c61db96f-6bcd-462f-9957-c920245070c6\") "
Oct 30 00:59:42.751557 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:42.751540    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c61db96f-6bcd-462f-9957-c920245070c6-kube-api-access-znzzx\" (UniqueName: \"kubernetes.io/projected/c61db96f-6bcd-462f-9957-c920245070c6-kube-api-access-znzzx\") pod \"c61db96f-6bcd-462f-9957-c920245070c6\" (UID: \"c61db96f-6bcd-462f-9957-c920245070c6\") "
Oct 30 00:59:42.751616 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:42.751574    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c61db96f-6bcd-462f-9957-c920245070c6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c61db96f-6bcd-462f-9957-c920245070c6-workload-socket\") pod \"c61db96f-6bcd-462f-9957-c920245070c6\" (UID: \"c61db96f-6bcd-462f-9957-c920245070c6\") "
Oct 30 00:59:42.752560 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:42.752534    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c61db96f-6bcd-462f-9957-c920245070c6-workload-socket" pod "c61db96f-6bcd-462f-9957-c920245070c6" (UID: "c61db96f-6bcd-462f-9957-c920245070c6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:59:42.752740 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:42.752722    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c61db96f-6bcd-462f-9957-c920245070c6-istio-data" pod "c61db96f-6bcd-462f-9957-c920245070c6" (UID: "c61db96f-6bcd-462f-9957-c920245070c6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:59:42.753033 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:42.753013    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c61db96f-6bcd-462f-9957-c920245070c6-istiod-ca-cert" pod "c61db96f-6bcd-462f-9957-c920245070c6" (UID: "c61db96f-6bcd-462f-9957-c920245070c6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:59:42.753742 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:42.753638    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c61db96f-6bcd-462f-9957-c920245070c6-credential-socket" pod "c61db96f-6bcd-462f-9957-c920245070c6" (UID: "c61db96f-6bcd-462f-9957-c920245070c6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:59:42.753972 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:42.753909    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c61db96f-6bcd-462f-9957-c920245070c6-workload-certs" pod "c61db96f-6bcd-462f-9957-c920245070c6" (UID: "c61db96f-6bcd-462f-9957-c920245070c6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:59:42.759756 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:42.759714    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c61db96f-6bcd-462f-9957-c920245070c6-istio-envoy" pod "c61db96f-6bcd-462f-9957-c920245070c6" (UID: "c61db96f-6bcd-462f-9957-c920245070c6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:59:42.773789 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:42.773502    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c61db96f-6bcd-462f-9957-c920245070c6-istio-podinfo" pod "c61db96f-6bcd-462f-9957-c920245070c6" (UID: "c61db96f-6bcd-462f-9957-c920245070c6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 00:59:42.781592 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:42.781552    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c61db96f-6bcd-462f-9957-c920245070c6-istio-token" pod "c61db96f-6bcd-462f-9957-c920245070c6" (UID: "c61db96f-6bcd-462f-9957-c920245070c6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:59:42.781721 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:42.781673    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c61db96f-6bcd-462f-9957-c920245070c6-kube-api-access-znzzx" pod "c61db96f-6bcd-462f-9957-c920245070c6" (UID: "c61db96f-6bcd-462f-9957-c920245070c6"). InnerVolumeSpecName "kube-api-access-znzzx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:59:42.854624 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:42.854471    5002 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c61db96f-6bcd-462f-9957-c920245070c6-workload-certs\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:59:42.854624 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:42.854506    5002 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c61db96f-6bcd-462f-9957-c920245070c6-credential-socket\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:59:42.854624 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:42.854521    5002 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c61db96f-6bcd-462f-9957-c920245070c6-istio-token\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:59:42.854624 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:42.854535    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-znzzx\" (UniqueName: \"kubernetes.io/projected/c61db96f-6bcd-462f-9957-c920245070c6-kube-api-access-znzzx\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:59:42.854624 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:42.854548    5002 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c61db96f-6bcd-462f-9957-c920245070c6-workload-socket\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:59:42.854624 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:42.854561    5002 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c61db96f-6bcd-462f-9957-c920245070c6-istio-data\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:59:42.854624 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:42.854573    5002 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c61db96f-6bcd-462f-9957-c920245070c6-istiod-ca-cert\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:59:42.854624 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:42.854586    5002 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c61db96f-6bcd-462f-9957-c920245070c6-istio-envoy\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:59:42.854624 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:42.854598    5002 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c61db96f-6bcd-462f-9957-c920245070c6-istio-podinfo\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:59:42.881329 ip-10-0-128-7 kubenswrapper[5002]: E1030 00:59:42.881202    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbjdq" containerName="extractor"
Oct 30 00:59:42.989686 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:42.989652    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-ztgc9"
Oct 30 00:59:43.063233 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:43.060794    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3146dece-bcb2-4625-b507-fec96b90ba59-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3146dece-bcb2-4625-b507-fec96b90ba59-istio-data\") pod \"3146dece-bcb2-4625-b507-fec96b90ba59\" (UID: \"3146dece-bcb2-4625-b507-fec96b90ba59\") "
Oct 30 00:59:43.063233 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:43.060834    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3146dece-bcb2-4625-b507-fec96b90ba59-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3146dece-bcb2-4625-b507-fec96b90ba59-credential-socket\") pod \"3146dece-bcb2-4625-b507-fec96b90ba59\" (UID: \"3146dece-bcb2-4625-b507-fec96b90ba59\") "
Oct 30 00:59:43.063233 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:43.060857    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3146dece-bcb2-4625-b507-fec96b90ba59-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3146dece-bcb2-4625-b507-fec96b90ba59-istio-podinfo\") pod \"3146dece-bcb2-4625-b507-fec96b90ba59\" (UID: \"3146dece-bcb2-4625-b507-fec96b90ba59\") "
Oct 30 00:59:43.063233 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:43.060879    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3146dece-bcb2-4625-b507-fec96b90ba59-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3146dece-bcb2-4625-b507-fec96b90ba59-istiod-ca-cert\") pod \"3146dece-bcb2-4625-b507-fec96b90ba59\" (UID: \"3146dece-bcb2-4625-b507-fec96b90ba59\") "
Oct 30 00:59:43.063233 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:43.060961    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3146dece-bcb2-4625-b507-fec96b90ba59-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3146dece-bcb2-4625-b507-fec96b90ba59-workload-socket\") pod \"3146dece-bcb2-4625-b507-fec96b90ba59\" (UID: \"3146dece-bcb2-4625-b507-fec96b90ba59\") "
Oct 30 00:59:43.063233 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:43.061002    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3146dece-bcb2-4625-b507-fec96b90ba59-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3146dece-bcb2-4625-b507-fec96b90ba59-workload-certs\") pod \"3146dece-bcb2-4625-b507-fec96b90ba59\" (UID: \"3146dece-bcb2-4625-b507-fec96b90ba59\") "
Oct 30 00:59:43.063233 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:43.061047    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3146dece-bcb2-4625-b507-fec96b90ba59-istio-token\" (UniqueName: \"kubernetes.io/projected/3146dece-bcb2-4625-b507-fec96b90ba59-istio-token\") pod \"3146dece-bcb2-4625-b507-fec96b90ba59\" (UID: \"3146dece-bcb2-4625-b507-fec96b90ba59\") "
Oct 30 00:59:43.063233 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:43.061070    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3146dece-bcb2-4625-b507-fec96b90ba59-kube-api-access-6swfb\" (UniqueName: \"kubernetes.io/projected/3146dece-bcb2-4625-b507-fec96b90ba59-kube-api-access-6swfb\") pod \"3146dece-bcb2-4625-b507-fec96b90ba59\" (UID: \"3146dece-bcb2-4625-b507-fec96b90ba59\") "
Oct 30 00:59:43.063233 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:43.061193    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3146dece-bcb2-4625-b507-fec96b90ba59-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3146dece-bcb2-4625-b507-fec96b90ba59-istio-envoy\") pod \"3146dece-bcb2-4625-b507-fec96b90ba59\" (UID: \"3146dece-bcb2-4625-b507-fec96b90ba59\") "
Oct 30 00:59:43.063233 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:43.062602    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3146dece-bcb2-4625-b507-fec96b90ba59-istiod-ca-cert" pod "3146dece-bcb2-4625-b507-fec96b90ba59" (UID: "3146dece-bcb2-4625-b507-fec96b90ba59"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:59:43.063233 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:43.062831    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3146dece-bcb2-4625-b507-fec96b90ba59-istio-data" pod "3146dece-bcb2-4625-b507-fec96b90ba59" (UID: "3146dece-bcb2-4625-b507-fec96b90ba59"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:59:43.063233 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:43.062992    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3146dece-bcb2-4625-b507-fec96b90ba59-credential-socket" pod "3146dece-bcb2-4625-b507-fec96b90ba59" (UID: "3146dece-bcb2-4625-b507-fec96b90ba59"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:59:43.064659 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:43.064621    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3146dece-bcb2-4625-b507-fec96b90ba59-workload-socket" pod "3146dece-bcb2-4625-b507-fec96b90ba59" (UID: "3146dece-bcb2-4625-b507-fec96b90ba59"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:59:43.069092 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:43.069064    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3146dece-bcb2-4625-b507-fec96b90ba59-workload-certs" pod "3146dece-bcb2-4625-b507-fec96b90ba59" (UID: "3146dece-bcb2-4625-b507-fec96b90ba59"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:59:43.073829 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:43.073513    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-9cjs5"
Oct 30 00:59:43.079844 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:43.079742    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3146dece-bcb2-4625-b507-fec96b90ba59-istio-podinfo" pod "3146dece-bcb2-4625-b507-fec96b90ba59" (UID: "3146dece-bcb2-4625-b507-fec96b90ba59"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 00:59:43.079844 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:43.079771    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3146dece-bcb2-4625-b507-fec96b90ba59-kube-api-access-6swfb" pod "3146dece-bcb2-4625-b507-fec96b90ba59" (UID: "3146dece-bcb2-4625-b507-fec96b90ba59"). InnerVolumeSpecName "kube-api-access-6swfb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:59:43.079844 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:43.079815    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3146dece-bcb2-4625-b507-fec96b90ba59-istio-envoy" pod "3146dece-bcb2-4625-b507-fec96b90ba59" (UID: "3146dece-bcb2-4625-b507-fec96b90ba59"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:59:43.081436 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:43.081415    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3146dece-bcb2-4625-b507-fec96b90ba59-istio-token" pod "3146dece-bcb2-4625-b507-fec96b90ba59" (UID: "3146dece-bcb2-4625-b507-fec96b90ba59"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:59:43.120580 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:43.120539    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-84b89f-openshift-default-f7d86d855-mddrw" event={"ID":"b88fa1c1-bbb8-4f74-bc2a-3cff7b647082","Type":"ContainerStarted","Data":"eaf8d2f564d2b486137ba14383111a62ab8163997c338185e309959ec11bba82"}
Oct 30 00:59:43.129757 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:43.129720    5002 generic.go:356] "Generic (PLEG): container finished" podID="34603daa-226c-4698-9bb5-774255e781a6" containerID="a5698f3ca1de1d0c16a72b38175949d2427581fb55270e1bd94653fe36131191" exitCode=0
Oct 30 00:59:43.130184 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:43.129782    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-9cjs5"
Oct 30 00:59:43.130184 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:43.129898    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-9cjs5" event={"ID":"34603daa-226c-4698-9bb5-774255e781a6","Type":"ContainerDied","Data":"a5698f3ca1de1d0c16a72b38175949d2427581fb55270e1bd94653fe36131191"}
Oct 30 00:59:43.130184 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:43.129941    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-9cjs5" event={"ID":"34603daa-226c-4698-9bb5-774255e781a6","Type":"ContainerDied","Data":"8f8af9b5c86b01211e93c20eb1a4ecc1794787267f59ab68393e3a4cfea38f32"}
Oct 30 00:59:43.130184 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:43.129969    5002 scope.go:122] "RemoveContainer" containerID="a5698f3ca1de1d0c16a72b38175949d2427581fb55270e1bd94653fe36131191"
Oct 30 00:59:43.132759 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:43.132732    5002 generic.go:356] "Generic (PLEG): container finished" podID="3146dece-bcb2-4625-b507-fec96b90ba59" containerID="283945a992c2b052408456a0f9c268ce1e243fbcaff3b487e36158037732072f" exitCode=0
Oct 30 00:59:43.132879 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:43.132778    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-ztgc9"
Oct 30 00:59:43.132879 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:43.132799    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-ztgc9" event={"ID":"3146dece-bcb2-4625-b507-fec96b90ba59","Type":"ContainerDied","Data":"283945a992c2b052408456a0f9c268ce1e243fbcaff3b487e36158037732072f"}
Oct 30 00:59:43.132879 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:43.132850    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-ztgc9" event={"ID":"3146dece-bcb2-4625-b507-fec96b90ba59","Type":"ContainerDied","Data":"1bf2b9029f3a5f7606afde95962d16a337e338584f9a3ae522003d4dcaf02031"}
Oct 30 00:59:43.144707 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:43.144682    5002 scope.go:122] "RemoveContainer" containerID="a5698f3ca1de1d0c16a72b38175949d2427581fb55270e1bd94653fe36131191"
Oct 30 00:59:43.145158 ip-10-0-128-7 kubenswrapper[5002]: E1030 00:59:43.145135    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5698f3ca1de1d0c16a72b38175949d2427581fb55270e1bd94653fe36131191\": container with ID starting with a5698f3ca1de1d0c16a72b38175949d2427581fb55270e1bd94653fe36131191 not found: ID does not exist" containerID="a5698f3ca1de1d0c16a72b38175949d2427581fb55270e1bd94653fe36131191"
Oct 30 00:59:43.145158 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:43.145166    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5698f3ca1de1d0c16a72b38175949d2427581fb55270e1bd94653fe36131191"} err="failed to get container status \"a5698f3ca1de1d0c16a72b38175949d2427581fb55270e1bd94653fe36131191\": rpc error: code = NotFound desc = could not find container \"a5698f3ca1de1d0c16a72b38175949d2427581fb55270e1bd94653fe36131191\": container with ID starting with a5698f3ca1de1d0c16a72b38175949d2427581fb55270e1bd94653fe36131191 not found: ID does not exist"
Oct 30 00:59:43.145352 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:43.145183    5002 scope.go:122] "RemoveContainer" containerID="283945a992c2b052408456a0f9c268ce1e243fbcaff3b487e36158037732072f"
Oct 30 00:59:43.145815 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:43.145794    5002 generic.go:356] "Generic (PLEG): container finished" podID="c61db96f-6bcd-462f-9957-c920245070c6" containerID="6452327d99acd74fd2b200c6d4939407a52f0ef83d35586c5e35bfe98114dbe8" exitCode=0
Oct 30 00:59:43.146132 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:43.146114    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-znxq4"
Oct 30 00:59:43.146551 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:43.146531    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-znxq4" event={"ID":"c61db96f-6bcd-462f-9957-c920245070c6","Type":"ContainerDied","Data":"6452327d99acd74fd2b200c6d4939407a52f0ef83d35586c5e35bfe98114dbe8"}
Oct 30 00:59:43.146840 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:43.146783    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-znxq4" event={"ID":"c61db96f-6bcd-462f-9957-c920245070c6","Type":"ContainerDied","Data":"7c54953bebbd351c55001ab37df75edfc59956db809e924cc101700c6e238027"}
Oct 30 00:59:43.162593 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:43.162530    5002 scope.go:122] "RemoveContainer" containerID="283945a992c2b052408456a0f9c268ce1e243fbcaff3b487e36158037732072f"
Oct 30 00:59:43.162935 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:43.162903    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34603daa-226c-4698-9bb5-774255e781a6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34603daa-226c-4698-9bb5-774255e781a6-istio-envoy\") pod \"34603daa-226c-4698-9bb5-774255e781a6\" (UID: \"34603daa-226c-4698-9bb5-774255e781a6\") "
Oct 30 00:59:43.163044 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:43.162958    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34603daa-226c-4698-9bb5-774255e781a6-istio-token\" (UniqueName: \"kubernetes.io/projected/34603daa-226c-4698-9bb5-774255e781a6-istio-token\") pod \"34603daa-226c-4698-9bb5-774255e781a6\" (UID: \"34603daa-226c-4698-9bb5-774255e781a6\") "
Oct 30 00:59:43.163044 ip-10-0-128-7 kubenswrapper[5002]: E1030 00:59:43.162907    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"283945a992c2b052408456a0f9c268ce1e243fbcaff3b487e36158037732072f\": container with ID starting with 283945a992c2b052408456a0f9c268ce1e243fbcaff3b487e36158037732072f not found: ID does not exist" containerID="283945a992c2b052408456a0f9c268ce1e243fbcaff3b487e36158037732072f"
Oct 30 00:59:43.163044 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:43.163023    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"283945a992c2b052408456a0f9c268ce1e243fbcaff3b487e36158037732072f"} err="failed to get container status \"283945a992c2b052408456a0f9c268ce1e243fbcaff3b487e36158037732072f\": rpc error: code = NotFound desc = could not find container \"283945a992c2b052408456a0f9c268ce1e243fbcaff3b487e36158037732072f\": container with ID starting with 283945a992c2b052408456a0f9c268ce1e243fbcaff3b487e36158037732072f not found: ID does not exist"
Oct 30 00:59:43.163211 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:43.163046    5002 scope.go:122] "RemoveContainer" containerID="6452327d99acd74fd2b200c6d4939407a52f0ef83d35586c5e35bfe98114dbe8"
Oct 30 00:59:43.163211 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:43.162995    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34603daa-226c-4698-9bb5-774255e781a6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34603daa-226c-4698-9bb5-774255e781a6-workload-certs\") pod \"34603daa-226c-4698-9bb5-774255e781a6\" (UID: \"34603daa-226c-4698-9bb5-774255e781a6\") "
Oct 30 00:59:43.163316 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:43.163242    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34603daa-226c-4698-9bb5-774255e781a6-workload-certs" pod "34603daa-226c-4698-9bb5-774255e781a6" (UID: "34603daa-226c-4698-9bb5-774255e781a6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:59:43.163610 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:43.163585    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34603daa-226c-4698-9bb5-774255e781a6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34603daa-226c-4698-9bb5-774255e781a6-istiod-ca-cert\") pod \"34603daa-226c-4698-9bb5-774255e781a6\" (UID: \"34603daa-226c-4698-9bb5-774255e781a6\") "
Oct 30 00:59:43.163731 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:43.163696    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34603daa-226c-4698-9bb5-774255e781a6-kube-api-access-ksfpr\" (UniqueName: \"kubernetes.io/projected/34603daa-226c-4698-9bb5-774255e781a6-kube-api-access-ksfpr\") pod \"34603daa-226c-4698-9bb5-774255e781a6\" (UID: \"34603daa-226c-4698-9bb5-774255e781a6\") "
Oct 30 00:59:43.163844 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:43.163826    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34603daa-226c-4698-9bb5-774255e781a6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34603daa-226c-4698-9bb5-774255e781a6-credential-socket\") pod \"34603daa-226c-4698-9bb5-774255e781a6\" (UID: \"34603daa-226c-4698-9bb5-774255e781a6\") "
Oct 30 00:59:43.164000 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:43.163983    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34603daa-226c-4698-9bb5-774255e781a6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34603daa-226c-4698-9bb5-774255e781a6-istio-data\") pod \"34603daa-226c-4698-9bb5-774255e781a6\" (UID: \"34603daa-226c-4698-9bb5-774255e781a6\") "
Oct 30 00:59:43.164088 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:43.164067    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34603daa-226c-4698-9bb5-774255e781a6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34603daa-226c-4698-9bb5-774255e781a6-workload-socket\") pod \"34603daa-226c-4698-9bb5-774255e781a6\" (UID: \"34603daa-226c-4698-9bb5-774255e781a6\") "
Oct 30 00:59:43.164235 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:43.164214    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/34603daa-226c-4698-9bb5-774255e781a6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34603daa-226c-4698-9bb5-774255e781a6-istio-podinfo\") pod \"34603daa-226c-4698-9bb5-774255e781a6\" (UID: \"34603daa-226c-4698-9bb5-774255e781a6\") "
Oct 30 00:59:43.164903 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:43.164705    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34603daa-226c-4698-9bb5-774255e781a6-workload-socket" pod "34603daa-226c-4698-9bb5-774255e781a6" (UID: "34603daa-226c-4698-9bb5-774255e781a6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:59:43.164903 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:43.164719    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34603daa-226c-4698-9bb5-774255e781a6-istio-data" pod "34603daa-226c-4698-9bb5-774255e781a6" (UID: "34603daa-226c-4698-9bb5-774255e781a6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:59:43.165584 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:43.165093    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34603daa-226c-4698-9bb5-774255e781a6-istiod-ca-cert" pod "34603daa-226c-4698-9bb5-774255e781a6" (UID: "34603daa-226c-4698-9bb5-774255e781a6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:59:43.165584 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:43.165394    5002 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34603daa-226c-4698-9bb5-774255e781a6-workload-certs\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:59:43.166356 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:43.165418    5002 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34603daa-226c-4698-9bb5-774255e781a6-istiod-ca-cert\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:59:43.166534 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:43.166370    5002 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3146dece-bcb2-4625-b507-fec96b90ba59-istio-envoy\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:59:43.166534 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:43.166401    5002 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3146dece-bcb2-4625-b507-fec96b90ba59-istio-data\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:59:43.166534 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:43.166414    5002 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3146dece-bcb2-4625-b507-fec96b90ba59-credential-socket\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:59:43.166534 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:43.166427    5002 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3146dece-bcb2-4625-b507-fec96b90ba59-istio-podinfo\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:59:43.166534 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:43.166440    5002 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3146dece-bcb2-4625-b507-fec96b90ba59-istiod-ca-cert\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:59:43.166534 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:43.166452    5002 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34603daa-226c-4698-9bb5-774255e781a6-istio-data\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:59:43.166534 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:43.166464    5002 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3146dece-bcb2-4625-b507-fec96b90ba59-workload-socket\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:59:43.166534 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:43.166476    5002 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34603daa-226c-4698-9bb5-774255e781a6-workload-socket\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:59:43.166534 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:43.166488    5002 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3146dece-bcb2-4625-b507-fec96b90ba59-workload-certs\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:59:43.166534 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:43.166501    5002 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3146dece-bcb2-4625-b507-fec96b90ba59-istio-token\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:59:43.166534 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:43.166514    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6swfb\" (UniqueName: \"kubernetes.io/projected/3146dece-bcb2-4625-b507-fec96b90ba59-kube-api-access-6swfb\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:59:43.166534 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:43.166327    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34603daa-226c-4698-9bb5-774255e781a6-credential-socket" pod "34603daa-226c-4698-9bb5-774255e781a6" (UID: "34603daa-226c-4698-9bb5-774255e781a6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:59:43.167981 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:43.167961    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/34603daa-226c-4698-9bb5-774255e781a6-istio-podinfo" pod "34603daa-226c-4698-9bb5-774255e781a6" (UID: "34603daa-226c-4698-9bb5-774255e781a6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 00:59:43.169461 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:43.169437    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34603daa-226c-4698-9bb5-774255e781a6-istio-token" pod "34603daa-226c-4698-9bb5-774255e781a6" (UID: "34603daa-226c-4698-9bb5-774255e781a6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:59:43.178990 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:43.178961    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34603daa-226c-4698-9bb5-774255e781a6-istio-envoy" pod "34603daa-226c-4698-9bb5-774255e781a6" (UID: "34603daa-226c-4698-9bb5-774255e781a6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:59:43.190728 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:43.190698    5002 scope.go:122] "RemoveContainer" containerID="6452327d99acd74fd2b200c6d4939407a52f0ef83d35586c5e35bfe98114dbe8"
Oct 30 00:59:43.191112 ip-10-0-128-7 kubenswrapper[5002]: E1030 00:59:43.191087    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6452327d99acd74fd2b200c6d4939407a52f0ef83d35586c5e35bfe98114dbe8\": container with ID starting with 6452327d99acd74fd2b200c6d4939407a52f0ef83d35586c5e35bfe98114dbe8 not found: ID does not exist" containerID="6452327d99acd74fd2b200c6d4939407a52f0ef83d35586c5e35bfe98114dbe8"
Oct 30 00:59:43.191181 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:43.191122    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6452327d99acd74fd2b200c6d4939407a52f0ef83d35586c5e35bfe98114dbe8"} err="failed to get container status \"6452327d99acd74fd2b200c6d4939407a52f0ef83d35586c5e35bfe98114dbe8\": rpc error: code = NotFound desc = could not find container \"6452327d99acd74fd2b200c6d4939407a52f0ef83d35586c5e35bfe98114dbe8\": container with ID starting with 6452327d99acd74fd2b200c6d4939407a52f0ef83d35586c5e35bfe98114dbe8 not found: ID does not exist"
Oct 30 00:59:43.197636 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:43.197605    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34603daa-226c-4698-9bb5-774255e781a6-kube-api-access-ksfpr" pod "34603daa-226c-4698-9bb5-774255e781a6" (UID: "34603daa-226c-4698-9bb5-774255e781a6"). InnerVolumeSpecName "kube-api-access-ksfpr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:59:43.222128 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:43.222096    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-znxq4"]
Oct 30 00:59:43.226007 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:43.225982    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-znxq4"]
Oct 30 00:59:43.261745 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:43.261694    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-ztgc9"]
Oct 30 00:59:43.267270 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:43.267243    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-ztgc9"]
Oct 30 00:59:43.268526 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:43.268497    5002 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34603daa-226c-4698-9bb5-774255e781a6-istio-envoy\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:59:43.268526 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:43.268525    5002 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34603daa-226c-4698-9bb5-774255e781a6-istio-token\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:59:43.268735 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:43.268539    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ksfpr\" (UniqueName: \"kubernetes.io/projected/34603daa-226c-4698-9bb5-774255e781a6-kube-api-access-ksfpr\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:59:43.268735 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:43.268553    5002 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34603daa-226c-4698-9bb5-774255e781a6-credential-socket\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:59:43.268735 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:43.268567    5002 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34603daa-226c-4698-9bb5-774255e781a6-istio-podinfo\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:59:43.501979 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:43.501952    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-9cjs5"]
Oct 30 00:59:43.506462 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:43.506419    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-9cjs5"]
Oct 30 00:59:43.537937 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:43.537913    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-7rsqp"
Oct 30 00:59:43.573295 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:43.573252    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79f13664-8af6-4e86-bf3f-8f2dcd645788-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79f13664-8af6-4e86-bf3f-8f2dcd645788-istiod-ca-cert\") pod \"79f13664-8af6-4e86-bf3f-8f2dcd645788\" (UID: \"79f13664-8af6-4e86-bf3f-8f2dcd645788\") "
Oct 30 00:59:43.573511 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:43.573370    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79f13664-8af6-4e86-bf3f-8f2dcd645788-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79f13664-8af6-4e86-bf3f-8f2dcd645788-istio-data\") pod \"79f13664-8af6-4e86-bf3f-8f2dcd645788\" (UID: \"79f13664-8af6-4e86-bf3f-8f2dcd645788\") "
Oct 30 00:59:43.573579 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:43.573522    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79f13664-8af6-4e86-bf3f-8f2dcd645788-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79f13664-8af6-4e86-bf3f-8f2dcd645788-workload-certs\") pod \"79f13664-8af6-4e86-bf3f-8f2dcd645788\" (UID: \"79f13664-8af6-4e86-bf3f-8f2dcd645788\") "
Oct 30 00:59:43.573652 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:43.573595    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79f13664-8af6-4e86-bf3f-8f2dcd645788-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79f13664-8af6-4e86-bf3f-8f2dcd645788-credential-socket\") pod \"79f13664-8af6-4e86-bf3f-8f2dcd645788\" (UID: \"79f13664-8af6-4e86-bf3f-8f2dcd645788\") "
Oct 30 00:59:43.573713 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:43.573655    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79f13664-8af6-4e86-bf3f-8f2dcd645788-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79f13664-8af6-4e86-bf3f-8f2dcd645788-istio-envoy\") pod \"79f13664-8af6-4e86-bf3f-8f2dcd645788\" (UID: \"79f13664-8af6-4e86-bf3f-8f2dcd645788\") "
Oct 30 00:59:43.573713 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:43.573697    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79f13664-8af6-4e86-bf3f-8f2dcd645788-istio-token\" (UniqueName: \"kubernetes.io/projected/79f13664-8af6-4e86-bf3f-8f2dcd645788-istio-token\") pod \"79f13664-8af6-4e86-bf3f-8f2dcd645788\" (UID: \"79f13664-8af6-4e86-bf3f-8f2dcd645788\") "
Oct 30 00:59:43.573713 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:43.573694    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79f13664-8af6-4e86-bf3f-8f2dcd645788-istiod-ca-cert" pod "79f13664-8af6-4e86-bf3f-8f2dcd645788" (UID: "79f13664-8af6-4e86-bf3f-8f2dcd645788"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:59:43.573879 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:43.573831    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/79f13664-8af6-4e86-bf3f-8f2dcd645788-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79f13664-8af6-4e86-bf3f-8f2dcd645788-istio-podinfo\") pod \"79f13664-8af6-4e86-bf3f-8f2dcd645788\" (UID: \"79f13664-8af6-4e86-bf3f-8f2dcd645788\") "
Oct 30 00:59:43.573936 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:43.573905    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79f13664-8af6-4e86-bf3f-8f2dcd645788-kube-api-access-cdqnc\" (UniqueName: \"kubernetes.io/projected/79f13664-8af6-4e86-bf3f-8f2dcd645788-kube-api-access-cdqnc\") pod \"79f13664-8af6-4e86-bf3f-8f2dcd645788\" (UID: \"79f13664-8af6-4e86-bf3f-8f2dcd645788\") "
Oct 30 00:59:43.573993 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:43.573965    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79f13664-8af6-4e86-bf3f-8f2dcd645788-workload-certs" pod "79f13664-8af6-4e86-bf3f-8f2dcd645788" (UID: "79f13664-8af6-4e86-bf3f-8f2dcd645788"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:59:43.573993 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:43.573974    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79f13664-8af6-4e86-bf3f-8f2dcd645788-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79f13664-8af6-4e86-bf3f-8f2dcd645788-workload-socket\") pod \"79f13664-8af6-4e86-bf3f-8f2dcd645788\" (UID: \"79f13664-8af6-4e86-bf3f-8f2dcd645788\") "
Oct 30 00:59:43.574489 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:43.574131    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79f13664-8af6-4e86-bf3f-8f2dcd645788-istio-data" pod "79f13664-8af6-4e86-bf3f-8f2dcd645788" (UID: "79f13664-8af6-4e86-bf3f-8f2dcd645788"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:59:43.574622 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:43.574506    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79f13664-8af6-4e86-bf3f-8f2dcd645788-credential-socket" pod "79f13664-8af6-4e86-bf3f-8f2dcd645788" (UID: "79f13664-8af6-4e86-bf3f-8f2dcd645788"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:59:43.574828 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:43.574807    5002 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79f13664-8af6-4e86-bf3f-8f2dcd645788-istiod-ca-cert\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:59:43.574905 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:43.574834    5002 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79f13664-8af6-4e86-bf3f-8f2dcd645788-istio-data\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:59:43.574905 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:43.574848    5002 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79f13664-8af6-4e86-bf3f-8f2dcd645788-workload-certs\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:59:43.574905 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:43.574862    5002 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79f13664-8af6-4e86-bf3f-8f2dcd645788-credential-socket\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:59:43.575138 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:43.575111    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79f13664-8af6-4e86-bf3f-8f2dcd645788-workload-socket" pod "79f13664-8af6-4e86-bf3f-8f2dcd645788" (UID: "79f13664-8af6-4e86-bf3f-8f2dcd645788"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:59:43.577757 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:43.577728    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/79f13664-8af6-4e86-bf3f-8f2dcd645788-istio-podinfo" pod "79f13664-8af6-4e86-bf3f-8f2dcd645788" (UID: "79f13664-8af6-4e86-bf3f-8f2dcd645788"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 00:59:43.577895 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:43.577755    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79f13664-8af6-4e86-bf3f-8f2dcd645788-kube-api-access-cdqnc" pod "79f13664-8af6-4e86-bf3f-8f2dcd645788" (UID: "79f13664-8af6-4e86-bf3f-8f2dcd645788"). InnerVolumeSpecName "kube-api-access-cdqnc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:59:43.578229 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:43.578205    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79f13664-8af6-4e86-bf3f-8f2dcd645788-istio-token" pod "79f13664-8af6-4e86-bf3f-8f2dcd645788" (UID: "79f13664-8af6-4e86-bf3f-8f2dcd645788"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:59:43.578508 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:43.578482    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79f13664-8af6-4e86-bf3f-8f2dcd645788-istio-envoy" pod "79f13664-8af6-4e86-bf3f-8f2dcd645788" (UID: "79f13664-8af6-4e86-bf3f-8f2dcd645788"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:59:43.600910 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:43.600882    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-s5rzw"
Oct 30 00:59:43.676154 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:43.676118    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc529c8b-d888-412d-b861-f3648aaa473d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc529c8b-d888-412d-b861-f3648aaa473d-workload-socket\") pod \"cc529c8b-d888-412d-b861-f3648aaa473d\" (UID: \"cc529c8b-d888-412d-b861-f3648aaa473d\") "
Oct 30 00:59:43.676362 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:43.676242    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc529c8b-d888-412d-b861-f3648aaa473d-kube-api-access-9dbnt\" (UniqueName: \"kubernetes.io/projected/cc529c8b-d888-412d-b861-f3648aaa473d-kube-api-access-9dbnt\") pod \"cc529c8b-d888-412d-b861-f3648aaa473d\" (UID: \"cc529c8b-d888-412d-b861-f3648aaa473d\") "
Oct 30 00:59:43.676362 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:43.676280    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc529c8b-d888-412d-b861-f3648aaa473d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc529c8b-d888-412d-b861-f3648aaa473d-credential-socket\") pod \"cc529c8b-d888-412d-b861-f3648aaa473d\" (UID: \"cc529c8b-d888-412d-b861-f3648aaa473d\") "
Oct 30 00:59:43.676503 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:43.676478    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc529c8b-d888-412d-b861-f3648aaa473d-workload-socket" pod "cc529c8b-d888-412d-b861-f3648aaa473d" (UID: "cc529c8b-d888-412d-b861-f3648aaa473d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:59:43.676577 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:43.676555    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc529c8b-d888-412d-b861-f3648aaa473d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc529c8b-d888-412d-b861-f3648aaa473d-istio-data\") pod \"cc529c8b-d888-412d-b861-f3648aaa473d\" (UID: \"cc529c8b-d888-412d-b861-f3648aaa473d\") "
Oct 30 00:59:43.676633 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:43.676616    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc529c8b-d888-412d-b861-f3648aaa473d-istio-token\" (UniqueName: \"kubernetes.io/projected/cc529c8b-d888-412d-b861-f3648aaa473d-istio-token\") pod \"cc529c8b-d888-412d-b861-f3648aaa473d\" (UID: \"cc529c8b-d888-412d-b861-f3648aaa473d\") "
Oct 30 00:59:43.676692 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:43.676673    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc529c8b-d888-412d-b861-f3648aaa473d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc529c8b-d888-412d-b861-f3648aaa473d-istio-envoy\") pod \"cc529c8b-d888-412d-b861-f3648aaa473d\" (UID: \"cc529c8b-d888-412d-b861-f3648aaa473d\") "
Oct 30 00:59:43.676793 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:43.676771    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc529c8b-d888-412d-b861-f3648aaa473d-credential-socket" pod "cc529c8b-d888-412d-b861-f3648aaa473d" (UID: "cc529c8b-d888-412d-b861-f3648aaa473d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:59:43.676875 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:43.676780    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cc529c8b-d888-412d-b861-f3648aaa473d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc529c8b-d888-412d-b861-f3648aaa473d-istio-podinfo\") pod \"cc529c8b-d888-412d-b861-f3648aaa473d\" (UID: \"cc529c8b-d888-412d-b861-f3648aaa473d\") "
Oct 30 00:59:43.676875 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:43.676841    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc529c8b-d888-412d-b861-f3648aaa473d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc529c8b-d888-412d-b861-f3648aaa473d-istiod-ca-cert\") pod \"cc529c8b-d888-412d-b861-f3648aaa473d\" (UID: \"cc529c8b-d888-412d-b861-f3648aaa473d\") "
Oct 30 00:59:43.676981 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:43.676883    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc529c8b-d888-412d-b861-f3648aaa473d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc529c8b-d888-412d-b861-f3648aaa473d-workload-certs\") pod \"cc529c8b-d888-412d-b861-f3648aaa473d\" (UID: \"cc529c8b-d888-412d-b861-f3648aaa473d\") "
Oct 30 00:59:43.677628 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:43.677605    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc529c8b-d888-412d-b861-f3648aaa473d-istio-data" pod "cc529c8b-d888-412d-b861-f3648aaa473d" (UID: "cc529c8b-d888-412d-b861-f3648aaa473d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:59:43.677762 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:43.677691    5002 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc529c8b-d888-412d-b861-f3648aaa473d-workload-socket\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:59:43.677883 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:43.677841    5002 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc529c8b-d888-412d-b861-f3648aaa473d-credential-socket\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:59:43.678003 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:43.677991    5002 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79f13664-8af6-4e86-bf3f-8f2dcd645788-istio-envoy\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:59:43.678113 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:43.678101    5002 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79f13664-8af6-4e86-bf3f-8f2dcd645788-istio-token\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:59:43.678232 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:43.678221    5002 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79f13664-8af6-4e86-bf3f-8f2dcd645788-istio-podinfo\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:59:43.678327 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:43.678316    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cdqnc\" (UniqueName: \"kubernetes.io/projected/79f13664-8af6-4e86-bf3f-8f2dcd645788-kube-api-access-cdqnc\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:59:43.678453 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:43.678441    5002 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79f13664-8af6-4e86-bf3f-8f2dcd645788-workload-socket\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:59:43.678535 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:43.677925    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc529c8b-d888-412d-b861-f3648aaa473d-workload-certs" pod "cc529c8b-d888-412d-b861-f3648aaa473d" (UID: "cc529c8b-d888-412d-b861-f3648aaa473d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:59:43.678624 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:43.678031    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc529c8b-d888-412d-b861-f3648aaa473d-istiod-ca-cert" pod "cc529c8b-d888-412d-b861-f3648aaa473d" (UID: "cc529c8b-d888-412d-b861-f3648aaa473d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:59:43.681659 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:43.681571    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc529c8b-d888-412d-b861-f3648aaa473d-istio-envoy" pod "cc529c8b-d888-412d-b861-f3648aaa473d" (UID: "cc529c8b-d888-412d-b861-f3648aaa473d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:59:43.682136 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:43.682111    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc529c8b-d888-412d-b861-f3648aaa473d-kube-api-access-9dbnt" pod "cc529c8b-d888-412d-b861-f3648aaa473d" (UID: "cc529c8b-d888-412d-b861-f3648aaa473d"). InnerVolumeSpecName "kube-api-access-9dbnt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:59:43.682677 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:43.682644    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc529c8b-d888-412d-b861-f3648aaa473d-istio-token" pod "cc529c8b-d888-412d-b861-f3648aaa473d" (UID: "cc529c8b-d888-412d-b861-f3648aaa473d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:59:43.682913 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:43.682887    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cc529c8b-d888-412d-b861-f3648aaa473d-istio-podinfo" pod "cc529c8b-d888-412d-b861-f3648aaa473d" (UID: "cc529c8b-d888-412d-b861-f3648aaa473d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 00:59:43.779714 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:43.779669    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9dbnt\" (UniqueName: \"kubernetes.io/projected/cc529c8b-d888-412d-b861-f3648aaa473d-kube-api-access-9dbnt\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:59:43.779714 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:43.779702    5002 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc529c8b-d888-412d-b861-f3648aaa473d-istio-data\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:59:43.779714 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:43.779713    5002 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc529c8b-d888-412d-b861-f3648aaa473d-istio-token\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:59:43.779714 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:43.779721    5002 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc529c8b-d888-412d-b861-f3648aaa473d-istio-envoy\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:59:43.780363 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:43.779731    5002 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc529c8b-d888-412d-b861-f3648aaa473d-istio-podinfo\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:59:43.780363 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:43.779739    5002 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc529c8b-d888-412d-b861-f3648aaa473d-istiod-ca-cert\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:59:43.780363 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:43.779746    5002 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc529c8b-d888-412d-b861-f3648aaa473d-workload-certs\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 00:59:43.963525 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:43.963492    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-84b89f-openshift-default-f7d86d855-mddrw"
Oct 30 00:59:43.965143 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:43.964799    5002 patch_prober.go:29] interesting pod/e2e-inf-b-84b89f-openshift-default-f7d86d855-mddrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body=
Oct 30 00:59:43.965143 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:43.964865    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-84b89f-openshift-default-f7d86d855-mddrw" podUID="b88fa1c1-bbb8-4f74-bc2a-3cff7b647082" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused"
Oct 30 00:59:44.188145 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:44.188099    5002 generic.go:356] "Generic (PLEG): container finished" podID="cc529c8b-d888-412d-b861-f3648aaa473d" containerID="f05d8bc4d1168bc3c880bca7107be748bd8bee929866d757b6caa9787dcc0e2d" exitCode=0
Oct 30 00:59:44.191973 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:44.188168    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-s5rzw"
Oct 30 00:59:44.191973 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:44.188203    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-s5rzw" event={"ID":"cc529c8b-d888-412d-b861-f3648aaa473d","Type":"ContainerDied","Data":"f05d8bc4d1168bc3c880bca7107be748bd8bee929866d757b6caa9787dcc0e2d"}
Oct 30 00:59:44.191973 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:44.188809    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-s5rzw" event={"ID":"cc529c8b-d888-412d-b861-f3648aaa473d","Type":"ContainerDied","Data":"ecd57d08fad9bca1f88fbc9847427a61df4f1aa1c5676a0a2f3ffd963d7ce170"}
Oct 30 00:59:44.191973 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:44.188847    5002 scope.go:122] "RemoveContainer" containerID="f05d8bc4d1168bc3c880bca7107be748bd8bee929866d757b6caa9787dcc0e2d"
Oct 30 00:59:44.191973 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:44.190783    5002 generic.go:356] "Generic (PLEG): container finished" podID="79f13664-8af6-4e86-bf3f-8f2dcd645788" containerID="d6e427cdce0e74b1ec0aca7c222adf7df59ae9bdaa1dbaf7b41417d71810200d" exitCode=0
Oct 30 00:59:44.191973 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:44.190903    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-7rsqp" event={"ID":"79f13664-8af6-4e86-bf3f-8f2dcd645788","Type":"ContainerDied","Data":"d6e427cdce0e74b1ec0aca7c222adf7df59ae9bdaa1dbaf7b41417d71810200d"}
Oct 30 00:59:44.191973 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:44.190939    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-7rsqp" event={"ID":"79f13664-8af6-4e86-bf3f-8f2dcd645788","Type":"ContainerDied","Data":"5db85cf4f22afa67643679ba8abdce7e4bb78f9bfa2f9bda88821c74812b6dd5"}
Oct 30 00:59:44.191973 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:44.191004    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-7rsqp"
Oct 30 00:59:44.204523 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:44.204501    5002 scope.go:122] "RemoveContainer" containerID="f05d8bc4d1168bc3c880bca7107be748bd8bee929866d757b6caa9787dcc0e2d"
Oct 30 00:59:44.204801 ip-10-0-128-7 kubenswrapper[5002]: E1030 00:59:44.204780    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f05d8bc4d1168bc3c880bca7107be748bd8bee929866d757b6caa9787dcc0e2d\": container with ID starting with f05d8bc4d1168bc3c880bca7107be748bd8bee929866d757b6caa9787dcc0e2d not found: ID does not exist" containerID="f05d8bc4d1168bc3c880bca7107be748bd8bee929866d757b6caa9787dcc0e2d"
Oct 30 00:59:44.204889 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:44.204807    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f05d8bc4d1168bc3c880bca7107be748bd8bee929866d757b6caa9787dcc0e2d"} err="failed to get container status \"f05d8bc4d1168bc3c880bca7107be748bd8bee929866d757b6caa9787dcc0e2d\": rpc error: code = NotFound desc = could not find container \"f05d8bc4d1168bc3c880bca7107be748bd8bee929866d757b6caa9787dcc0e2d\": container with ID starting with f05d8bc4d1168bc3c880bca7107be748bd8bee929866d757b6caa9787dcc0e2d not found: ID does not exist"
Oct 30 00:59:44.204889 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:44.204824    5002 scope.go:122] "RemoveContainer" containerID="d6e427cdce0e74b1ec0aca7c222adf7df59ae9bdaa1dbaf7b41417d71810200d"
Oct 30 00:59:44.214632 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:44.214614    5002 scope.go:122] "RemoveContainer" containerID="d6e427cdce0e74b1ec0aca7c222adf7df59ae9bdaa1dbaf7b41417d71810200d"
Oct 30 00:59:44.214892 ip-10-0-128-7 kubenswrapper[5002]: E1030 00:59:44.214872    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6e427cdce0e74b1ec0aca7c222adf7df59ae9bdaa1dbaf7b41417d71810200d\": container with ID starting with d6e427cdce0e74b1ec0aca7c222adf7df59ae9bdaa1dbaf7b41417d71810200d not found: ID does not exist" containerID="d6e427cdce0e74b1ec0aca7c222adf7df59ae9bdaa1dbaf7b41417d71810200d"
Oct 30 00:59:44.214946 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:44.214899    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6e427cdce0e74b1ec0aca7c222adf7df59ae9bdaa1dbaf7b41417d71810200d"} err="failed to get container status \"d6e427cdce0e74b1ec0aca7c222adf7df59ae9bdaa1dbaf7b41417d71810200d\": rpc error: code = NotFound desc = could not find container \"d6e427cdce0e74b1ec0aca7c222adf7df59ae9bdaa1dbaf7b41417d71810200d\": container with ID starting with d6e427cdce0e74b1ec0aca7c222adf7df59ae9bdaa1dbaf7b41417d71810200d not found: ID does not exist"
Oct 30 00:59:44.247077 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:44.247045    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-7rsqp"]
Oct 30 00:59:44.283439 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:44.269513    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-7rsqp"]
Oct 30 00:59:44.292028 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:44.291995    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-s5rzw"]
Oct 30 00:59:44.297298 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:44.297265    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-s5rzw"]
Oct 30 00:59:44.498070 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:44.497980    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3146dece-bcb2-4625-b507-fec96b90ba59" path="/var/lib/kubelet/pods/3146dece-bcb2-4625-b507-fec96b90ba59/volumes"
Oct 30 00:59:44.498542 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:44.498525    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34603daa-226c-4698-9bb5-774255e781a6" path="/var/lib/kubelet/pods/34603daa-226c-4698-9bb5-774255e781a6/volumes"
Oct 30 00:59:44.498990 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:44.498976    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79f13664-8af6-4e86-bf3f-8f2dcd645788" path="/var/lib/kubelet/pods/79f13664-8af6-4e86-bf3f-8f2dcd645788/volumes"
Oct 30 00:59:44.499759 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:44.499740    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c61db96f-6bcd-462f-9957-c920245070c6" path="/var/lib/kubelet/pods/c61db96f-6bcd-462f-9957-c920245070c6/volumes"
Oct 30 00:59:44.500178 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:44.500164    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc529c8b-d888-412d-b861-f3648aaa473d" path="/var/lib/kubelet/pods/cc529c8b-d888-412d-b861-f3648aaa473d/volumes"
Oct 30 00:59:44.965816 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:44.965779    5002 patch_prober.go:29] interesting pod/e2e-inf-b-84b89f-openshift-default-f7d86d855-mddrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body=
Oct 30 00:59:44.966441 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:44.966407    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-84b89f-openshift-default-f7d86d855-mddrw" podUID="b88fa1c1-bbb8-4f74-bc2a-3cff7b647082" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused"
Oct 30 00:59:45.965404 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:45.964809    5002 patch_prober.go:29] interesting pod/e2e-inf-b-84b89f-openshift-default-f7d86d855-mddrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body=
Oct 30 00:59:45.965404 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:45.964877    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-84b89f-openshift-default-f7d86d855-mddrw" podUID="b88fa1c1-bbb8-4f74-bc2a-3cff7b647082" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused"
Oct 30 00:59:46.588545 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:46.588511    5002 scope.go:122] "RemoveContainer" containerID="381641d2380b8b0643d458eaafbfabc023a82fd006f21e5abc952fdb4296ba77"
Oct 30 00:59:46.971403 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:46.964276    5002 patch_prober.go:29] interesting pod/e2e-inf-b-84b89f-openshift-default-f7d86d855-mddrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body=
Oct 30 00:59:46.971403 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:46.964335    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-84b89f-openshift-default-f7d86d855-mddrw" podUID="b88fa1c1-bbb8-4f74-bc2a-3cff7b647082" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused"
Oct 30 00:59:47.967407 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:47.966832    5002 patch_prober.go:29] interesting pod/e2e-inf-b-84b89f-openshift-default-f7d86d855-mddrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body=
Oct 30 00:59:47.967407 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:47.966900    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-84b89f-openshift-default-f7d86d855-mddrw" podUID="b88fa1c1-bbb8-4f74-bc2a-3cff7b647082" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused"
Oct 30 00:59:48.183406 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:48.181872    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-84b89f-openshift-default-f7d86d855-mddrw"
Oct 30 00:59:48.963816 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:48.963775    5002 patch_prober.go:29] interesting pod/e2e-inf-b-84b89f-openshift-default-f7d86d855-mddrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body=
Oct 30 00:59:48.964074 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:48.964045    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-84b89f-openshift-default-f7d86d855-mddrw" podUID="b88fa1c1-bbb8-4f74-bc2a-3cff7b647082" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused"
Oct 30 00:59:49.999308 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:49.997699    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-84b89f-openshift-default-f7d86d855-mddrw"
Oct 30 00:59:50.019555 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:50.016619    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-84b89f-openshift-default-f7d86d855-mddrw"
Oct 30 00:59:54.789574 ip-10-0-128-7 kubenswrapper[5002]: I1030 00:59:54.789518    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7d8z4"]
Oct 30 01:00:00.203403 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:00.202697    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8e0f66-29888700-tnj7q"]
Oct 30 01:00:00.203898 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:00.203690    5002 cpu_manager.go:414] "containerMap: removing container" podUID="79f13664-8af6-4e86-bf3f-8f2dcd645788" containerName="istio-proxy"
Oct 30 01:00:00.203898 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:00.203716    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79f13664-8af6-4e86-bf3f-8f2dcd645788" containerName="istio-proxy"
Oct 30 01:00:00.203898 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:00.203743    5002 cpu_manager.go:414] "containerMap: removing container" podUID="c61db96f-6bcd-462f-9957-c920245070c6" containerName="istio-proxy"
Oct 30 01:00:00.203898 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:00.203751    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c61db96f-6bcd-462f-9957-c920245070c6" containerName="istio-proxy"
Oct 30 01:00:00.203898 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:00.203764    5002 cpu_manager.go:414] "containerMap: removing container" podUID="cc529c8b-d888-412d-b861-f3648aaa473d" containerName="istio-proxy"
Oct 30 01:00:00.203898 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:00.203771    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc529c8b-d888-412d-b861-f3648aaa473d" containerName="istio-proxy"
Oct 30 01:00:00.203898 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:00.203781    5002 cpu_manager.go:414] "containerMap: removing container" podUID="34603daa-226c-4698-9bb5-774255e781a6" containerName="istio-proxy"
Oct 30 01:00:00.203898 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:00.203789    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34603daa-226c-4698-9bb5-774255e781a6" containerName="istio-proxy"
Oct 30 01:00:00.203898 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:00.203877    5002 cpu_manager.go:414] "containerMap: removing container" podUID="3146dece-bcb2-4625-b507-fec96b90ba59" containerName="istio-proxy"
Oct 30 01:00:00.203898 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:00.203884    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3146dece-bcb2-4625-b507-fec96b90ba59" containerName="istio-proxy"
Oct 30 01:00:00.204367 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:00.204049    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="3146dece-bcb2-4625-b507-fec96b90ba59" containerName="istio-proxy"
Oct 30 01:00:00.204367 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:00.204063    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="79f13664-8af6-4e86-bf3f-8f2dcd645788" containerName="istio-proxy"
Oct 30 01:00:00.204367 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:00.204078    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="c61db96f-6bcd-462f-9957-c920245070c6" containerName="istio-proxy"
Oct 30 01:00:00.204367 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:00.204089    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc529c8b-d888-412d-b861-f3648aaa473d" containerName="istio-proxy"
Oct 30 01:00:00.204367 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:00.204099    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="34603daa-226c-4698-9bb5-774255e781a6" containerName="istio-proxy"
Oct 30 01:00:00.240825 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:00.240794    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-52650f-29888700-8mvn6"]
Oct 30 01:00:00.241118 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:00.240969    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8e0f66-29888700-tnj7q"
Oct 30 01:00:00.268592 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:00.267090    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:00:00.268592 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:00.267809    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:00:00.291408 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:00.283581    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888700-k6pvc"]
Oct 30 01:00:00.319404 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:00.302494    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fa9212f-5c4d-47a2-a855-2c18a807533e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-8e0f66-29888700-tnj7q\" (UID: \"5fa9212f-5c4d-47a2-a855-2c18a807533e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8e0f66-29888700-tnj7q"
Oct 30 01:00:00.319404 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:00.302736    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jv2gh\" (UniqueName: \"kubernetes.io/projected/5fa9212f-5c4d-47a2-a855-2c18a807533e-kube-api-access-jv2gh\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-8e0f66-29888700-tnj7q\" (UID: \"5fa9212f-5c4d-47a2-a855-2c18a807533e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8e0f66-29888700-tnj7q"
Oct 30 01:00:00.319404 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:00.304159    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5phsh\""
Oct 30 01:00:00.319404 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:00.284834    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-52650f-29888700-8mvn6"
Oct 30 01:00:00.411483 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:00.411435    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/09fb128c-0670-4260-a77b-ae32de23fa0d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-52650f-29888700-8mvn6\" (UID: \"09fb128c-0670-4260-a77b-ae32de23fa0d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-52650f-29888700-8mvn6"
Oct 30 01:00:00.411723 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:00.411704    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brqqq\" (UniqueName: \"kubernetes.io/projected/09fb128c-0670-4260-a77b-ae32de23fa0d-kube-api-access-brqqq\") pod \"maas-api-key-cleanup-e2e-worker-w0-52650f-29888700-8mvn6\" (UID: \"09fb128c-0670-4260-a77b-ae32de23fa0d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-52650f-29888700-8mvn6"
Oct 30 01:00:00.411902 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:00.411885    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jv2gh\" (UniqueName: \"kubernetes.io/projected/5fa9212f-5c4d-47a2-a855-2c18a807533e-kube-api-access-jv2gh\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-8e0f66-29888700-tnj7q\" (UID: \"5fa9212f-5c4d-47a2-a855-2c18a807533e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8e0f66-29888700-tnj7q"
Oct 30 01:00:00.412129 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:00.412115    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fa9212f-5c4d-47a2-a855-2c18a807533e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-8e0f66-29888700-tnj7q\" (UID: \"5fa9212f-5c4d-47a2-a855-2c18a807533e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8e0f66-29888700-tnj7q"
Oct 30 01:00:00.413099 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:00.413072    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fa9212f-5c4d-47a2-a855-2c18a807533e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-8e0f66-29888700-tnj7q\" (UID: \"5fa9212f-5c4d-47a2-a855-2c18a807533e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8e0f66-29888700-tnj7q"
Oct 30 01:00:00.436406 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:00.427439    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-11f521-29888700-khq6f"]
Oct 30 01:00:00.436406 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:00.430701    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888700-k6pvc"
Oct 30 01:00:00.439575 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:00.439543    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5f387c-29888700-vtvtq"]
Oct 30 01:00:00.439889 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:00.439868    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-11f521-29888700-khq6f"
Oct 30 01:00:00.444149 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:00.444087    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jv2gh\" (UniqueName: \"kubernetes.io/projected/5fa9212f-5c4d-47a2-a855-2c18a807533e-kube-api-access-jv2gh\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-8e0f66-29888700-tnj7q\" (UID: \"5fa9212f-5c4d-47a2-a855-2c18a807533e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8e0f66-29888700-tnj7q"
Oct 30 01:00:00.518022 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:00.517986    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4sfq\" (UniqueName: \"kubernetes.io/projected/d5279cb1-29b0-494a-b3b3-527e8af641dc-kube-api-access-h4sfq\") pod \"maas-api-key-cleanup-29888700-k6pvc\" (UID: \"d5279cb1-29b0-494a-b3b3-527e8af641dc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888700-k6pvc"
Oct 30 01:00:00.518307 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:00.518286    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ns8b9\" (UniqueName: \"kubernetes.io/projected/7f443227-6ffc-40f4-80aa-1485ae41aa02-kube-api-access-ns8b9\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-11f521-29888700-khq6f\" (UID: \"7f443227-6ffc-40f4-80aa-1485ae41aa02\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-11f521-29888700-khq6f"
Oct 30 01:00:00.519081 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:00.518702    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5279cb1-29b0-494a-b3b3-527e8af641dc-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888700-k6pvc\" (UID: \"d5279cb1-29b0-494a-b3b3-527e8af641dc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888700-k6pvc"
Oct 30 01:00:00.519438 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:00.519413    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/09fb128c-0670-4260-a77b-ae32de23fa0d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-52650f-29888700-8mvn6\" (UID: \"09fb128c-0670-4260-a77b-ae32de23fa0d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-52650f-29888700-8mvn6"
Oct 30 01:00:00.519559 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:00.519453    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brqqq\" (UniqueName: \"kubernetes.io/projected/09fb128c-0670-4260-a77b-ae32de23fa0d-kube-api-access-brqqq\") pod \"maas-api-key-cleanup-e2e-worker-w0-52650f-29888700-8mvn6\" (UID: \"09fb128c-0670-4260-a77b-ae32de23fa0d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-52650f-29888700-8mvn6"
Oct 30 01:00:00.519559 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:00.519507    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f443227-6ffc-40f4-80aa-1485ae41aa02-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-11f521-29888700-khq6f\" (UID: \"7f443227-6ffc-40f4-80aa-1485ae41aa02\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-11f521-29888700-khq6f"
Oct 30 01:00:00.520739 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:00.520715    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/09fb128c-0670-4260-a77b-ae32de23fa0d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-52650f-29888700-8mvn6\" (UID: \"09fb128c-0670-4260-a77b-ae32de23fa0d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-52650f-29888700-8mvn6"
Oct 30 01:00:00.522370 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:00.522349    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1a11cf-29888700-d9s5c"]
Oct 30 01:00:00.523708 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:00.523685    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5f387c-29888700-vtvtq"
Oct 30 01:00:00.527669 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:00.527653    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-cce513-29888700-744n2"]
Oct 30 01:00:00.527949 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:00.527874    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1a11cf-29888700-d9s5c"
Oct 30 01:00:00.531173 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:00.531147    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-cce513-29888700-744n2"
Oct 30 01:00:00.574406 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:00.550585    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8e0f66-29888700-tnj7q"]
Oct 30 01:00:00.574406 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:00.550657    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-52650f-29888700-8mvn6"]
Oct 30 01:00:00.574406 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:00.550695    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888700-k6pvc"]
Oct 30 01:00:00.574406 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:00.550730    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-5007e1-29888700-dvm8j"]
Oct 30 01:00:00.574406 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:00.566873    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5f387c-29888700-vtvtq"]
Oct 30 01:00:00.574406 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:00.566927    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ca5363-29888700-trbnd"]
Oct 30 01:00:00.598492 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:00.577863    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8d2a70-29888700-jlmvv"]
Oct 30 01:00:00.598492 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:00.582953    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-5007e1-29888700-dvm8j"
Oct 30 01:00:00.598492 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:00.583991    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1a11cf-29888700-d9s5c"]
Oct 30 01:00:00.598492 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:00.584035    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-cce513-29888700-744n2"]
Oct 30 01:00:00.598492 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:00.584071    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-481ab7-29888700-8mt68"]
Oct 30 01:00:00.598492 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:00.592102    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ca5363-29888700-trbnd"
Oct 30 01:00:00.598492 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:00.592635    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8d2a70-29888700-jlmvv"
Oct 30 01:00:00.611115 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:00.611080    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-brqqq\" (UniqueName: \"kubernetes.io/projected/09fb128c-0670-4260-a77b-ae32de23fa0d-kube-api-access-brqqq\") pod \"maas-api-key-cleanup-e2e-worker-w0-52650f-29888700-8mvn6\" (UID: \"09fb128c-0670-4260-a77b-ae32de23fa0d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-52650f-29888700-8mvn6"
Oct 30 01:00:00.618361 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:00.618330    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-5007e1-29888700-dvm8j"]
Oct 30 01:00:00.618534 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:00.618409    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8d2a70-29888700-jlmvv"]
Oct 30 01:00:00.618534 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:00.618529    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-481ab7-29888700-8mt68"
Oct 30 01:00:00.622663 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:00.620941    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38a379b2-37b6-4185-8319-2e62dcec38c8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-5007e1-29888700-dvm8j\" (UID: \"38a379b2-37b6-4185-8319-2e62dcec38c8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-5007e1-29888700-dvm8j"
Oct 30 01:00:00.622663 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:00.621001    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79jrw\" (UniqueName: \"kubernetes.io/projected/4af343ba-bc4d-4ec3-92df-1534780d3089-kube-api-access-79jrw\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-5f387c-29888700-vtvtq\" (UID: \"4af343ba-bc4d-4ec3-92df-1534780d3089\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5f387c-29888700-vtvtq"
Oct 30 01:00:00.622663 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:00.621060    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5279cb1-29b0-494a-b3b3-527e8af641dc-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888700-k6pvc\" (UID: \"d5279cb1-29b0-494a-b3b3-527e8af641dc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888700-k6pvc"
Oct 30 01:00:00.622663 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:00.621126    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f443227-6ffc-40f4-80aa-1485ae41aa02-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-11f521-29888700-khq6f\" (UID: \"7f443227-6ffc-40f4-80aa-1485ae41aa02\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-11f521-29888700-khq6f"
Oct 30 01:00:00.622663 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:00.621179    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzwmv\" (UniqueName: \"kubernetes.io/projected/38a379b2-37b6-4185-8319-2e62dcec38c8-kube-api-access-dzwmv\") pod \"maas-api-key-cleanup-e2e-inf-b-5007e1-29888700-dvm8j\" (UID: \"38a379b2-37b6-4185-8319-2e62dcec38c8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-5007e1-29888700-dvm8j"
Oct 30 01:00:00.622663 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:00.621215    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/743ebbf6-b4db-4774-b836-de5d7c716ca7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-cce513-29888700-744n2\" (UID: \"743ebbf6-b4db-4774-b836-de5d7c716ca7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-cce513-29888700-744n2"
Oct 30 01:00:00.622663 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:00.621244    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqn4x\" (UniqueName: \"kubernetes.io/projected/d22fc624-e367-49a0-ba47-9a5dcabdde1f-kube-api-access-cqn4x\") pod \"maas-api-key-cleanup-e2e-inf-a-8d2a70-29888700-jlmvv\" (UID: \"d22fc624-e367-49a0-ba47-9a5dcabdde1f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8d2a70-29888700-jlmvv"
Oct 30 01:00:00.622663 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:00.621335    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c111db1f-7e6f-44b8-b211-1658aee31337-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-1a11cf-29888700-d9s5c\" (UID: \"c111db1f-7e6f-44b8-b211-1658aee31337\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1a11cf-29888700-d9s5c"
Oct 30 01:00:00.622663 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:00.621363    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4sfq\" (UniqueName: \"kubernetes.io/projected/d5279cb1-29b0-494a-b3b3-527e8af641dc-kube-api-access-h4sfq\") pod \"maas-api-key-cleanup-29888700-k6pvc\" (UID: \"d5279cb1-29b0-494a-b3b3-527e8af641dc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888700-k6pvc"
Oct 30 01:00:00.622663 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:00.621459    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ns8b9\" (UniqueName: \"kubernetes.io/projected/7f443227-6ffc-40f4-80aa-1485ae41aa02-kube-api-access-ns8b9\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-11f521-29888700-khq6f\" (UID: \"7f443227-6ffc-40f4-80aa-1485ae41aa02\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-11f521-29888700-khq6f"
Oct 30 01:00:00.622663 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:00.621491    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bjnc\" (UniqueName: \"kubernetes.io/projected/5a907e40-190d-44fe-89b8-a51e621c04db-kube-api-access-5bjnc\") pod \"maas-api-key-cleanup-e2e-worker-w4-ca5363-29888700-trbnd\" (UID: \"5a907e40-190d-44fe-89b8-a51e621c04db\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ca5363-29888700-trbnd"
Oct 30 01:00:00.622663 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:00.621517    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4af343ba-bc4d-4ec3-92df-1534780d3089-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-5f387c-29888700-vtvtq\" (UID: \"4af343ba-bc4d-4ec3-92df-1534780d3089\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5f387c-29888700-vtvtq"
Oct 30 01:00:00.622663 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:00.621540    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gw4pf\" (UniqueName: \"kubernetes.io/projected/743ebbf6-b4db-4774-b836-de5d7c716ca7-kube-api-access-gw4pf\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-cce513-29888700-744n2\" (UID: \"743ebbf6-b4db-4774-b836-de5d7c716ca7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-cce513-29888700-744n2"
Oct 30 01:00:00.622663 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:00.621567    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d22fc624-e367-49a0-ba47-9a5dcabdde1f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-8d2a70-29888700-jlmvv\" (UID: \"d22fc624-e367-49a0-ba47-9a5dcabdde1f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8d2a70-29888700-jlmvv"
Oct 30 01:00:00.622663 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:00.621618    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a907e40-190d-44fe-89b8-a51e621c04db-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-ca5363-29888700-trbnd\" (UID: \"5a907e40-190d-44fe-89b8-a51e621c04db\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ca5363-29888700-trbnd"
Oct 30 01:00:00.622663 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:00.621907    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2xg2\" (UniqueName: \"kubernetes.io/projected/c111db1f-7e6f-44b8-b211-1658aee31337-kube-api-access-n2xg2\") pod \"maas-api-key-cleanup-e2e-worker-w2-1a11cf-29888700-d9s5c\" (UID: \"c111db1f-7e6f-44b8-b211-1658aee31337\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1a11cf-29888700-d9s5c"
Oct 30 01:00:00.622663 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:00.622413    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5279cb1-29b0-494a-b3b3-527e8af641dc-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888700-k6pvc\" (UID: \"d5279cb1-29b0-494a-b3b3-527e8af641dc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888700-k6pvc"
Oct 30 01:00:00.623551 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:00.622711    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f443227-6ffc-40f4-80aa-1485ae41aa02-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-11f521-29888700-khq6f\" (UID: \"7f443227-6ffc-40f4-80aa-1485ae41aa02\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-11f521-29888700-khq6f"
Oct 30 01:00:00.626061 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:00.626033    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ca5363-29888700-trbnd"]
Oct 30 01:00:00.630587 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:00.630254    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b3bdf9-29888700-pzr6b"]
Oct 30 01:00:00.638526 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:00.638505    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6f4fba-29888700-zvdkx"]
Oct 30 01:00:00.638766 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:00.638750    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b3bdf9-29888700-pzr6b"
Oct 30 01:00:00.643728 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:00.643710    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-bab19a-29888700-f7tvc"]
Oct 30 01:00:00.643848 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:00.643827    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6f4fba-29888700-zvdkx"
Oct 30 01:00:00.647746 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:00.647722    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4sfq\" (UniqueName: \"kubernetes.io/projected/d5279cb1-29b0-494a-b3b3-527e8af641dc-kube-api-access-h4sfq\") pod \"maas-api-key-cleanup-29888700-k6pvc\" (UID: \"d5279cb1-29b0-494a-b3b3-527e8af641dc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888700-k6pvc"
Oct 30 01:00:00.648262 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:00.648240    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b3bdf9-29888700-pzr6b"]
Oct 30 01:00:00.648454 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:00.648440    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-bab19a-29888700-f7tvc"
Oct 30 01:00:00.648612 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:00.648444    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-29be73-29888700-gvwp4"]
Oct 30 01:00:00.688122 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:00.688094    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ns8b9\" (UniqueName: \"kubernetes.io/projected/7f443227-6ffc-40f4-80aa-1485ae41aa02-kube-api-access-ns8b9\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-11f521-29888700-khq6f\" (UID: \"7f443227-6ffc-40f4-80aa-1485ae41aa02\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-11f521-29888700-khq6f"
Oct 30 01:00:00.704276 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:00.702691    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6f4fba-29888700-zvdkx"]
Oct 30 01:00:00.704276 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:00.702767    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-bab19a-29888700-f7tvc"]
Oct 30 01:00:00.704276 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:00.702801    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-11f521-29888700-khq6f"]
Oct 30 01:00:00.704276 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:00.702866    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-29be73-29888700-gvwp4"
Oct 30 01:00:00.704276 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:00.703245    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8e0f66-29888700-tnj7q"
Oct 30 01:00:00.707872 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:00.707831    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-481ab7-29888700-8mt68"]
Oct 30 01:00:00.716093 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:00.716072    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-29be73-29888700-gvwp4"]
Oct 30 01:00:00.722769 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:00.722741    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqn4x\" (UniqueName: \"kubernetes.io/projected/d22fc624-e367-49a0-ba47-9a5dcabdde1f-kube-api-access-cqn4x\") pod \"maas-api-key-cleanup-e2e-inf-a-8d2a70-29888700-jlmvv\" (UID: \"d22fc624-e367-49a0-ba47-9a5dcabdde1f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8d2a70-29888700-jlmvv"
Oct 30 01:00:00.722868 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:00.722794    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-st6zv\" (UniqueName: \"kubernetes.io/projected/6302c577-afc3-4632-a383-cb1fd8159992-kube-api-access-st6zv\") pod \"maas-api-key-cleanup-e2e-worker-w5-6f4fba-29888700-zvdkx\" (UID: \"6302c577-afc3-4632-a383-cb1fd8159992\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6f4fba-29888700-zvdkx"
Oct 30 01:00:00.722868 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:00.722825    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6c630bd-ac41-4607-8ed9-5f68cf00be38-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-bab19a-29888700-f7tvc\" (UID: \"b6c630bd-ac41-4607-8ed9-5f68cf00be38\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-bab19a-29888700-f7tvc"
Oct 30 01:00:00.723108 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:00.722879    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c111db1f-7e6f-44b8-b211-1658aee31337-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-1a11cf-29888700-d9s5c\" (UID: \"c111db1f-7e6f-44b8-b211-1658aee31337\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1a11cf-29888700-d9s5c"
Oct 30 01:00:00.723108 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:00.722958    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bjnc\" (UniqueName: \"kubernetes.io/projected/5a907e40-190d-44fe-89b8-a51e621c04db-kube-api-access-5bjnc\") pod \"maas-api-key-cleanup-e2e-worker-w4-ca5363-29888700-trbnd\" (UID: \"5a907e40-190d-44fe-89b8-a51e621c04db\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ca5363-29888700-trbnd"
Oct 30 01:00:00.723108 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:00.722987    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4af343ba-bc4d-4ec3-92df-1534780d3089-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-5f387c-29888700-vtvtq\" (UID: \"4af343ba-bc4d-4ec3-92df-1534780d3089\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5f387c-29888700-vtvtq"
Oct 30 01:00:00.723108 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:00.723012    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gw4pf\" (UniqueName: \"kubernetes.io/projected/743ebbf6-b4db-4774-b836-de5d7c716ca7-kube-api-access-gw4pf\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-cce513-29888700-744n2\" (UID: \"743ebbf6-b4db-4774-b836-de5d7c716ca7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-cce513-29888700-744n2"
Oct 30 01:00:00.723108 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:00.723038    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d22fc624-e367-49a0-ba47-9a5dcabdde1f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-8d2a70-29888700-jlmvv\" (UID: \"d22fc624-e367-49a0-ba47-9a5dcabdde1f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8d2a70-29888700-jlmvv"
Oct 30 01:00:00.723108 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:00.723074    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd72bc53-220d-40cd-abb8-36bc71b66094-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-b3bdf9-29888700-pzr6b\" (UID: \"bd72bc53-220d-40cd-abb8-36bc71b66094\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b3bdf9-29888700-pzr6b"
Oct 30 01:00:00.723314 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:00.723117    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a907e40-190d-44fe-89b8-a51e621c04db-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-ca5363-29888700-trbnd\" (UID: \"5a907e40-190d-44fe-89b8-a51e621c04db\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ca5363-29888700-trbnd"
Oct 30 01:00:00.723314 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:00.723151    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qp985\" (UniqueName: \"kubernetes.io/projected/b6c630bd-ac41-4607-8ed9-5f68cf00be38-kube-api-access-qp985\") pod \"maas-api-key-cleanup-e2e-api-b-bab19a-29888700-f7tvc\" (UID: \"b6c630bd-ac41-4607-8ed9-5f68cf00be38\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-bab19a-29888700-f7tvc"
Oct 30 01:00:00.723314 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:00.723232    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrkbd\" (UniqueName: \"kubernetes.io/projected/bd72bc53-220d-40cd-abb8-36bc71b66094-kube-api-access-qrkbd\") pod \"maas-api-key-cleanup-e2e-inf-b-b3bdf9-29888700-pzr6b\" (UID: \"bd72bc53-220d-40cd-abb8-36bc71b66094\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b3bdf9-29888700-pzr6b"
Oct 30 01:00:00.724069 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:00.724047    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c111db1f-7e6f-44b8-b211-1658aee31337-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-1a11cf-29888700-d9s5c\" (UID: \"c111db1f-7e6f-44b8-b211-1658aee31337\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1a11cf-29888700-d9s5c"
Oct 30 01:00:00.724115 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:00.724047    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a907e40-190d-44fe-89b8-a51e621c04db-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-ca5363-29888700-trbnd\" (UID: \"5a907e40-190d-44fe-89b8-a51e621c04db\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ca5363-29888700-trbnd"
Oct 30 01:00:00.724222 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:00.724203    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6302c577-afc3-4632-a383-cb1fd8159992-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-6f4fba-29888700-zvdkx\" (UID: \"6302c577-afc3-4632-a383-cb1fd8159992\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6f4fba-29888700-zvdkx"
Oct 30 01:00:00.724313 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:00.724251    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2xg2\" (UniqueName: \"kubernetes.io/projected/c111db1f-7e6f-44b8-b211-1658aee31337-kube-api-access-n2xg2\") pod \"maas-api-key-cleanup-e2e-worker-w2-1a11cf-29888700-d9s5c\" (UID: \"c111db1f-7e6f-44b8-b211-1658aee31337\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1a11cf-29888700-d9s5c"
Oct 30 01:00:00.724313 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:00.724280    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b15c8d59-8957-4e66-8e15-998c8a33e690-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-481ab7-29888700-8mt68\" (UID: \"b15c8d59-8957-4e66-8e15-998c8a33e690\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-481ab7-29888700-8mt68"
Oct 30 01:00:00.724410 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:00.724310    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tt8fq\" (UniqueName: \"kubernetes.io/projected/b15c8d59-8957-4e66-8e15-998c8a33e690-kube-api-access-tt8fq\") pod \"maas-api-key-cleanup-e2e-inf-a-481ab7-29888700-8mt68\" (UID: \"b15c8d59-8957-4e66-8e15-998c8a33e690\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-481ab7-29888700-8mt68"
Oct 30 01:00:00.724410 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:00.724369    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38a379b2-37b6-4185-8319-2e62dcec38c8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-5007e1-29888700-dvm8j\" (UID: \"38a379b2-37b6-4185-8319-2e62dcec38c8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-5007e1-29888700-dvm8j"
Oct 30 01:00:00.724488 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:00.724436    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79jrw\" (UniqueName: \"kubernetes.io/projected/4af343ba-bc4d-4ec3-92df-1534780d3089-kube-api-access-79jrw\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-5f387c-29888700-vtvtq\" (UID: \"4af343ba-bc4d-4ec3-92df-1534780d3089\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5f387c-29888700-vtvtq"
Oct 30 01:00:00.725069 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:00.725052    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4af343ba-bc4d-4ec3-92df-1534780d3089-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-5f387c-29888700-vtvtq\" (UID: \"4af343ba-bc4d-4ec3-92df-1534780d3089\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5f387c-29888700-vtvtq"
Oct 30 01:00:00.725528 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:00.725511    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38a379b2-37b6-4185-8319-2e62dcec38c8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-5007e1-29888700-dvm8j\" (UID: \"38a379b2-37b6-4185-8319-2e62dcec38c8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-5007e1-29888700-dvm8j"
Oct 30 01:00:00.725612 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:00.725601    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzwmv\" (UniqueName: \"kubernetes.io/projected/38a379b2-37b6-4185-8319-2e62dcec38c8-kube-api-access-dzwmv\") pod \"maas-api-key-cleanup-e2e-inf-b-5007e1-29888700-dvm8j\" (UID: \"38a379b2-37b6-4185-8319-2e62dcec38c8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-5007e1-29888700-dvm8j"
Oct 30 01:00:00.725650 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:00.725626    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/743ebbf6-b4db-4774-b836-de5d7c716ca7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-cce513-29888700-744n2\" (UID: \"743ebbf6-b4db-4774-b836-de5d7c716ca7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-cce513-29888700-744n2"
Oct 30 01:00:00.726128 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:00.726112    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/743ebbf6-b4db-4774-b836-de5d7c716ca7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-cce513-29888700-744n2\" (UID: \"743ebbf6-b4db-4774-b836-de5d7c716ca7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-cce513-29888700-744n2"
Oct 30 01:00:00.726772 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:00.726756    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d22fc624-e367-49a0-ba47-9a5dcabdde1f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-8d2a70-29888700-jlmvv\" (UID: \"d22fc624-e367-49a0-ba47-9a5dcabdde1f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8d2a70-29888700-jlmvv"
Oct 30 01:00:00.738809 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:00.738786    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-52650f-29888700-8mvn6"
Oct 30 01:00:00.773443 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:00.749610    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqn4x\" (UniqueName: \"kubernetes.io/projected/d22fc624-e367-49a0-ba47-9a5dcabdde1f-kube-api-access-cqn4x\") pod \"maas-api-key-cleanup-e2e-inf-a-8d2a70-29888700-jlmvv\" (UID: \"d22fc624-e367-49a0-ba47-9a5dcabdde1f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8d2a70-29888700-jlmvv"
Oct 30 01:00:00.796225 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:00.796154    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bjnc\" (UniqueName: \"kubernetes.io/projected/5a907e40-190d-44fe-89b8-a51e621c04db-kube-api-access-5bjnc\") pod \"maas-api-key-cleanup-e2e-worker-w4-ca5363-29888700-trbnd\" (UID: \"5a907e40-190d-44fe-89b8-a51e621c04db\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ca5363-29888700-trbnd"
Oct 30 01:00:00.796399 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:00.796350    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2xg2\" (UniqueName: \"kubernetes.io/projected/c111db1f-7e6f-44b8-b211-1658aee31337-kube-api-access-n2xg2\") pod \"maas-api-key-cleanup-e2e-worker-w2-1a11cf-29888700-d9s5c\" (UID: \"c111db1f-7e6f-44b8-b211-1658aee31337\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1a11cf-29888700-d9s5c"
Oct 30 01:00:00.797546 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:00.797508    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79jrw\" (UniqueName: \"kubernetes.io/projected/4af343ba-bc4d-4ec3-92df-1534780d3089-kube-api-access-79jrw\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-5f387c-29888700-vtvtq\" (UID: \"4af343ba-bc4d-4ec3-92df-1534780d3089\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5f387c-29888700-vtvtq"
Oct 30 01:00:00.798012 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:00.797994    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gw4pf\" (UniqueName: \"kubernetes.io/projected/743ebbf6-b4db-4774-b836-de5d7c716ca7-kube-api-access-gw4pf\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-cce513-29888700-744n2\" (UID: \"743ebbf6-b4db-4774-b836-de5d7c716ca7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-cce513-29888700-744n2"
Oct 30 01:00:00.800311 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:00.800266    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzwmv\" (UniqueName: \"kubernetes.io/projected/38a379b2-37b6-4185-8319-2e62dcec38c8-kube-api-access-dzwmv\") pod \"maas-api-key-cleanup-e2e-inf-b-5007e1-29888700-dvm8j\" (UID: \"38a379b2-37b6-4185-8319-2e62dcec38c8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-5007e1-29888700-dvm8j"
Oct 30 01:00:00.803756 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:00.803591    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8d2a70-29888700-jlmvv"
Oct 30 01:00:00.824857 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:00.824586    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888700-k6pvc"
Oct 30 01:00:00.895081 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:00.829529    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-st6zv\" (UniqueName: \"kubernetes.io/projected/6302c577-afc3-4632-a383-cb1fd8159992-kube-api-access-st6zv\") pod \"maas-api-key-cleanup-e2e-worker-w5-6f4fba-29888700-zvdkx\" (UID: \"6302c577-afc3-4632-a383-cb1fd8159992\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6f4fba-29888700-zvdkx"
Oct 30 01:00:00.895081 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:00.829572    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6c630bd-ac41-4607-8ed9-5f68cf00be38-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-bab19a-29888700-f7tvc\" (UID: \"b6c630bd-ac41-4607-8ed9-5f68cf00be38\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-bab19a-29888700-f7tvc"
Oct 30 01:00:00.895081 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:00.829632    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86bc1f67-d9db-4d33-8bfc-e57494705d80-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-29be73-29888700-gvwp4\" (UID: \"86bc1f67-d9db-4d33-8bfc-e57494705d80\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-29be73-29888700-gvwp4"
Oct 30 01:00:00.895081 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:00.829709    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd72bc53-220d-40cd-abb8-36bc71b66094-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-b3bdf9-29888700-pzr6b\" (UID: \"bd72bc53-220d-40cd-abb8-36bc71b66094\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b3bdf9-29888700-pzr6b"
Oct 30 01:00:00.895081 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:00.829836    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qp985\" (UniqueName: \"kubernetes.io/projected/b6c630bd-ac41-4607-8ed9-5f68cf00be38-kube-api-access-qp985\") pod \"maas-api-key-cleanup-e2e-api-b-bab19a-29888700-f7tvc\" (UID: \"b6c630bd-ac41-4607-8ed9-5f68cf00be38\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-bab19a-29888700-f7tvc"
Oct 30 01:00:00.895081 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:00.829991    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrkbd\" (UniqueName: \"kubernetes.io/projected/bd72bc53-220d-40cd-abb8-36bc71b66094-kube-api-access-qrkbd\") pod \"maas-api-key-cleanup-e2e-inf-b-b3bdf9-29888700-pzr6b\" (UID: \"bd72bc53-220d-40cd-abb8-36bc71b66094\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b3bdf9-29888700-pzr6b"
Oct 30 01:00:00.895081 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:00.830044    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7l5xn\" (UniqueName: \"kubernetes.io/projected/86bc1f67-d9db-4d33-8bfc-e57494705d80-kube-api-access-7l5xn\") pod \"maas-api-key-cleanup-e2e-worker-w1-29be73-29888700-gvwp4\" (UID: \"86bc1f67-d9db-4d33-8bfc-e57494705d80\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-29be73-29888700-gvwp4"
Oct 30 01:00:00.895081 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:00.830077    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6302c577-afc3-4632-a383-cb1fd8159992-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-6f4fba-29888700-zvdkx\" (UID: \"6302c577-afc3-4632-a383-cb1fd8159992\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6f4fba-29888700-zvdkx"
Oct 30 01:00:00.895081 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:00.830117    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b15c8d59-8957-4e66-8e15-998c8a33e690-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-481ab7-29888700-8mt68\" (UID: \"b15c8d59-8957-4e66-8e15-998c8a33e690\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-481ab7-29888700-8mt68"
Oct 30 01:00:00.895081 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:00.830146    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tt8fq\" (UniqueName: \"kubernetes.io/projected/b15c8d59-8957-4e66-8e15-998c8a33e690-kube-api-access-tt8fq\") pod \"maas-api-key-cleanup-e2e-inf-a-481ab7-29888700-8mt68\" (UID: \"b15c8d59-8957-4e66-8e15-998c8a33e690\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-481ab7-29888700-8mt68"
Oct 30 01:00:00.973540 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:00.899577    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6302c577-afc3-4632-a383-cb1fd8159992-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-6f4fba-29888700-zvdkx\" (UID: \"6302c577-afc3-4632-a383-cb1fd8159992\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6f4fba-29888700-zvdkx"
Oct 30 01:00:00.973540 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:00.900232    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b15c8d59-8957-4e66-8e15-998c8a33e690-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-481ab7-29888700-8mt68\" (UID: \"b15c8d59-8957-4e66-8e15-998c8a33e690\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-481ab7-29888700-8mt68"
Oct 30 01:00:00.973540 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:00.933590    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd72bc53-220d-40cd-abb8-36bc71b66094-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-b3bdf9-29888700-pzr6b\" (UID: \"bd72bc53-220d-40cd-abb8-36bc71b66094\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b3bdf9-29888700-pzr6b"
Oct 30 01:00:00.973540 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:00.937243    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5f387c-29888700-vtvtq"
Oct 30 01:00:00.973540 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:00.937410    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-11f521-29888700-khq6f"
Oct 30 01:00:00.973540 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:00.939311    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7l5xn\" (UniqueName: \"kubernetes.io/projected/86bc1f67-d9db-4d33-8bfc-e57494705d80-kube-api-access-7l5xn\") pod \"maas-api-key-cleanup-e2e-worker-w1-29be73-29888700-gvwp4\" (UID: \"86bc1f67-d9db-4d33-8bfc-e57494705d80\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-29be73-29888700-gvwp4"
Oct 30 01:00:00.973540 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:00.939576    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86bc1f67-d9db-4d33-8bfc-e57494705d80-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-29be73-29888700-gvwp4\" (UID: \"86bc1f67-d9db-4d33-8bfc-e57494705d80\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-29be73-29888700-gvwp4"
Oct 30 01:00:00.973540 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:00.950043    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86bc1f67-d9db-4d33-8bfc-e57494705d80-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-29be73-29888700-gvwp4\" (UID: \"86bc1f67-d9db-4d33-8bfc-e57494705d80\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-29be73-29888700-gvwp4"
Oct 30 01:00:00.973540 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:00.965681    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrkbd\" (UniqueName: \"kubernetes.io/projected/bd72bc53-220d-40cd-abb8-36bc71b66094-kube-api-access-qrkbd\") pod \"maas-api-key-cleanup-e2e-inf-b-b3bdf9-29888700-pzr6b\" (UID: \"bd72bc53-220d-40cd-abb8-36bc71b66094\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b3bdf9-29888700-pzr6b"
Oct 30 01:00:01.076822 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:01.009211    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1a11cf-29888700-d9s5c"
Oct 30 01:00:01.076822 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:01.010673    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-cce513-29888700-744n2"
Oct 30 01:00:01.076822 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:01.024647    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qp985\" (UniqueName: \"kubernetes.io/projected/b6c630bd-ac41-4607-8ed9-5f68cf00be38-kube-api-access-qp985\") pod \"maas-api-key-cleanup-e2e-api-b-bab19a-29888700-f7tvc\" (UID: \"b6c630bd-ac41-4607-8ed9-5f68cf00be38\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-bab19a-29888700-f7tvc"
Oct 30 01:00:01.076822 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:01.030648    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-5007e1-29888700-dvm8j"
Oct 30 01:00:01.076822 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:01.030738    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6c630bd-ac41-4607-8ed9-5f68cf00be38-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-bab19a-29888700-f7tvc\" (UID: \"b6c630bd-ac41-4607-8ed9-5f68cf00be38\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-bab19a-29888700-f7tvc"
Oct 30 01:00:01.076822 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:01.032027    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tt8fq\" (UniqueName: \"kubernetes.io/projected/b15c8d59-8957-4e66-8e15-998c8a33e690-kube-api-access-tt8fq\") pod \"maas-api-key-cleanup-e2e-inf-a-481ab7-29888700-8mt68\" (UID: \"b15c8d59-8957-4e66-8e15-998c8a33e690\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-481ab7-29888700-8mt68"
Oct 30 01:00:01.076822 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:01.071817    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b3bdf9-29888700-pzr6b"
Oct 30 01:00:01.076822 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:01.073102    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ca5363-29888700-trbnd"
Oct 30 01:00:01.201526 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:01.199852    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-481ab7-29888700-8mt68"
Oct 30 01:00:01.327079 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:01.326966    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7l5xn\" (UniqueName: \"kubernetes.io/projected/86bc1f67-d9db-4d33-8bfc-e57494705d80-kube-api-access-7l5xn\") pod \"maas-api-key-cleanup-e2e-worker-w1-29be73-29888700-gvwp4\" (UID: \"86bc1f67-d9db-4d33-8bfc-e57494705d80\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-29be73-29888700-gvwp4"
Oct 30 01:00:01.332835 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:01.332805    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-29be73-29888700-gvwp4"
Oct 30 01:00:01.343767 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:01.343735    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-bab19a-29888700-f7tvc"
Oct 30 01:00:01.414034 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:01.412831    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-st6zv\" (UniqueName: \"kubernetes.io/projected/6302c577-afc3-4632-a383-cb1fd8159992-kube-api-access-st6zv\") pod \"maas-api-key-cleanup-e2e-worker-w5-6f4fba-29888700-zvdkx\" (UID: \"6302c577-afc3-4632-a383-cb1fd8159992\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6f4fba-29888700-zvdkx"
Oct 30 01:00:01.527157 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:01.524264    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-03d892/test-model-03d892-kserve-7f8c7cb699-kp2mb"]
Oct 30 01:00:01.542417 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:01.541452    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-03d892/test-model-03d892-kserve-7f8c7cb699-kp2mb"]
Oct 30 01:00:01.542417 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:01.541590    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-03d892/test-model-03d892-kserve-7f8c7cb699-kp2mb"
Oct 30 01:00:01.562986 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:01.562953    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-03d892\"/\"kube-root-ca.crt\""
Oct 30 01:00:01.563131 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:01.563029    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-03d892\"/\"openshift-service-ca.crt\""
Oct 30 01:00:01.580158 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:01.566226    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-03d892\"/\"test-model-03d892-kserve-self-signed-certs\""
Oct 30 01:00:01.580158 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:01.566408    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-03d892\"/\"default-dockercfg-rtj8n\""
Oct 30 01:00:01.711084 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:01.711056    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6f4fba-29888700-zvdkx"
Oct 30 01:00:01.713805 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:01.713761    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3e31ee3-d2eb-42b1-84dc-173ed4434f84-model-cache\") pod \"test-model-03d892-kserve-7f8c7cb699-kp2mb\" (UID: \"b3e31ee3-d2eb-42b1-84dc-173ed4434f84\") " pod="ai-tenant-e2e-inf-b-03d892/test-model-03d892-kserve-7f8c7cb699-kp2mb"
Oct 30 01:00:01.714219 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:01.714196    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8l9n7\" (UniqueName: \"kubernetes.io/projected/b3e31ee3-d2eb-42b1-84dc-173ed4434f84-kube-api-access-8l9n7\") pod \"test-model-03d892-kserve-7f8c7cb699-kp2mb\" (UID: \"b3e31ee3-d2eb-42b1-84dc-173ed4434f84\") " pod="ai-tenant-e2e-inf-b-03d892/test-model-03d892-kserve-7f8c7cb699-kp2mb"
Oct 30 01:00:01.714442 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:01.714391    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b3e31ee3-d2eb-42b1-84dc-173ed4434f84-dshm\") pod \"test-model-03d892-kserve-7f8c7cb699-kp2mb\" (UID: \"b3e31ee3-d2eb-42b1-84dc-173ed4434f84\") " pod="ai-tenant-e2e-inf-b-03d892/test-model-03d892-kserve-7f8c7cb699-kp2mb"
Oct 30 01:00:01.714681 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:01.714663    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3e31ee3-d2eb-42b1-84dc-173ed4434f84-tmp-dir\") pod \"test-model-03d892-kserve-7f8c7cb699-kp2mb\" (UID: \"b3e31ee3-d2eb-42b1-84dc-173ed4434f84\") " pod="ai-tenant-e2e-inf-b-03d892/test-model-03d892-kserve-7f8c7cb699-kp2mb"
Oct 30 01:00:01.714780 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:01.714767    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3e31ee3-d2eb-42b1-84dc-173ed4434f84-tls-certs\") pod \"test-model-03d892-kserve-7f8c7cb699-kp2mb\" (UID: \"b3e31ee3-d2eb-42b1-84dc-173ed4434f84\") " pod="ai-tenant-e2e-inf-b-03d892/test-model-03d892-kserve-7f8c7cb699-kp2mb"
Oct 30 01:00:01.728569 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:01.728533    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b3e31ee3-d2eb-42b1-84dc-173ed4434f84-home\") pod \"test-model-03d892-kserve-7f8c7cb699-kp2mb\" (UID: \"b3e31ee3-d2eb-42b1-84dc-173ed4434f84\") " pod="ai-tenant-e2e-inf-b-03d892/test-model-03d892-kserve-7f8c7cb699-kp2mb"
Oct 30 01:00:02.070668 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:02.049003    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3e31ee3-d2eb-42b1-84dc-173ed4434f84-tmp-dir\") pod \"test-model-03d892-kserve-7f8c7cb699-kp2mb\" (UID: \"b3e31ee3-d2eb-42b1-84dc-173ed4434f84\") " pod="ai-tenant-e2e-inf-b-03d892/test-model-03d892-kserve-7f8c7cb699-kp2mb"
Oct 30 01:00:02.070668 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:02.049068    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3e31ee3-d2eb-42b1-84dc-173ed4434f84-tls-certs\") pod \"test-model-03d892-kserve-7f8c7cb699-kp2mb\" (UID: \"b3e31ee3-d2eb-42b1-84dc-173ed4434f84\") " pod="ai-tenant-e2e-inf-b-03d892/test-model-03d892-kserve-7f8c7cb699-kp2mb"
Oct 30 01:00:02.070668 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:02.049105    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b3e31ee3-d2eb-42b1-84dc-173ed4434f84-home\") pod \"test-model-03d892-kserve-7f8c7cb699-kp2mb\" (UID: \"b3e31ee3-d2eb-42b1-84dc-173ed4434f84\") " pod="ai-tenant-e2e-inf-b-03d892/test-model-03d892-kserve-7f8c7cb699-kp2mb"
Oct 30 01:00:02.070668 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:02.049359    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3e31ee3-d2eb-42b1-84dc-173ed4434f84-model-cache\") pod \"test-model-03d892-kserve-7f8c7cb699-kp2mb\" (UID: \"b3e31ee3-d2eb-42b1-84dc-173ed4434f84\") " pod="ai-tenant-e2e-inf-b-03d892/test-model-03d892-kserve-7f8c7cb699-kp2mb"
Oct 30 01:00:02.070668 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:02.049498    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8l9n7\" (UniqueName: \"kubernetes.io/projected/b3e31ee3-d2eb-42b1-84dc-173ed4434f84-kube-api-access-8l9n7\") pod \"test-model-03d892-kserve-7f8c7cb699-kp2mb\" (UID: \"b3e31ee3-d2eb-42b1-84dc-173ed4434f84\") " pod="ai-tenant-e2e-inf-b-03d892/test-model-03d892-kserve-7f8c7cb699-kp2mb"
Oct 30 01:00:02.070668 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:02.049560    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b3e31ee3-d2eb-42b1-84dc-173ed4434f84-dshm\") pod \"test-model-03d892-kserve-7f8c7cb699-kp2mb\" (UID: \"b3e31ee3-d2eb-42b1-84dc-173ed4434f84\") " pod="ai-tenant-e2e-inf-b-03d892/test-model-03d892-kserve-7f8c7cb699-kp2mb"
Oct 30 01:00:02.087407 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:02.081580    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3e31ee3-d2eb-42b1-84dc-173ed4434f84-tmp-dir\") pod \"test-model-03d892-kserve-7f8c7cb699-kp2mb\" (UID: \"b3e31ee3-d2eb-42b1-84dc-173ed4434f84\") " pod="ai-tenant-e2e-inf-b-03d892/test-model-03d892-kserve-7f8c7cb699-kp2mb"
Oct 30 01:00:02.101442 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:02.099958    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3e31ee3-d2eb-42b1-84dc-173ed4434f84-model-cache\") pod \"test-model-03d892-kserve-7f8c7cb699-kp2mb\" (UID: \"b3e31ee3-d2eb-42b1-84dc-173ed4434f84\") " pod="ai-tenant-e2e-inf-b-03d892/test-model-03d892-kserve-7f8c7cb699-kp2mb"
Oct 30 01:00:02.101442 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:02.100247    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b3e31ee3-d2eb-42b1-84dc-173ed4434f84-home\") pod \"test-model-03d892-kserve-7f8c7cb699-kp2mb\" (UID: \"b3e31ee3-d2eb-42b1-84dc-173ed4434f84\") " pod="ai-tenant-e2e-inf-b-03d892/test-model-03d892-kserve-7f8c7cb699-kp2mb"
Oct 30 01:00:02.220487 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:02.217305    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8l9n7\" (UniqueName: \"kubernetes.io/projected/b3e31ee3-d2eb-42b1-84dc-173ed4434f84-kube-api-access-8l9n7\") pod \"test-model-03d892-kserve-7f8c7cb699-kp2mb\" (UID: \"b3e31ee3-d2eb-42b1-84dc-173ed4434f84\") " pod="ai-tenant-e2e-inf-b-03d892/test-model-03d892-kserve-7f8c7cb699-kp2mb"
Oct 30 01:00:02.230984 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:02.230954    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3e31ee3-d2eb-42b1-84dc-173ed4434f84-tls-certs\") pod \"test-model-03d892-kserve-7f8c7cb699-kp2mb\" (UID: \"b3e31ee3-d2eb-42b1-84dc-173ed4434f84\") " pod="ai-tenant-e2e-inf-b-03d892/test-model-03d892-kserve-7f8c7cb699-kp2mb"
Oct 30 01:00:02.237283 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:02.237255    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b3e31ee3-d2eb-42b1-84dc-173ed4434f84-dshm\") pod \"test-model-03d892-kserve-7f8c7cb699-kp2mb\" (UID: \"b3e31ee3-d2eb-42b1-84dc-173ed4434f84\") " pod="ai-tenant-e2e-inf-b-03d892/test-model-03d892-kserve-7f8c7cb699-kp2mb"
Oct 30 01:00:02.240061 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:02.240036    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8e0f66-29888700-tnj7q"]
Oct 30 01:00:02.507934 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:02.507783    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-03d892/test-model-03d892-kserve-7f8c7cb699-kp2mb"
Oct 30 01:00:02.641523 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:02.641478    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8e0f66-29888700-tnj7q" event={"ID":"5fa9212f-5c4d-47a2-a855-2c18a807533e","Type":"ContainerStarted","Data":"643b6f3b817905b70040447dc85d07aaf97d0845e7234d7ec03a1ebb9f6666a1"}
Oct 30 01:00:03.045828 ip-10-0-128-7 kubenswrapper[5002]: W1030 01:00:03.045793    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09fb128c_0670_4260_a77b_ae32de23fa0d.slice/crio-a1bb82837f5663136ee41cba1c81976e1b0622b6dc75047cd199f96448ee7ddf WatchSource:0}: Error finding container a1bb82837f5663136ee41cba1c81976e1b0622b6dc75047cd199f96448ee7ddf: Status 404 returned error can't find the container with id a1bb82837f5663136ee41cba1c81976e1b0622b6dc75047cd199f96448ee7ddf
Oct 30 01:00:03.047946 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:03.047629    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-52650f-29888700-8mvn6"]
Oct 30 01:00:03.256487 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:03.253560    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8d2a70-29888700-jlmvv"]
Oct 30 01:00:03.382129 ip-10-0-128-7 kubenswrapper[5002]: W1030 01:00:03.382072    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc111db1f_7e6f_44b8_b211_1658aee31337.slice/crio-142521fcab562ea0b58b9231fdab8633931b469ee378fc9eafcf72a814912499 WatchSource:0}: Error finding container 142521fcab562ea0b58b9231fdab8633931b469ee378fc9eafcf72a814912499: Status 404 returned error can't find the container with id 142521fcab562ea0b58b9231fdab8633931b469ee378fc9eafcf72a814912499
Oct 30 01:00:03.382247 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:03.382122    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1a11cf-29888700-d9s5c"]
Oct 30 01:00:03.384980 ip-10-0-128-7 kubenswrapper[5002]: W1030 01:00:03.384802    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a907e40_190d_44fe_89b8_a51e621c04db.slice/crio-595e74c00f33e815ccc45c5ac91f8a591834b4e3cd2e92c84b4b2a6c45a8b522 WatchSource:0}: Error finding container 595e74c00f33e815ccc45c5ac91f8a591834b4e3cd2e92c84b4b2a6c45a8b522: Status 404 returned error can't find the container with id 595e74c00f33e815ccc45c5ac91f8a591834b4e3cd2e92c84b4b2a6c45a8b522
Oct 30 01:00:03.389663 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:03.387817    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ca5363-29888700-trbnd"]
Oct 30 01:00:03.394665 ip-10-0-128-7 kubenswrapper[5002]: W1030 01:00:03.394614    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6302c577_afc3_4632_a383_cb1fd8159992.slice/crio-4a8d23dd4f36183583e7ca7ac1a6c372b5e478a9a3d7b54f1e3f521ca34d2227 WatchSource:0}: Error finding container 4a8d23dd4f36183583e7ca7ac1a6c372b5e478a9a3d7b54f1e3f521ca34d2227: Status 404 returned error can't find the container with id 4a8d23dd4f36183583e7ca7ac1a6c372b5e478a9a3d7b54f1e3f521ca34d2227
Oct 30 01:00:03.397780 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:03.397680    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6f4fba-29888700-zvdkx"]
Oct 30 01:00:03.412717 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:03.412691    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888700-k6pvc"]
Oct 30 01:00:03.435870 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:03.435845    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-481ab7-29888700-8mt68"]
Oct 30 01:00:03.448184 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:03.448158    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-11f521-29888700-khq6f"]
Oct 30 01:00:03.464719 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:03.459044    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-29be73-29888700-gvwp4"]
Oct 30 01:00:03.473275 ip-10-0-128-7 kubenswrapper[5002]: W1030 01:00:03.473235    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f443227_6ffc_40f4_80aa_1485ae41aa02.slice/crio-6080d5850f8b58432aba64ab7f83c20f4392b3f9d229b27c062521c90284f67f WatchSource:0}: Error finding container 6080d5850f8b58432aba64ab7f83c20f4392b3f9d229b27c062521c90284f67f: Status 404 returned error can't find the container with id 6080d5850f8b58432aba64ab7f83c20f4392b3f9d229b27c062521c90284f67f
Oct 30 01:00:03.511523 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:03.510504    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-5007e1-29888700-dvm8j"]
Oct 30 01:00:03.516162 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:03.516116    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-bab19a-29888700-f7tvc"]
Oct 30 01:00:03.523512 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:03.522483    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5f387c-29888700-vtvtq"]
Oct 30 01:00:03.529845 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:03.529186    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-cce513-29888700-744n2"]
Oct 30 01:00:03.539399 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:03.539353    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-03d892/test-model-03d892-kserve-7f8c7cb699-kp2mb"]
Oct 30 01:00:03.548549 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:03.548527    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b3bdf9-29888700-pzr6b"]
Oct 30 01:00:03.580660 ip-10-0-128-7 kubenswrapper[5002]: W1030 01:00:03.580625    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb3e31ee3_d2eb_42b1_84dc_173ed4434f84.slice/crio-21a76f73e177bad352b4f33477c04d62cd011d33833f0b8e4a2dd1334ac17658 WatchSource:0}: Error finding container 21a76f73e177bad352b4f33477c04d62cd011d33833f0b8e4a2dd1334ac17658: Status 404 returned error can't find the container with id 21a76f73e177bad352b4f33477c04d62cd011d33833f0b8e4a2dd1334ac17658
Oct 30 01:00:03.664835 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:03.664768    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888700-k6pvc" event={"ID":"d5279cb1-29b0-494a-b3b3-527e8af641dc","Type":"ContainerStarted","Data":"2b32619ef3a63ad753c8612623a8fc61f2e9d29aebee0628a89d053a2c3e35e0"}
Oct 30 01:00:03.682495 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:03.681536    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-52650f-29888700-8mvn6" event={"ID":"09fb128c-0670-4260-a77b-ae32de23fa0d","Type":"ContainerStarted","Data":"edf3f54ca05163562bb6b26d29347c522a0aaf4b18301e8b82d4bb20e3748d59"}
Oct 30 01:00:03.682495 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:03.681633    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-52650f-29888700-8mvn6" event={"ID":"09fb128c-0670-4260-a77b-ae32de23fa0d","Type":"ContainerStarted","Data":"a1bb82837f5663136ee41cba1c81976e1b0622b6dc75047cd199f96448ee7ddf"}
Oct 30 01:00:03.685246 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:03.685218    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-29be73-29888700-gvwp4" event={"ID":"86bc1f67-d9db-4d33-8bfc-e57494705d80","Type":"ContainerStarted","Data":"4072e33018d75cc2ff780267df1f19570810527919cca1d70810556f99fcad80"}
Oct 30 01:00:03.686730 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:03.686681    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-5007e1-29888700-dvm8j" event={"ID":"38a379b2-37b6-4185-8319-2e62dcec38c8","Type":"ContainerStarted","Data":"9b99a1f2a2e06266a062454ab57913512ef59c758eb1ad3c8eb89c28c4b6c2e4"}
Oct 30 01:00:03.692399 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:03.688447    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b3bdf9-29888700-pzr6b" event={"ID":"bd72bc53-220d-40cd-abb8-36bc71b66094","Type":"ContainerStarted","Data":"735dfe0534ef8e9b6e6a2607faa556c025a3be9234b1edd214910f13eee8b2a1"}
Oct 30 01:00:03.692399 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:03.691012    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-cce513-29888700-744n2" event={"ID":"743ebbf6-b4db-4774-b836-de5d7c716ca7","Type":"ContainerStarted","Data":"93731423201ab10194ea8664832f92c54c4c039bb16056f112d14d83fdb611a4"}
Oct 30 01:00:03.693181 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:03.693098    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ca5363-29888700-trbnd" event={"ID":"5a907e40-190d-44fe-89b8-a51e621c04db","Type":"ContainerStarted","Data":"595e74c00f33e815ccc45c5ac91f8a591834b4e3cd2e92c84b4b2a6c45a8b522"}
Oct 30 01:00:03.743483 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:03.742344    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8d2a70-29888700-jlmvv" event={"ID":"d22fc624-e367-49a0-ba47-9a5dcabdde1f","Type":"ContainerStarted","Data":"09eb8897e3e1223c880ca52859273b20820261f9857fe6d90588ce957326b359"}
Oct 30 01:00:03.744026 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:03.744002    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8e0f66-29888700-tnj7q" event={"ID":"5fa9212f-5c4d-47a2-a855-2c18a807533e","Type":"ContainerStarted","Data":"cf86b1ce356cb7a39624de52160b766d4f9fd06a894479ffb91de86e4a7c60f8"}
Oct 30 01:00:03.746372 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:03.746343    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1a11cf-29888700-d9s5c" event={"ID":"c111db1f-7e6f-44b8-b211-1658aee31337","Type":"ContainerStarted","Data":"142521fcab562ea0b58b9231fdab8633931b469ee378fc9eafcf72a814912499"}
Oct 30 01:00:03.748107 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:03.748083    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-481ab7-29888700-8mt68" event={"ID":"b15c8d59-8957-4e66-8e15-998c8a33e690","Type":"ContainerStarted","Data":"581327734d6adbc745024694636fbc74dfc6e7dce3652db02111447ed747b59a"}
Oct 30 01:00:03.749685 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:03.749625    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-03d892/test-model-03d892-kserve-7f8c7cb699-kp2mb" event={"ID":"b3e31ee3-d2eb-42b1-84dc-173ed4434f84","Type":"ContainerStarted","Data":"21a76f73e177bad352b4f33477c04d62cd011d33833f0b8e4a2dd1334ac17658"}
Oct 30 01:00:03.751565 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:03.751495    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5f387c-29888700-vtvtq" event={"ID":"4af343ba-bc4d-4ec3-92df-1534780d3089","Type":"ContainerStarted","Data":"d1332ba06dbf8ad951d7b5f26be348b2d3b05ec9905a3f16c7d1b103a4b0433c"}
Oct 30 01:00:03.753471 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:03.753443    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-11f521-29888700-khq6f" event={"ID":"7f443227-6ffc-40f4-80aa-1485ae41aa02","Type":"ContainerStarted","Data":"6080d5850f8b58432aba64ab7f83c20f4392b3f9d229b27c062521c90284f67f"}
Oct 30 01:00:03.765279 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:03.758292    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6f4fba-29888700-zvdkx" event={"ID":"6302c577-afc3-4632-a383-cb1fd8159992","Type":"ContainerStarted","Data":"4a8d23dd4f36183583e7ca7ac1a6c372b5e478a9a3d7b54f1e3f521ca34d2227"}
Oct 30 01:00:03.774339 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:03.774307    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-bab19a-29888700-f7tvc" event={"ID":"b6c630bd-ac41-4607-8ed9-5f68cf00be38","Type":"ContainerStarted","Data":"44c5a3ceda959040a702d560da55c8274a12f3d852664a3f4cda1030ce1e4518"}
Oct 30 01:00:03.778269 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:03.776102    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-52650f-29888700-8mvn6" podStartSLOduration=3.7760856350000003 podStartE2EDuration="3.776085635s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:00:03.770161697 +0000 UTC m=+1699.386594577" watchObservedRunningTime="2026-10-30 01:00:03.776085635 +0000 UTC m=+1699.392518515"
Oct 30 01:00:03.816664 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:03.816617    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8e0f66-29888700-tnj7q" podStartSLOduration=3.816604786 podStartE2EDuration="3.816604786s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:00:03.813195228 +0000 UTC m=+1699.429628107" watchObservedRunningTime="2026-10-30 01:00:03.816604786 +0000 UTC m=+1699.433037664"
Oct 30 01:00:04.789299 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:04.789259    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b3bdf9-29888700-pzr6b" event={"ID":"bd72bc53-220d-40cd-abb8-36bc71b66094","Type":"ContainerStarted","Data":"61a1e40c3800360da75d3ce8fb241ae2dbf721ea6a79c9aac48865164fa192b7"}
Oct 30 01:00:04.791541 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:04.791510    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-cce513-29888700-744n2" event={"ID":"743ebbf6-b4db-4774-b836-de5d7c716ca7","Type":"ContainerStarted","Data":"053d4f03cdf6b600f44c88fe32abd8026a1297e0282c3db274c68b5ef54be262"}
Oct 30 01:00:04.793806 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:04.793773    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ca5363-29888700-trbnd" event={"ID":"5a907e40-190d-44fe-89b8-a51e621c04db","Type":"ContainerStarted","Data":"c1094ccf29ebd0350595ecfaf9d7a1a899e6251d81d823378cb77e2d278323ea"}
Oct 30 01:00:04.796061 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:04.796037    5002 generic.go:356] "Generic (PLEG): container finished" podID="d22fc624-e367-49a0-ba47-9a5dcabdde1f" containerID="045acf7668cdea11f4232a9ca237f4af38275fa1db7e352fd8e8177a43b750bf" exitCode=60
Oct 30 01:00:04.796169 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:04.796140    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8d2a70-29888700-jlmvv" event={"ID":"d22fc624-e367-49a0-ba47-9a5dcabdde1f","Type":"ContainerDied","Data":"045acf7668cdea11f4232a9ca237f4af38275fa1db7e352fd8e8177a43b750bf"}
Oct 30 01:00:04.803368 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:04.802430    5002 generic.go:356] "Generic (PLEG): container finished" podID="5fa9212f-5c4d-47a2-a855-2c18a807533e" containerID="cf86b1ce356cb7a39624de52160b766d4f9fd06a894479ffb91de86e4a7c60f8" exitCode=60
Oct 30 01:00:04.803368 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:04.802511    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8e0f66-29888700-tnj7q" event={"ID":"5fa9212f-5c4d-47a2-a855-2c18a807533e","Type":"ContainerDied","Data":"cf86b1ce356cb7a39624de52160b766d4f9fd06a894479ffb91de86e4a7c60f8"}
Oct 30 01:00:04.803368 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:04.802681    5002 scope.go:122] "RemoveContainer" containerID="045acf7668cdea11f4232a9ca237f4af38275fa1db7e352fd8e8177a43b750bf"
Oct 30 01:00:04.803368 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:04.802886    5002 scope.go:122] "RemoveContainer" containerID="cf86b1ce356cb7a39624de52160b766d4f9fd06a894479ffb91de86e4a7c60f8"
Oct 30 01:00:04.807698 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:04.806004    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1a11cf-29888700-d9s5c" event={"ID":"c111db1f-7e6f-44b8-b211-1658aee31337","Type":"ContainerStarted","Data":"3c6e2bf7dceb1d54d1c4c53a5d21c230103227b05226f2694da0915caa2b1116"}
Oct 30 01:00:04.809030 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:04.809009    5002 generic.go:356] "Generic (PLEG): container finished" podID="b15c8d59-8957-4e66-8e15-998c8a33e690" containerID="52c9d295b68ae5078aabcd76ac15895dbf420ca4acef8376a98de071d43c300d" exitCode=60
Oct 30 01:00:04.809179 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:04.809139    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-481ab7-29888700-8mt68" event={"ID":"b15c8d59-8957-4e66-8e15-998c8a33e690","Type":"ContainerDied","Data":"52c9d295b68ae5078aabcd76ac15895dbf420ca4acef8376a98de071d43c300d"}
Oct 30 01:00:04.809391 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:04.809358    5002 scope.go:122] "RemoveContainer" containerID="52c9d295b68ae5078aabcd76ac15895dbf420ca4acef8376a98de071d43c300d"
Oct 30 01:00:04.811191 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:04.811153    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-03d892/test-model-03d892-kserve-7f8c7cb699-kp2mb" event={"ID":"b3e31ee3-d2eb-42b1-84dc-173ed4434f84","Type":"ContainerStarted","Data":"15b3aa67bbfbaebf2826c2b7f5c0eed9397463d1f8b9a8e96d4995cc869f44b0"}
Oct 30 01:00:04.813187 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:04.813166    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5f387c-29888700-vtvtq" event={"ID":"4af343ba-bc4d-4ec3-92df-1534780d3089","Type":"ContainerStarted","Data":"0fa6fe9f2795f1f4f1abfda12e0a23cfa20e538b42c02af58243bdd14259a76b"}
Oct 30 01:00:04.817282 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:04.817257    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-11f521-29888700-khq6f" event={"ID":"7f443227-6ffc-40f4-80aa-1485ae41aa02","Type":"ContainerStarted","Data":"c109642b68e463948dbb8c97d17c2ec6a9366806f3954d92c10d82b11e38904d"}
Oct 30 01:00:04.819064 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:04.819012    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6f4fba-29888700-zvdkx" event={"ID":"6302c577-afc3-4632-a383-cb1fd8159992","Type":"ContainerStarted","Data":"8a2517195696db0e79173b39fd30320c2caf536242cba3a61c963ac076751684"}
Oct 30 01:00:04.821212 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:04.821189    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-bab19a-29888700-f7tvc" event={"ID":"b6c630bd-ac41-4607-8ed9-5f68cf00be38","Type":"ContainerStarted","Data":"a1aa53166218d68b5fd4a05db81ec3763f92ab61ff70fe9946f3d6b84e1849ba"}
Oct 30 01:00:04.825746 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:04.824016    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888700-k6pvc" event={"ID":"d5279cb1-29b0-494a-b3b3-527e8af641dc","Type":"ContainerStarted","Data":"90724187aa0bc3067c03416c5ab67aaab18e67fd9507493fe756975aed82135b"}
Oct 30 01:00:04.827024 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:04.826999    5002 generic.go:356] "Generic (PLEG): container finished" podID="09fb128c-0670-4260-a77b-ae32de23fa0d" containerID="edf3f54ca05163562bb6b26d29347c522a0aaf4b18301e8b82d4bb20e3748d59" exitCode=60
Oct 30 01:00:04.827129 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:04.827084    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-52650f-29888700-8mvn6" event={"ID":"09fb128c-0670-4260-a77b-ae32de23fa0d","Type":"ContainerDied","Data":"edf3f54ca05163562bb6b26d29347c522a0aaf4b18301e8b82d4bb20e3748d59"}
Oct 30 01:00:04.827357 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:04.827340    5002 scope.go:122] "RemoveContainer" containerID="edf3f54ca05163562bb6b26d29347c522a0aaf4b18301e8b82d4bb20e3748d59"
Oct 30 01:00:04.831475 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:04.831157    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-29be73-29888700-gvwp4" event={"ID":"86bc1f67-d9db-4d33-8bfc-e57494705d80","Type":"ContainerStarted","Data":"513a2274677b814499c24ff7a5631d4feee5d943b01a66222225276e3a899a31"}
Oct 30 01:00:04.847145 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:04.845883    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-5007e1-29888700-dvm8j" event={"ID":"38a379b2-37b6-4185-8319-2e62dcec38c8","Type":"ContainerStarted","Data":"d19ebf66055fd510c189028a5211caa9acc967ee4cce25d4fd2037d352ee5f6b"}
Oct 30 01:00:04.881647 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:04.881585    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b3bdf9-29888700-pzr6b" podStartSLOduration=4.881567316 podStartE2EDuration="4.881567316s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:00:04.876663928 +0000 UTC m=+1700.493096833" watchObservedRunningTime="2026-10-30 01:00:04.881567316 +0000 UTC m=+1700.498000197"
Oct 30 01:00:04.985431 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:04.985366    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888700-k6pvc" podStartSLOduration=4.985350761 podStartE2EDuration="4.985350761s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:00:04.981859457 +0000 UTC m=+1700.598292338" watchObservedRunningTime="2026-10-30 01:00:04.985350761 +0000 UTC m=+1700.601783640"
Oct 30 01:00:05.026955 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:05.026896    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-29be73-29888700-gvwp4" podStartSLOduration=5.026882389 podStartE2EDuration="5.026882389s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:00:05.02606496 +0000 UTC m=+1700.642497854" watchObservedRunningTime="2026-10-30 01:00:05.026882389 +0000 UTC m=+1700.643315268"
Oct 30 01:00:05.121724 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:05.121674    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5f387c-29888700-vtvtq" podStartSLOduration=5.121659097 podStartE2EDuration="5.121659097s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:00:05.121579413 +0000 UTC m=+1700.738012295" watchObservedRunningTime="2026-10-30 01:00:05.121659097 +0000 UTC m=+1700.738091975"
Oct 30 01:00:05.168320 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:05.168261    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-cce513-29888700-744n2" podStartSLOduration=5.168240991 podStartE2EDuration="5.168240991s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:00:05.167808778 +0000 UTC m=+1700.784241671" watchObservedRunningTime="2026-10-30 01:00:05.168240991 +0000 UTC m=+1700.784673872"
Oct 30 01:00:05.239836 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:05.237135    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-03d892/test-model-03d892-kserve-7f8c7cb699-kp2mb" podStartSLOduration=4.237116335 podStartE2EDuration="4.237116335s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:00:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:00:05.232645926 +0000 UTC m=+1700.849078828" watchObservedRunningTime="2026-10-30 01:00:05.237116335 +0000 UTC m=+1700.853549215"
Oct 30 01:00:05.326135 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:05.326012    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-5007e1-29888700-dvm8j" podStartSLOduration=5.325992993 podStartE2EDuration="5.325992993s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:00:05.322103179 +0000 UTC m=+1700.938536058" watchObservedRunningTime="2026-10-30 01:00:05.325992993 +0000 UTC m=+1700.942425872"
Oct 30 01:00:05.382406 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:05.377043    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-bab19a-29888700-f7tvc" podStartSLOduration=5.377022631 podStartE2EDuration="5.377022631s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:00:05.36623661 +0000 UTC m=+1700.982669491" watchObservedRunningTime="2026-10-30 01:00:05.377022631 +0000 UTC m=+1700.993455512"
Oct 30 01:00:05.447761 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:05.446674    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-11f521-29888700-khq6f" podStartSLOduration=5.446658286 podStartE2EDuration="5.446658286s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:00:05.442848254 +0000 UTC m=+1701.059281161" watchObservedRunningTime="2026-10-30 01:00:05.446658286 +0000 UTC m=+1701.063091166"
Oct 30 01:00:05.492606 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:05.492548    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6f4fba-29888700-zvdkx" podStartSLOduration=5.492533867 podStartE2EDuration="5.492533867s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:00:05.488225503 +0000 UTC m=+1701.104658404" watchObservedRunningTime="2026-10-30 01:00:05.492533867 +0000 UTC m=+1701.108966740"
Oct 30 01:00:05.508224 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:05.508188    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-03d892/test-model-03d892-kserve-7f8c7cb699-kp2mb"
Oct 30 01:00:05.532691 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:05.532628    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ca5363-29888700-trbnd" podStartSLOduration=5.532607797 podStartE2EDuration="5.532607797s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:00:05.530676227 +0000 UTC m=+1701.147109108" watchObservedRunningTime="2026-10-30 01:00:05.532607797 +0000 UTC m=+1701.149040678"
Oct 30 01:00:05.575124 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:05.575068    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1a11cf-29888700-d9s5c" podStartSLOduration=5.5750544269999995 podStartE2EDuration="5.575054427s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:00:05.572440349 +0000 UTC m=+1701.188873253" watchObservedRunningTime="2026-10-30 01:00:05.575054427 +0000 UTC m=+1701.191487319"
Oct 30 01:00:05.859337 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:05.859297    5002 generic.go:356] "Generic (PLEG): container finished" podID="4af343ba-bc4d-4ec3-92df-1534780d3089" containerID="0fa6fe9f2795f1f4f1abfda12e0a23cfa20e538b42c02af58243bdd14259a76b" exitCode=60
Oct 30 01:00:05.859836 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:05.859542    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5f387c-29888700-vtvtq" event={"ID":"4af343ba-bc4d-4ec3-92df-1534780d3089","Type":"ContainerDied","Data":"0fa6fe9f2795f1f4f1abfda12e0a23cfa20e538b42c02af58243bdd14259a76b"}
Oct 30 01:00:05.859836 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:05.859763    5002 scope.go:122] "RemoveContainer" containerID="0fa6fe9f2795f1f4f1abfda12e0a23cfa20e538b42c02af58243bdd14259a76b"
Oct 30 01:00:05.864896 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:05.864870    5002 generic.go:356] "Generic (PLEG): container finished" podID="7f443227-6ffc-40f4-80aa-1485ae41aa02" containerID="c109642b68e463948dbb8c97d17c2ec6a9366806f3954d92c10d82b11e38904d" exitCode=60
Oct 30 01:00:05.865113 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:05.865095    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-11f521-29888700-khq6f" event={"ID":"7f443227-6ffc-40f4-80aa-1485ae41aa02","Type":"ContainerDied","Data":"c109642b68e463948dbb8c97d17c2ec6a9366806f3954d92c10d82b11e38904d"}
Oct 30 01:00:05.868717 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:05.865289    5002 scope.go:122] "RemoveContainer" containerID="c109642b68e463948dbb8c97d17c2ec6a9366806f3954d92c10d82b11e38904d"
Oct 30 01:00:05.868717 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:05.867579    5002 generic.go:356] "Generic (PLEG): container finished" podID="6302c577-afc3-4632-a383-cb1fd8159992" containerID="8a2517195696db0e79173b39fd30320c2caf536242cba3a61c963ac076751684" exitCode=60
Oct 30 01:00:05.868717 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:05.867653    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6f4fba-29888700-zvdkx" event={"ID":"6302c577-afc3-4632-a383-cb1fd8159992","Type":"ContainerDied","Data":"8a2517195696db0e79173b39fd30320c2caf536242cba3a61c963ac076751684"}
Oct 30 01:00:05.868717 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:05.867985    5002 scope.go:122] "RemoveContainer" containerID="8a2517195696db0e79173b39fd30320c2caf536242cba3a61c963ac076751684"
Oct 30 01:00:05.870237 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:05.870170    5002 generic.go:356] "Generic (PLEG): container finished" podID="b6c630bd-ac41-4607-8ed9-5f68cf00be38" containerID="a1aa53166218d68b5fd4a05db81ec3763f92ab61ff70fe9946f3d6b84e1849ba" exitCode=60
Oct 30 01:00:05.870337 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:05.870271    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-bab19a-29888700-f7tvc" event={"ID":"b6c630bd-ac41-4607-8ed9-5f68cf00be38","Type":"ContainerDied","Data":"a1aa53166218d68b5fd4a05db81ec3763f92ab61ff70fe9946f3d6b84e1849ba"}
Oct 30 01:00:05.870512 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:05.870488    5002 scope.go:122] "RemoveContainer" containerID="a1aa53166218d68b5fd4a05db81ec3763f92ab61ff70fe9946f3d6b84e1849ba"
Oct 30 01:00:05.873043 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:05.873020    5002 generic.go:356] "Generic (PLEG): container finished" podID="d5279cb1-29b0-494a-b3b3-527e8af641dc" containerID="90724187aa0bc3067c03416c5ab67aaab18e67fd9507493fe756975aed82135b" exitCode=60
Oct 30 01:00:05.873129 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:05.873103    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888700-k6pvc" event={"ID":"d5279cb1-29b0-494a-b3b3-527e8af641dc","Type":"ContainerDied","Data":"90724187aa0bc3067c03416c5ab67aaab18e67fd9507493fe756975aed82135b"}
Oct 30 01:00:05.873638 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:05.873410    5002 scope.go:122] "RemoveContainer" containerID="90724187aa0bc3067c03416c5ab67aaab18e67fd9507493fe756975aed82135b"
Oct 30 01:00:05.878103 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:05.878080    5002 generic.go:356] "Generic (PLEG): container finished" podID="86bc1f67-d9db-4d33-8bfc-e57494705d80" containerID="513a2274677b814499c24ff7a5631d4feee5d943b01a66222225276e3a899a31" exitCode=60
Oct 30 01:00:05.878193 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:05.878177    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-29be73-29888700-gvwp4" event={"ID":"86bc1f67-d9db-4d33-8bfc-e57494705d80","Type":"ContainerDied","Data":"513a2274677b814499c24ff7a5631d4feee5d943b01a66222225276e3a899a31"}
Oct 30 01:00:05.878532 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:05.878515    5002 scope.go:122] "RemoveContainer" containerID="513a2274677b814499c24ff7a5631d4feee5d943b01a66222225276e3a899a31"
Oct 30 01:00:05.880784 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:05.880765    5002 generic.go:356] "Generic (PLEG): container finished" podID="38a379b2-37b6-4185-8319-2e62dcec38c8" containerID="d19ebf66055fd510c189028a5211caa9acc967ee4cce25d4fd2037d352ee5f6b" exitCode=60
Oct 30 01:00:05.881042 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:05.881022    5002 scope.go:122] "RemoveContainer" containerID="d19ebf66055fd510c189028a5211caa9acc967ee4cce25d4fd2037d352ee5f6b"
Oct 30 01:00:05.881148 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:05.881040    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-5007e1-29888700-dvm8j" event={"ID":"38a379b2-37b6-4185-8319-2e62dcec38c8","Type":"ContainerDied","Data":"d19ebf66055fd510c189028a5211caa9acc967ee4cce25d4fd2037d352ee5f6b"}
Oct 30 01:00:05.883248 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:05.883119    5002 generic.go:356] "Generic (PLEG): container finished" podID="bd72bc53-220d-40cd-abb8-36bc71b66094" containerID="61a1e40c3800360da75d3ce8fb241ae2dbf721ea6a79c9aac48865164fa192b7" exitCode=60
Oct 30 01:00:05.883248 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:05.883198    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b3bdf9-29888700-pzr6b" event={"ID":"bd72bc53-220d-40cd-abb8-36bc71b66094","Type":"ContainerDied","Data":"61a1e40c3800360da75d3ce8fb241ae2dbf721ea6a79c9aac48865164fa192b7"}
Oct 30 01:00:05.883749 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:05.883481    5002 scope.go:122] "RemoveContainer" containerID="61a1e40c3800360da75d3ce8fb241ae2dbf721ea6a79c9aac48865164fa192b7"
Oct 30 01:00:05.888433 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:05.887155    5002 generic.go:356] "Generic (PLEG): container finished" podID="743ebbf6-b4db-4774-b836-de5d7c716ca7" containerID="053d4f03cdf6b600f44c88fe32abd8026a1297e0282c3db274c68b5ef54be262" exitCode=60
Oct 30 01:00:05.888433 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:05.887233    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-cce513-29888700-744n2" event={"ID":"743ebbf6-b4db-4774-b836-de5d7c716ca7","Type":"ContainerDied","Data":"053d4f03cdf6b600f44c88fe32abd8026a1297e0282c3db274c68b5ef54be262"}
Oct 30 01:00:05.888433 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:05.887577    5002 scope.go:122] "RemoveContainer" containerID="053d4f03cdf6b600f44c88fe32abd8026a1297e0282c3db274c68b5ef54be262"
Oct 30 01:00:05.896583 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:05.895932    5002 generic.go:356] "Generic (PLEG): container finished" podID="5a907e40-190d-44fe-89b8-a51e621c04db" containerID="c1094ccf29ebd0350595ecfaf9d7a1a899e6251d81d823378cb77e2d278323ea" exitCode=60
Oct 30 01:00:05.896583 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:05.896016    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ca5363-29888700-trbnd" event={"ID":"5a907e40-190d-44fe-89b8-a51e621c04db","Type":"ContainerDied","Data":"c1094ccf29ebd0350595ecfaf9d7a1a899e6251d81d823378cb77e2d278323ea"}
Oct 30 01:00:05.896583 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:05.896331    5002 scope.go:122] "RemoveContainer" containerID="c1094ccf29ebd0350595ecfaf9d7a1a899e6251d81d823378cb77e2d278323ea"
Oct 30 01:00:05.914819 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:05.914788    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8d2a70-29888700-jlmvv" event={"ID":"d22fc624-e367-49a0-ba47-9a5dcabdde1f","Type":"ContainerStarted","Data":"54153540b9cbb314851c0da45b9629e291ef335f37fd4fa18b2d95a4c5dec048"}
Oct 30 01:00:05.918230 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:05.918204    5002 generic.go:356] "Generic (PLEG): container finished" podID="c111db1f-7e6f-44b8-b211-1658aee31337" containerID="3c6e2bf7dceb1d54d1c4c53a5d21c230103227b05226f2694da0915caa2b1116" exitCode=60
Oct 30 01:00:05.918350 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:05.918328    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1a11cf-29888700-d9s5c" event={"ID":"c111db1f-7e6f-44b8-b211-1658aee31337","Type":"ContainerDied","Data":"3c6e2bf7dceb1d54d1c4c53a5d21c230103227b05226f2694da0915caa2b1116"}
Oct 30 01:00:05.919235 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:05.919213    5002 scope.go:122] "RemoveContainer" containerID="3c6e2bf7dceb1d54d1c4c53a5d21c230103227b05226f2694da0915caa2b1116"
Oct 30 01:00:06.305854 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:06.305820    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-8dbe73/test-model-8dbe73-kserve-78dbd9d5cc-5zdxp"]
Oct 30 01:00:06.312224 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:06.312199    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-8dbe73/test-model-8dbe73-kserve-78dbd9d5cc-5zdxp"
Oct 30 01:00:06.336300 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:06.335550    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-8dbe73\"/\"test-model-8dbe73-kserve-self-signed-certs\""
Oct 30 01:00:06.336710 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:06.336681    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-8dbe73\"/\"default-dockercfg-shgjv\""
Oct 30 01:00:06.342761 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:06.342731    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-8dbe73/test-model-8dbe73-kserve-78dbd9d5cc-5zdxp"]
Oct 30 01:00:06.362119 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:06.362077    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-8dbe73\"/\"openshift-service-ca.crt\""
Oct 30 01:00:06.362301 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:06.362287    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-8dbe73\"/\"kube-root-ca.crt\""
Oct 30 01:00:06.414737 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:06.414683    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8d2a70-29888700-jlmvv" podStartSLOduration=6.414669189 podStartE2EDuration="6.414669189s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:00:06.411461 +0000 UTC m=+1702.027893863" watchObservedRunningTime="2026-10-30 01:00:06.414669189 +0000 UTC m=+1702.031102081"
Oct 30 01:00:06.441349 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:06.441307    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3cd6ac5-b9a0-4096-94ee-851e2042e9fa-tls-certs\") pod \"test-model-8dbe73-kserve-78dbd9d5cc-5zdxp\" (UID: \"f3cd6ac5-b9a0-4096-94ee-851e2042e9fa\") " pod="ai-tenant-e2e-inf-a-8dbe73/test-model-8dbe73-kserve-78dbd9d5cc-5zdxp"
Oct 30 01:00:06.441546 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:06.441394    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3cd6ac5-b9a0-4096-94ee-851e2042e9fa-home\") pod \"test-model-8dbe73-kserve-78dbd9d5cc-5zdxp\" (UID: \"f3cd6ac5-b9a0-4096-94ee-851e2042e9fa\") " pod="ai-tenant-e2e-inf-a-8dbe73/test-model-8dbe73-kserve-78dbd9d5cc-5zdxp"
Oct 30 01:00:06.441546 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:06.441519    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3cd6ac5-b9a0-4096-94ee-851e2042e9fa-dshm\") pod \"test-model-8dbe73-kserve-78dbd9d5cc-5zdxp\" (UID: \"f3cd6ac5-b9a0-4096-94ee-851e2042e9fa\") " pod="ai-tenant-e2e-inf-a-8dbe73/test-model-8dbe73-kserve-78dbd9d5cc-5zdxp"
Oct 30 01:00:06.441679 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:06.441582    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tctb\" (UniqueName: \"kubernetes.io/projected/f3cd6ac5-b9a0-4096-94ee-851e2042e9fa-kube-api-access-6tctb\") pod \"test-model-8dbe73-kserve-78dbd9d5cc-5zdxp\" (UID: \"f3cd6ac5-b9a0-4096-94ee-851e2042e9fa\") " pod="ai-tenant-e2e-inf-a-8dbe73/test-model-8dbe73-kserve-78dbd9d5cc-5zdxp"
Oct 30 01:00:06.441735 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:06.441687    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3cd6ac5-b9a0-4096-94ee-851e2042e9fa-tmp-dir\") pod \"test-model-8dbe73-kserve-78dbd9d5cc-5zdxp\" (UID: \"f3cd6ac5-b9a0-4096-94ee-851e2042e9fa\") " pod="ai-tenant-e2e-inf-a-8dbe73/test-model-8dbe73-kserve-78dbd9d5cc-5zdxp"
Oct 30 01:00:06.441788 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:06.441745    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3cd6ac5-b9a0-4096-94ee-851e2042e9fa-model-cache\") pod \"test-model-8dbe73-kserve-78dbd9d5cc-5zdxp\" (UID: \"f3cd6ac5-b9a0-4096-94ee-851e2042e9fa\") " pod="ai-tenant-e2e-inf-a-8dbe73/test-model-8dbe73-kserve-78dbd9d5cc-5zdxp"
Oct 30 01:00:06.544416 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:06.543875    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3cd6ac5-b9a0-4096-94ee-851e2042e9fa-home\") pod \"test-model-8dbe73-kserve-78dbd9d5cc-5zdxp\" (UID: \"f3cd6ac5-b9a0-4096-94ee-851e2042e9fa\") " pod="ai-tenant-e2e-inf-a-8dbe73/test-model-8dbe73-kserve-78dbd9d5cc-5zdxp"
Oct 30 01:00:06.564655 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:06.563538    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3cd6ac5-b9a0-4096-94ee-851e2042e9fa-home\") pod \"test-model-8dbe73-kserve-78dbd9d5cc-5zdxp\" (UID: \"f3cd6ac5-b9a0-4096-94ee-851e2042e9fa\") " pod="ai-tenant-e2e-inf-a-8dbe73/test-model-8dbe73-kserve-78dbd9d5cc-5zdxp"
Oct 30 01:00:06.564967 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:06.564946    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3cd6ac5-b9a0-4096-94ee-851e2042e9fa-dshm\") pod \"test-model-8dbe73-kserve-78dbd9d5cc-5zdxp\" (UID: \"f3cd6ac5-b9a0-4096-94ee-851e2042e9fa\") " pod="ai-tenant-e2e-inf-a-8dbe73/test-model-8dbe73-kserve-78dbd9d5cc-5zdxp"
Oct 30 01:00:06.583689 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:06.583643    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tctb\" (UniqueName: \"kubernetes.io/projected/f3cd6ac5-b9a0-4096-94ee-851e2042e9fa-kube-api-access-6tctb\") pod \"test-model-8dbe73-kserve-78dbd9d5cc-5zdxp\" (UID: \"f3cd6ac5-b9a0-4096-94ee-851e2042e9fa\") " pod="ai-tenant-e2e-inf-a-8dbe73/test-model-8dbe73-kserve-78dbd9d5cc-5zdxp"
Oct 30 01:00:06.584272 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:06.584246    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3cd6ac5-b9a0-4096-94ee-851e2042e9fa-tmp-dir\") pod \"test-model-8dbe73-kserve-78dbd9d5cc-5zdxp\" (UID: \"f3cd6ac5-b9a0-4096-94ee-851e2042e9fa\") " pod="ai-tenant-e2e-inf-a-8dbe73/test-model-8dbe73-kserve-78dbd9d5cc-5zdxp"
Oct 30 01:00:06.584966 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:06.584946    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3cd6ac5-b9a0-4096-94ee-851e2042e9fa-model-cache\") pod \"test-model-8dbe73-kserve-78dbd9d5cc-5zdxp\" (UID: \"f3cd6ac5-b9a0-4096-94ee-851e2042e9fa\") " pod="ai-tenant-e2e-inf-a-8dbe73/test-model-8dbe73-kserve-78dbd9d5cc-5zdxp"
Oct 30 01:00:06.607799 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:06.584857    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3cd6ac5-b9a0-4096-94ee-851e2042e9fa-tmp-dir\") pod \"test-model-8dbe73-kserve-78dbd9d5cc-5zdxp\" (UID: \"f3cd6ac5-b9a0-4096-94ee-851e2042e9fa\") " pod="ai-tenant-e2e-inf-a-8dbe73/test-model-8dbe73-kserve-78dbd9d5cc-5zdxp"
Oct 30 01:00:06.607799 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:06.585460    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3cd6ac5-b9a0-4096-94ee-851e2042e9fa-model-cache\") pod \"test-model-8dbe73-kserve-78dbd9d5cc-5zdxp\" (UID: \"f3cd6ac5-b9a0-4096-94ee-851e2042e9fa\") " pod="ai-tenant-e2e-inf-a-8dbe73/test-model-8dbe73-kserve-78dbd9d5cc-5zdxp"
Oct 30 01:00:06.623900 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:06.616671    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3cd6ac5-b9a0-4096-94ee-851e2042e9fa-tls-certs\") pod \"test-model-8dbe73-kserve-78dbd9d5cc-5zdxp\" (UID: \"f3cd6ac5-b9a0-4096-94ee-851e2042e9fa\") " pod="ai-tenant-e2e-inf-a-8dbe73/test-model-8dbe73-kserve-78dbd9d5cc-5zdxp"
Oct 30 01:00:06.766746 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:06.735260    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3cd6ac5-b9a0-4096-94ee-851e2042e9fa-dshm\") pod \"test-model-8dbe73-kserve-78dbd9d5cc-5zdxp\" (UID: \"f3cd6ac5-b9a0-4096-94ee-851e2042e9fa\") " pod="ai-tenant-e2e-inf-a-8dbe73/test-model-8dbe73-kserve-78dbd9d5cc-5zdxp"
Oct 30 01:00:06.766746 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:06.746152    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tctb\" (UniqueName: \"kubernetes.io/projected/f3cd6ac5-b9a0-4096-94ee-851e2042e9fa-kube-api-access-6tctb\") pod \"test-model-8dbe73-kserve-78dbd9d5cc-5zdxp\" (UID: \"f3cd6ac5-b9a0-4096-94ee-851e2042e9fa\") " pod="ai-tenant-e2e-inf-a-8dbe73/test-model-8dbe73-kserve-78dbd9d5cc-5zdxp"
Oct 30 01:00:06.770253 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:06.770222    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3cd6ac5-b9a0-4096-94ee-851e2042e9fa-tls-certs\") pod \"test-model-8dbe73-kserve-78dbd9d5cc-5zdxp\" (UID: \"f3cd6ac5-b9a0-4096-94ee-851e2042e9fa\") " pod="ai-tenant-e2e-inf-a-8dbe73/test-model-8dbe73-kserve-78dbd9d5cc-5zdxp"
Oct 30 01:00:06.934409 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:06.934363    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-5007e1-29888700-dvm8j" event={"ID":"38a379b2-37b6-4185-8319-2e62dcec38c8","Type":"ContainerStarted","Data":"1323635381a3d79b74e0afced5180f6f4f2bcc65686aed8d50d6bb29559460d0"}
Oct 30 01:00:06.938173 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:06.937658    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b3bdf9-29888700-pzr6b" event={"ID":"bd72bc53-220d-40cd-abb8-36bc71b66094","Type":"ContainerStarted","Data":"f42d54614481c6397b7e65761d0312be90c869f167d5624324fe55f0d754bfa1"}
Oct 30 01:00:06.941814 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:06.941788    5002 generic.go:356] "Generic (PLEG): container finished" podID="d22fc624-e367-49a0-ba47-9a5dcabdde1f" containerID="54153540b9cbb314851c0da45b9629e291ef335f37fd4fa18b2d95a4c5dec048" exitCode=60
Oct 30 01:00:06.941905 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:06.941893    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8d2a70-29888700-jlmvv" event={"ID":"d22fc624-e367-49a0-ba47-9a5dcabdde1f","Type":"ContainerDied","Data":"54153540b9cbb314851c0da45b9629e291ef335f37fd4fa18b2d95a4c5dec048"}
Oct 30 01:00:06.941970 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:06.941946    5002 scope.go:122] "RemoveContainer" containerID="045acf7668cdea11f4232a9ca237f4af38275fa1db7e352fd8e8177a43b750bf"
Oct 30 01:00:06.942091 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:06.942069    5002 scope.go:122] "RemoveContainer" containerID="54153540b9cbb314851c0da45b9629e291ef335f37fd4fa18b2d95a4c5dec048"
Oct 30 01:00:06.942354 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:00:06.942327    5002 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-8d2a70-29888700-jlmvv_odh-ai-gateway-infra(d22fc624-e367-49a0-ba47-9a5dcabdde1f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8d2a70-29888700-jlmvv" podUID="d22fc624-e367-49a0-ba47-9a5dcabdde1f"
Oct 30 01:00:06.942527 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:06.942509    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-8dbe73/test-model-8dbe73-kserve-78dbd9d5cc-5zdxp"
Oct 30 01:00:06.944310 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:06.944285    5002 generic.go:356] "Generic (PLEG): container finished" podID="5fa9212f-5c4d-47a2-a855-2c18a807533e" containerID="cdc6b212bb47ace68ace90e7eaed307569eb5a305ec64e50bafab9ca50204fbe" exitCode=60
Oct 30 01:00:06.944409 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:06.944391    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8e0f66-29888700-tnj7q" event={"ID":"5fa9212f-5c4d-47a2-a855-2c18a807533e","Type":"ContainerDied","Data":"cdc6b212bb47ace68ace90e7eaed307569eb5a305ec64e50bafab9ca50204fbe"}
Oct 30 01:00:06.944648 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:06.944630    5002 scope.go:122] "RemoveContainer" containerID="cdc6b212bb47ace68ace90e7eaed307569eb5a305ec64e50bafab9ca50204fbe"
Oct 30 01:00:06.944909 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:00:06.944885    5002 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-8e0f66-29888700-tnj7q_odh-ai-gateway-infra(5fa9212f-5c4d-47a2-a855-2c18a807533e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8e0f66-29888700-tnj7q" podUID="5fa9212f-5c4d-47a2-a855-2c18a807533e"
Oct 30 01:00:06.948817 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:06.948792    5002 generic.go:356] "Generic (PLEG): container finished" podID="b15c8d59-8957-4e66-8e15-998c8a33e690" containerID="3d3fdb0735f79018c13a82512cd7420f9099a1138403ae93a519dcf547e8deee" exitCode=60
Oct 30 01:00:06.948997 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:06.948940    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-481ab7-29888700-8mt68" event={"ID":"b15c8d59-8957-4e66-8e15-998c8a33e690","Type":"ContainerDied","Data":"3d3fdb0735f79018c13a82512cd7420f9099a1138403ae93a519dcf547e8deee"}
Oct 30 01:00:06.949753 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:06.949733    5002 scope.go:122] "RemoveContainer" containerID="3d3fdb0735f79018c13a82512cd7420f9099a1138403ae93a519dcf547e8deee"
Oct 30 01:00:06.950014 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:00:06.949994    5002 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-481ab7-29888700-8mt68_odh-ai-gateway-infra(b15c8d59-8957-4e66-8e15-998c8a33e690)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-481ab7-29888700-8mt68" podUID="b15c8d59-8957-4e66-8e15-998c8a33e690"
Oct 30 01:00:06.966516 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:06.966345    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5f387c-29888700-vtvtq" event={"ID":"4af343ba-bc4d-4ec3-92df-1534780d3089","Type":"ContainerStarted","Data":"70fa31a4dae40eb87d43802b774fae56d2a381fe759b25005c5751be178b8056"}
Oct 30 01:00:06.970178 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:06.969476    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-11f521-29888700-khq6f" event={"ID":"7f443227-6ffc-40f4-80aa-1485ae41aa02","Type":"ContainerStarted","Data":"1f4025cfa5689b2203a7a9e9de122debe44b072e0f2563a0f56343b9f15bb934"}
Oct 30 01:00:06.973467 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:06.972978    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6f4fba-29888700-zvdkx" event={"ID":"6302c577-afc3-4632-a383-cb1fd8159992","Type":"ContainerStarted","Data":"0643d42215bf6a5d84c1269e24289345566d810fed3f4f2079a63d302be67676"}
Oct 30 01:00:06.975966 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:06.975945    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-bab19a-29888700-f7tvc" event={"ID":"b6c630bd-ac41-4607-8ed9-5f68cf00be38","Type":"ContainerStarted","Data":"982c36549726383392d577034ce61f2700a7b1e1d242506a82e3064d1b865036"}
Oct 30 01:00:06.978828 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:06.978804    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888700-k6pvc" event={"ID":"d5279cb1-29b0-494a-b3b3-527e8af641dc","Type":"ContainerStarted","Data":"cf5baf6bf97a9f4e9b95b5efdd8d7180f6328f1e964d0b4cae74b92af25daaf7"}
Oct 30 01:00:06.981667 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:06.981638    5002 generic.go:356] "Generic (PLEG): container finished" podID="09fb128c-0670-4260-a77b-ae32de23fa0d" containerID="b09f8bdbe09a4f05a4237a051cf4304c30df663110fceab326eea8510faad135" exitCode=60
Oct 30 01:00:06.981754 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:06.981723    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-52650f-29888700-8mvn6" event={"ID":"09fb128c-0670-4260-a77b-ae32de23fa0d","Type":"ContainerDied","Data":"b09f8bdbe09a4f05a4237a051cf4304c30df663110fceab326eea8510faad135"}
Oct 30 01:00:06.982064 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:06.982046    5002 scope.go:122] "RemoveContainer" containerID="b09f8bdbe09a4f05a4237a051cf4304c30df663110fceab326eea8510faad135"
Oct 30 01:00:06.982321 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:00:06.982299    5002 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-52650f-29888700-8mvn6_odh-ai-gateway-infra(09fb128c-0670-4260-a77b-ae32de23fa0d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-52650f-29888700-8mvn6" podUID="09fb128c-0670-4260-a77b-ae32de23fa0d"
Oct 30 01:00:07.191995 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:07.191966    5002 scope.go:122] "RemoveContainer" containerID="cf86b1ce356cb7a39624de52160b766d4f9fd06a894479ffb91de86e4a7c60f8"
Oct 30 01:00:07.229444 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:07.229412    5002 scope.go:122] "RemoveContainer" containerID="52c9d295b68ae5078aabcd76ac15895dbf420ca4acef8376a98de071d43c300d"
Oct 30 01:00:07.256085 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:07.255012    5002 scope.go:122] "RemoveContainer" containerID="edf3f54ca05163562bb6b26d29347c522a0aaf4b18301e8b82d4bb20e3748d59"
Oct 30 01:00:07.306461 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:07.305345    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-8dbe73/test-model-8dbe73-kserve-78dbd9d5cc-5zdxp"]
Oct 30 01:00:07.401122 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:07.401086    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-t4s54"]
Oct 30 01:00:07.401580 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:07.401547    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-t4s54" podUID="3a107188-58d7-4c62-a2e0-e81eb181d540" containerName="istio-proxy" containerID="cri-o://cf7d7f440c1f4124242028ef09533f0122a4dc6177bbc75d06fc326e5c288fe1" gracePeriod=30
Oct 30 01:00:07.999010 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:07.998977    5002 generic.go:356] "Generic (PLEG): container finished" podID="86bc1f67-d9db-4d33-8bfc-e57494705d80" containerID="1c5c4fc069b26c0b4d51bf5a0fbd3c16dfa0e1f360ca6195142d58981827b7b6" exitCode=60
Oct 30 01:00:07.999609 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:07.999092    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-29be73-29888700-gvwp4" event={"ID":"86bc1f67-d9db-4d33-8bfc-e57494705d80","Type":"ContainerDied","Data":"1c5c4fc069b26c0b4d51bf5a0fbd3c16dfa0e1f360ca6195142d58981827b7b6"}
Oct 30 01:00:07.999609 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:07.999197    5002 scope.go:122] "RemoveContainer" containerID="513a2274677b814499c24ff7a5631d4feee5d943b01a66222225276e3a899a31"
Oct 30 01:00:08.000450 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:07.999772    5002 scope.go:122] "RemoveContainer" containerID="1c5c4fc069b26c0b4d51bf5a0fbd3c16dfa0e1f360ca6195142d58981827b7b6"
Oct 30 01:00:08.000450 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:00:08.000184    5002 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-29be73-29888700-gvwp4_odh-ai-gateway-infra(86bc1f67-d9db-4d33-8bfc-e57494705d80)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-29be73-29888700-gvwp4" podUID="86bc1f67-d9db-4d33-8bfc-e57494705d80"
Oct 30 01:00:08.003084 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:08.003050    5002 generic.go:356] "Generic (PLEG): container finished" podID="38a379b2-37b6-4185-8319-2e62dcec38c8" containerID="1323635381a3d79b74e0afced5180f6f4f2bcc65686aed8d50d6bb29559460d0" exitCode=60
Oct 30 01:00:08.003192 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:08.003103    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-5007e1-29888700-dvm8j" event={"ID":"38a379b2-37b6-4185-8319-2e62dcec38c8","Type":"ContainerDied","Data":"1323635381a3d79b74e0afced5180f6f4f2bcc65686aed8d50d6bb29559460d0"}
Oct 30 01:00:08.003495 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:08.003478    5002 scope.go:122] "RemoveContainer" containerID="1323635381a3d79b74e0afced5180f6f4f2bcc65686aed8d50d6bb29559460d0"
Oct 30 01:00:08.003944 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:00:08.003910    5002 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-5007e1-29888700-dvm8j_odh-ai-gateway-infra(38a379b2-37b6-4185-8319-2e62dcec38c8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-5007e1-29888700-dvm8j" podUID="38a379b2-37b6-4185-8319-2e62dcec38c8"
Oct 30 01:00:08.004769 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:08.004743    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-8dbe73/test-model-8dbe73-kserve-78dbd9d5cc-5zdxp" event={"ID":"f3cd6ac5-b9a0-4096-94ee-851e2042e9fa","Type":"ContainerStarted","Data":"7ae2474e8693bc900dc0769ef918dee3c9da684ac70c652c9571481b8c014215"}
Oct 30 01:00:08.006848 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:08.006824    5002 generic.go:356] "Generic (PLEG): container finished" podID="bd72bc53-220d-40cd-abb8-36bc71b66094" containerID="f42d54614481c6397b7e65761d0312be90c869f167d5624324fe55f0d754bfa1" exitCode=60
Oct 30 01:00:08.006955 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:08.006895    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b3bdf9-29888700-pzr6b" event={"ID":"bd72bc53-220d-40cd-abb8-36bc71b66094","Type":"ContainerDied","Data":"f42d54614481c6397b7e65761d0312be90c869f167d5624324fe55f0d754bfa1"}
Oct 30 01:00:08.007249 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:08.007220    5002 scope.go:122] "RemoveContainer" containerID="f42d54614481c6397b7e65761d0312be90c869f167d5624324fe55f0d754bfa1"
Oct 30 01:00:08.007619 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:00:08.007596    5002 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-b3bdf9-29888700-pzr6b_odh-ai-gateway-infra(bd72bc53-220d-40cd-abb8-36bc71b66094)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b3bdf9-29888700-pzr6b" podUID="bd72bc53-220d-40cd-abb8-36bc71b66094"
Oct 30 01:00:08.009063 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:08.009037    5002 generic.go:356] "Generic (PLEG): container finished" podID="743ebbf6-b4db-4774-b836-de5d7c716ca7" containerID="a4572a0daa660fcdf452acd07ef154e37825a339040563b851f9b439c994c008" exitCode=60
Oct 30 01:00:08.009151 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:08.009080    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-cce513-29888700-744n2" event={"ID":"743ebbf6-b4db-4774-b836-de5d7c716ca7","Type":"ContainerDied","Data":"a4572a0daa660fcdf452acd07ef154e37825a339040563b851f9b439c994c008"}
Oct 30 01:00:08.009506 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:08.009475    5002 scope.go:122] "RemoveContainer" containerID="a4572a0daa660fcdf452acd07ef154e37825a339040563b851f9b439c994c008"
Oct 30 01:00:08.009827 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:00:08.009791    5002 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-cce513-29888700-744n2_odh-ai-gateway-infra(743ebbf6-b4db-4774-b836-de5d7c716ca7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-cce513-29888700-744n2" podUID="743ebbf6-b4db-4774-b836-de5d7c716ca7"
Oct 30 01:00:08.011745 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:08.011152    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ca5363-29888700-trbnd" event={"ID":"5a907e40-190d-44fe-89b8-a51e621c04db","Type":"ContainerStarted","Data":"386fa1d5a2c4d3a43aa56a4e7887c0c518d65a7993cf95921b6bc0a401d6e2f8"}
Oct 30 01:00:08.013253 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:08.013233    5002 scope.go:122] "RemoveContainer" containerID="54153540b9cbb314851c0da45b9629e291ef335f37fd4fa18b2d95a4c5dec048"
Oct 30 01:00:08.013570 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:00:08.013544    5002 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-8d2a70-29888700-jlmvv_odh-ai-gateway-infra(d22fc624-e367-49a0-ba47-9a5dcabdde1f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8d2a70-29888700-jlmvv" podUID="d22fc624-e367-49a0-ba47-9a5dcabdde1f"
Oct 30 01:00:08.015640 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:08.015100    5002 scope.go:122] "RemoveContainer" containerID="cdc6b212bb47ace68ace90e7eaed307569eb5a305ec64e50bafab9ca50204fbe"
Oct 30 01:00:08.015640 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:00:08.015419    5002 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-8e0f66-29888700-tnj7q_odh-ai-gateway-infra(5fa9212f-5c4d-47a2-a855-2c18a807533e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8e0f66-29888700-tnj7q" podUID="5fa9212f-5c4d-47a2-a855-2c18a807533e"
Oct 30 01:00:08.021300 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:08.021277    5002 generic.go:356] "Generic (PLEG): container finished" podID="c111db1f-7e6f-44b8-b211-1658aee31337" containerID="441e5c13bff6f0f1fb7b82264ceaccf2a98dc69609e4fb64193662b9fcda5160" exitCode=60
Oct 30 01:00:08.021478 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:08.021355    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1a11cf-29888700-d9s5c" event={"ID":"c111db1f-7e6f-44b8-b211-1658aee31337","Type":"ContainerDied","Data":"441e5c13bff6f0f1fb7b82264ceaccf2a98dc69609e4fb64193662b9fcda5160"}
Oct 30 01:00:08.021766 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:08.021749    5002 scope.go:122] "RemoveContainer" containerID="441e5c13bff6f0f1fb7b82264ceaccf2a98dc69609e4fb64193662b9fcda5160"
Oct 30 01:00:08.022029 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:00:08.022010    5002 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-1a11cf-29888700-d9s5c_odh-ai-gateway-infra(c111db1f-7e6f-44b8-b211-1658aee31337)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1a11cf-29888700-d9s5c" podUID="c111db1f-7e6f-44b8-b211-1658aee31337"
Oct 30 01:00:08.024787 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:08.024768    5002 scope.go:122] "RemoveContainer" containerID="d19ebf66055fd510c189028a5211caa9acc967ee4cce25d4fd2037d352ee5f6b"
Oct 30 01:00:08.025803 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:08.025501    5002 scope.go:122] "RemoveContainer" containerID="3d3fdb0735f79018c13a82512cd7420f9099a1138403ae93a519dcf547e8deee"
Oct 30 01:00:08.025803 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:00:08.025764    5002 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-481ab7-29888700-8mt68_odh-ai-gateway-infra(b15c8d59-8957-4e66-8e15-998c8a33e690)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-481ab7-29888700-8mt68" podUID="b15c8d59-8957-4e66-8e15-998c8a33e690"
Oct 30 01:00:08.027564 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:08.027536    5002 generic.go:356] "Generic (PLEG): container finished" podID="4af343ba-bc4d-4ec3-92df-1534780d3089" containerID="70fa31a4dae40eb87d43802b774fae56d2a381fe759b25005c5751be178b8056" exitCode=60
Oct 30 01:00:08.027661 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:08.027645    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5f387c-29888700-vtvtq" event={"ID":"4af343ba-bc4d-4ec3-92df-1534780d3089","Type":"ContainerDied","Data":"70fa31a4dae40eb87d43802b774fae56d2a381fe759b25005c5751be178b8056"}
Oct 30 01:00:08.027816 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:08.027789    5002 scope.go:122] "RemoveContainer" containerID="70fa31a4dae40eb87d43802b774fae56d2a381fe759b25005c5751be178b8056"
Oct 30 01:00:08.028084 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:00:08.028065    5002 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-5f387c-29888700-vtvtq_odh-ai-gateway-infra(4af343ba-bc4d-4ec3-92df-1534780d3089)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5f387c-29888700-vtvtq" podUID="4af343ba-bc4d-4ec3-92df-1534780d3089"
Oct 30 01:00:08.029539 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:08.029464    5002 generic.go:356] "Generic (PLEG): container finished" podID="7f443227-6ffc-40f4-80aa-1485ae41aa02" containerID="1f4025cfa5689b2203a7a9e9de122debe44b072e0f2563a0f56343b9f15bb934" exitCode=60
Oct 30 01:00:08.029622 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:08.029540    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-11f521-29888700-khq6f" event={"ID":"7f443227-6ffc-40f4-80aa-1485ae41aa02","Type":"ContainerDied","Data":"1f4025cfa5689b2203a7a9e9de122debe44b072e0f2563a0f56343b9f15bb934"}
Oct 30 01:00:08.029717 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:08.029702    5002 scope.go:122] "RemoveContainer" containerID="1f4025cfa5689b2203a7a9e9de122debe44b072e0f2563a0f56343b9f15bb934"
Oct 30 01:00:08.029925 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:00:08.029908    5002 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-11f521-29888700-khq6f_odh-ai-gateway-infra(7f443227-6ffc-40f4-80aa-1485ae41aa02)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-11f521-29888700-khq6f" podUID="7f443227-6ffc-40f4-80aa-1485ae41aa02"
Oct 30 01:00:08.031813 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:08.031787    5002 generic.go:356] "Generic (PLEG): container finished" podID="6302c577-afc3-4632-a383-cb1fd8159992" containerID="0643d42215bf6a5d84c1269e24289345566d810fed3f4f2079a63d302be67676" exitCode=60
Oct 30 01:00:08.031895 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:08.031863    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6f4fba-29888700-zvdkx" event={"ID":"6302c577-afc3-4632-a383-cb1fd8159992","Type":"ContainerDied","Data":"0643d42215bf6a5d84c1269e24289345566d810fed3f4f2079a63d302be67676"}
Oct 30 01:00:08.032174 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:08.032142    5002 scope.go:122] "RemoveContainer" containerID="0643d42215bf6a5d84c1269e24289345566d810fed3f4f2079a63d302be67676"
Oct 30 01:00:08.032436 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:00:08.032415    5002 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-6f4fba-29888700-zvdkx_odh-ai-gateway-infra(6302c577-afc3-4632-a383-cb1fd8159992)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6f4fba-29888700-zvdkx" podUID="6302c577-afc3-4632-a383-cb1fd8159992"
Oct 30 01:00:08.036069 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:08.034859    5002 generic.go:356] "Generic (PLEG): container finished" podID="b6c630bd-ac41-4607-8ed9-5f68cf00be38" containerID="982c36549726383392d577034ce61f2700a7b1e1d242506a82e3064d1b865036" exitCode=60
Oct 30 01:00:08.036069 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:08.034945    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-bab19a-29888700-f7tvc" event={"ID":"b6c630bd-ac41-4607-8ed9-5f68cf00be38","Type":"ContainerDied","Data":"982c36549726383392d577034ce61f2700a7b1e1d242506a82e3064d1b865036"}
Oct 30 01:00:08.036069 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:08.035231    5002 scope.go:122] "RemoveContainer" containerID="982c36549726383392d577034ce61f2700a7b1e1d242506a82e3064d1b865036"
Oct 30 01:00:08.036069 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:00:08.035560    5002 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-bab19a-29888700-f7tvc_odh-ai-gateway-infra(b6c630bd-ac41-4607-8ed9-5f68cf00be38)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-bab19a-29888700-f7tvc" podUID="b6c630bd-ac41-4607-8ed9-5f68cf00be38"
Oct 30 01:00:08.037524 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:08.037506    5002 generic.go:356] "Generic (PLEG): container finished" podID="d5279cb1-29b0-494a-b3b3-527e8af641dc" containerID="cf5baf6bf97a9f4e9b95b5efdd8d7180f6328f1e964d0b4cae74b92af25daaf7" exitCode=60
Oct 30 01:00:08.037585 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:08.037563    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888700-k6pvc" event={"ID":"d5279cb1-29b0-494a-b3b3-527e8af641dc","Type":"ContainerDied","Data":"cf5baf6bf97a9f4e9b95b5efdd8d7180f6328f1e964d0b4cae74b92af25daaf7"}
Oct 30 01:00:08.037903 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:08.037884    5002 scope.go:122] "RemoveContainer" containerID="cf5baf6bf97a9f4e9b95b5efdd8d7180f6328f1e964d0b4cae74b92af25daaf7"
Oct 30 01:00:08.038189 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:00:08.038166    5002 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29888700-k6pvc_odh-ai-gateway-infra(d5279cb1-29b0-494a-b3b3-527e8af641dc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888700-k6pvc" podUID="d5279cb1-29b0-494a-b3b3-527e8af641dc"
Oct 30 01:00:08.039391 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:08.039364    5002 scope.go:122] "RemoveContainer" containerID="b09f8bdbe09a4f05a4237a051cf4304c30df663110fceab326eea8510faad135"
Oct 30 01:00:08.039611 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:00:08.039594    5002 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-52650f-29888700-8mvn6_odh-ai-gateway-infra(09fb128c-0670-4260-a77b-ae32de23fa0d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-52650f-29888700-8mvn6" podUID="09fb128c-0670-4260-a77b-ae32de23fa0d"
Oct 30 01:00:08.075411 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:08.075363    5002 scope.go:122] "RemoveContainer" containerID="61a1e40c3800360da75d3ce8fb241ae2dbf721ea6a79c9aac48865164fa192b7"
Oct 30 01:00:08.125562 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:08.125534    5002 scope.go:122] "RemoveContainer" containerID="053d4f03cdf6b600f44c88fe32abd8026a1297e0282c3db274c68b5ef54be262"
Oct 30 01:00:08.142973 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:08.142947    5002 scope.go:122] "RemoveContainer" containerID="3c6e2bf7dceb1d54d1c4c53a5d21c230103227b05226f2694da0915caa2b1116"
Oct 30 01:00:08.161161 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:08.161137    5002 scope.go:122] "RemoveContainer" containerID="0fa6fe9f2795f1f4f1abfda12e0a23cfa20e538b42c02af58243bdd14259a76b"
Oct 30 01:00:08.202561 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:08.202531    5002 scope.go:122] "RemoveContainer" containerID="c109642b68e463948dbb8c97d17c2ec6a9366806f3954d92c10d82b11e38904d"
Oct 30 01:00:08.204892 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:08.204824    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-84b89f/test-model-84b89f-kserve-d8c475485-5prlt"]
Oct 30 01:00:08.222069 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:08.222037    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-84b89f/test-model-84b89f-kserve-d8c475485-5prlt"
Oct 30 01:00:08.223632 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:08.223185    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-84b89f/test-model-84b89f-kserve-d8c475485-5prlt"]
Oct 30 01:00:08.240711 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:08.240682    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-84b89f\"/\"openshift-service-ca.crt\""
Oct 30 01:00:08.240904 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:08.240862    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-84b89f\"/\"default-dockercfg-rd4m2\""
Oct 30 01:00:08.241017 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:08.240991    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-84b89f\"/\"kube-root-ca.crt\""
Oct 30 01:00:08.241092 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:08.241063    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-84b89f\"/\"test-model-84b89f-kserve-self-signed-certs\""
Oct 30 01:00:08.242114 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:08.241994    5002 scope.go:122] "RemoveContainer" containerID="8a2517195696db0e79173b39fd30320c2caf536242cba3a61c963ac076751684"
Oct 30 01:00:08.262797 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:08.262423    5002 scope.go:122] "RemoveContainer" containerID="a1aa53166218d68b5fd4a05db81ec3763f92ab61ff70fe9946f3d6b84e1849ba"
Oct 30 01:00:08.296439 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:08.294162    5002 scope.go:122] "RemoveContainer" containerID="90724187aa0bc3067c03416c5ab67aaab18e67fd9507493fe756975aed82135b"
Oct 30 01:00:08.296439 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:08.295330    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/00a3e39a-01f0-4d98-9350-aa0b17a6c434-home\") pod \"test-model-84b89f-kserve-d8c475485-5prlt\" (UID: \"00a3e39a-01f0-4d98-9350-aa0b17a6c434\") " pod="ai-tenant-e2e-inf-b-84b89f/test-model-84b89f-kserve-d8c475485-5prlt"
Oct 30 01:00:08.296439 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:08.295395    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00a3e39a-01f0-4d98-9350-aa0b17a6c434-model-cache\") pod \"test-model-84b89f-kserve-d8c475485-5prlt\" (UID: \"00a3e39a-01f0-4d98-9350-aa0b17a6c434\") " pod="ai-tenant-e2e-inf-b-84b89f/test-model-84b89f-kserve-d8c475485-5prlt"
Oct 30 01:00:08.296439 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:08.295547    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/00a3e39a-01f0-4d98-9350-aa0b17a6c434-dshm\") pod \"test-model-84b89f-kserve-d8c475485-5prlt\" (UID: \"00a3e39a-01f0-4d98-9350-aa0b17a6c434\") " pod="ai-tenant-e2e-inf-b-84b89f/test-model-84b89f-kserve-d8c475485-5prlt"
Oct 30 01:00:08.296439 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:08.295630    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00a3e39a-01f0-4d98-9350-aa0b17a6c434-tls-certs\") pod \"test-model-84b89f-kserve-d8c475485-5prlt\" (UID: \"00a3e39a-01f0-4d98-9350-aa0b17a6c434\") " pod="ai-tenant-e2e-inf-b-84b89f/test-model-84b89f-kserve-d8c475485-5prlt"
Oct 30 01:00:08.296439 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:08.295670    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00a3e39a-01f0-4d98-9350-aa0b17a6c434-tmp-dir\") pod \"test-model-84b89f-kserve-d8c475485-5prlt\" (UID: \"00a3e39a-01f0-4d98-9350-aa0b17a6c434\") " pod="ai-tenant-e2e-inf-b-84b89f/test-model-84b89f-kserve-d8c475485-5prlt"
Oct 30 01:00:08.296439 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:08.295710    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5n62\" (UniqueName: \"kubernetes.io/projected/00a3e39a-01f0-4d98-9350-aa0b17a6c434-kube-api-access-k5n62\") pod \"test-model-84b89f-kserve-d8c475485-5prlt\" (UID: \"00a3e39a-01f0-4d98-9350-aa0b17a6c434\") " pod="ai-tenant-e2e-inf-b-84b89f/test-model-84b89f-kserve-d8c475485-5prlt"
Oct 30 01:00:08.396595 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:08.396564    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00a3e39a-01f0-4d98-9350-aa0b17a6c434-model-cache\") pod \"test-model-84b89f-kserve-d8c475485-5prlt\" (UID: \"00a3e39a-01f0-4d98-9350-aa0b17a6c434\") " pod="ai-tenant-e2e-inf-b-84b89f/test-model-84b89f-kserve-d8c475485-5prlt"
Oct 30 01:00:08.397038 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:08.397013    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00a3e39a-01f0-4d98-9350-aa0b17a6c434-model-cache\") pod \"test-model-84b89f-kserve-d8c475485-5prlt\" (UID: \"00a3e39a-01f0-4d98-9350-aa0b17a6c434\") " pod="ai-tenant-e2e-inf-b-84b89f/test-model-84b89f-kserve-d8c475485-5prlt"
Oct 30 01:00:08.397128 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:08.397011    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/00a3e39a-01f0-4d98-9350-aa0b17a6c434-dshm\") pod \"test-model-84b89f-kserve-d8c475485-5prlt\" (UID: \"00a3e39a-01f0-4d98-9350-aa0b17a6c434\") " pod="ai-tenant-e2e-inf-b-84b89f/test-model-84b89f-kserve-d8c475485-5prlt"
Oct 30 01:00:08.397199 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:08.397182    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00a3e39a-01f0-4d98-9350-aa0b17a6c434-tls-certs\") pod \"test-model-84b89f-kserve-d8c475485-5prlt\" (UID: \"00a3e39a-01f0-4d98-9350-aa0b17a6c434\") " pod="ai-tenant-e2e-inf-b-84b89f/test-model-84b89f-kserve-d8c475485-5prlt"
Oct 30 01:00:08.397256 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:08.397229    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00a3e39a-01f0-4d98-9350-aa0b17a6c434-tmp-dir\") pod \"test-model-84b89f-kserve-d8c475485-5prlt\" (UID: \"00a3e39a-01f0-4d98-9350-aa0b17a6c434\") " pod="ai-tenant-e2e-inf-b-84b89f/test-model-84b89f-kserve-d8c475485-5prlt"
Oct 30 01:00:08.397307 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:08.397284    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5n62\" (UniqueName: \"kubernetes.io/projected/00a3e39a-01f0-4d98-9350-aa0b17a6c434-kube-api-access-k5n62\") pod \"test-model-84b89f-kserve-d8c475485-5prlt\" (UID: \"00a3e39a-01f0-4d98-9350-aa0b17a6c434\") " pod="ai-tenant-e2e-inf-b-84b89f/test-model-84b89f-kserve-d8c475485-5prlt"
Oct 30 01:00:08.397364 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:08.397320    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/00a3e39a-01f0-4d98-9350-aa0b17a6c434-home\") pod \"test-model-84b89f-kserve-d8c475485-5prlt\" (UID: \"00a3e39a-01f0-4d98-9350-aa0b17a6c434\") " pod="ai-tenant-e2e-inf-b-84b89f/test-model-84b89f-kserve-d8c475485-5prlt"
Oct 30 01:00:08.397816 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:08.397787    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/00a3e39a-01f0-4d98-9350-aa0b17a6c434-home\") pod \"test-model-84b89f-kserve-d8c475485-5prlt\" (UID: \"00a3e39a-01f0-4d98-9350-aa0b17a6c434\") " pod="ai-tenant-e2e-inf-b-84b89f/test-model-84b89f-kserve-d8c475485-5prlt"
Oct 30 01:00:08.398323 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:08.398052    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00a3e39a-01f0-4d98-9350-aa0b17a6c434-tmp-dir\") pod \"test-model-84b89f-kserve-d8c475485-5prlt\" (UID: \"00a3e39a-01f0-4d98-9350-aa0b17a6c434\") " pod="ai-tenant-e2e-inf-b-84b89f/test-model-84b89f-kserve-d8c475485-5prlt"
Oct 30 01:00:08.399887 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:08.399866    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/00a3e39a-01f0-4d98-9350-aa0b17a6c434-dshm\") pod \"test-model-84b89f-kserve-d8c475485-5prlt\" (UID: \"00a3e39a-01f0-4d98-9350-aa0b17a6c434\") " pod="ai-tenant-e2e-inf-b-84b89f/test-model-84b89f-kserve-d8c475485-5prlt"
Oct 30 01:00:08.405073 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:08.405049    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00a3e39a-01f0-4d98-9350-aa0b17a6c434-tls-certs\") pod \"test-model-84b89f-kserve-d8c475485-5prlt\" (UID: \"00a3e39a-01f0-4d98-9350-aa0b17a6c434\") " pod="ai-tenant-e2e-inf-b-84b89f/test-model-84b89f-kserve-d8c475485-5prlt"
Oct 30 01:00:08.452201 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:08.452166    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5n62\" (UniqueName: \"kubernetes.io/projected/00a3e39a-01f0-4d98-9350-aa0b17a6c434-kube-api-access-k5n62\") pod \"test-model-84b89f-kserve-d8c475485-5prlt\" (UID: \"00a3e39a-01f0-4d98-9350-aa0b17a6c434\") " pod="ai-tenant-e2e-inf-b-84b89f/test-model-84b89f-kserve-d8c475485-5prlt"
Oct 30 01:00:08.566455 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:08.562692    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-84b89f/test-model-84b89f-kserve-d8c475485-5prlt"
Oct 30 01:00:09.061815 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:09.061782    5002 scope.go:122] "RemoveContainer" containerID="cf5baf6bf97a9f4e9b95b5efdd8d7180f6328f1e964d0b4cae74b92af25daaf7"
Oct 30 01:00:09.065043 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:00:09.064881    5002 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29888700-k6pvc_odh-ai-gateway-infra(d5279cb1-29b0-494a-b3b3-527e8af641dc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888700-k6pvc" podUID="d5279cb1-29b0-494a-b3b3-527e8af641dc"
Oct 30 01:00:09.071798 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:09.069856    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-84b89f/test-model-84b89f-kserve-d8c475485-5prlt"]
Oct 30 01:00:09.097955 ip-10-0-128-7 kubenswrapper[5002]: W1030 01:00:09.097921    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod00a3e39a_01f0_4d98_9350_aa0b17a6c434.slice/crio-fefc066dec24ba55b9133855ddb925c8d54386a2e8b68b8fddf18ea6e733d6c2 WatchSource:0}: Error finding container fefc066dec24ba55b9133855ddb925c8d54386a2e8b68b8fddf18ea6e733d6c2: Status 404 returned error can't find the container with id fefc066dec24ba55b9133855ddb925c8d54386a2e8b68b8fddf18ea6e733d6c2
Oct 30 01:00:09.098942 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:09.098913    5002 scope.go:122] "RemoveContainer" containerID="1323635381a3d79b74e0afced5180f6f4f2bcc65686aed8d50d6bb29559460d0"
Oct 30 01:00:09.099567 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:00:09.099496    5002 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-5007e1-29888700-dvm8j_odh-ai-gateway-infra(38a379b2-37b6-4185-8319-2e62dcec38c8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-5007e1-29888700-dvm8j" podUID="38a379b2-37b6-4185-8319-2e62dcec38c8"
Oct 30 01:00:09.103541 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:09.103518    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-8dbe73/test-model-8dbe73-kserve-78dbd9d5cc-5zdxp" event={"ID":"f3cd6ac5-b9a0-4096-94ee-851e2042e9fa","Type":"ContainerStarted","Data":"dced08146389e2646b17865cb9b689035cb4d3defe5a42def75fd1c96b6f67c3"}
Oct 30 01:00:09.108037 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:09.108015    5002 scope.go:122] "RemoveContainer" containerID="f42d54614481c6397b7e65761d0312be90c869f167d5624324fe55f0d754bfa1"
Oct 30 01:00:09.108403 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:00:09.108354    5002 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-b3bdf9-29888700-pzr6b_odh-ai-gateway-infra(bd72bc53-220d-40cd-abb8-36bc71b66094)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b3bdf9-29888700-pzr6b" podUID="bd72bc53-220d-40cd-abb8-36bc71b66094"
Oct 30 01:00:09.112353 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:09.112324    5002 generic.go:356] "Generic (PLEG): container finished" podID="5a907e40-190d-44fe-89b8-a51e621c04db" containerID="386fa1d5a2c4d3a43aa56a4e7887c0c518d65a7993cf95921b6bc0a401d6e2f8" exitCode=60
Oct 30 01:00:09.112544 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:09.112520    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ca5363-29888700-trbnd" event={"ID":"5a907e40-190d-44fe-89b8-a51e621c04db","Type":"ContainerDied","Data":"386fa1d5a2c4d3a43aa56a4e7887c0c518d65a7993cf95921b6bc0a401d6e2f8"}
Oct 30 01:00:09.112630 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:09.112616    5002 scope.go:122] "RemoveContainer" containerID="c1094ccf29ebd0350595ecfaf9d7a1a899e6251d81d823378cb77e2d278323ea"
Oct 30 01:00:09.112786 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:09.112767    5002 scope.go:122] "RemoveContainer" containerID="386fa1d5a2c4d3a43aa56a4e7887c0c518d65a7993cf95921b6bc0a401d6e2f8"
Oct 30 01:00:09.113092 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:00:09.113068    5002 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-ca5363-29888700-trbnd_odh-ai-gateway-infra(5a907e40-190d-44fe-89b8-a51e621c04db)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ca5363-29888700-trbnd" podUID="5a907e40-190d-44fe-89b8-a51e621c04db"
Oct 30 01:00:09.126969 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:09.126942    5002 scope.go:122] "RemoveContainer" containerID="70fa31a4dae40eb87d43802b774fae56d2a381fe759b25005c5751be178b8056"
Oct 30 01:00:09.127470 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:00:09.127434    5002 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-5f387c-29888700-vtvtq_odh-ai-gateway-infra(4af343ba-bc4d-4ec3-92df-1534780d3089)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5f387c-29888700-vtvtq" podUID="4af343ba-bc4d-4ec3-92df-1534780d3089"
Oct 30 01:00:09.134639 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:09.128871    5002 scope.go:122] "RemoveContainer" containerID="1f4025cfa5689b2203a7a9e9de122debe44b072e0f2563a0f56343b9f15bb934"
Oct 30 01:00:09.134639 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:00:09.129149    5002 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-11f521-29888700-khq6f_odh-ai-gateway-infra(7f443227-6ffc-40f4-80aa-1485ae41aa02)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-11f521-29888700-khq6f" podUID="7f443227-6ffc-40f4-80aa-1485ae41aa02"
Oct 30 01:00:09.134639 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:09.130776    5002 scope.go:122] "RemoveContainer" containerID="0643d42215bf6a5d84c1269e24289345566d810fed3f4f2079a63d302be67676"
Oct 30 01:00:09.134639 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:00:09.131058    5002 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-6f4fba-29888700-zvdkx_odh-ai-gateway-infra(6302c577-afc3-4632-a383-cb1fd8159992)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6f4fba-29888700-zvdkx" podUID="6302c577-afc3-4632-a383-cb1fd8159992"
Oct 30 01:00:09.134639 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:09.133483    5002 scope.go:122] "RemoveContainer" containerID="982c36549726383392d577034ce61f2700a7b1e1d242506a82e3064d1b865036"
Oct 30 01:00:09.134639 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:00:09.133706    5002 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-bab19a-29888700-f7tvc_odh-ai-gateway-infra(b6c630bd-ac41-4607-8ed9-5f68cf00be38)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-bab19a-29888700-f7tvc" podUID="b6c630bd-ac41-4607-8ed9-5f68cf00be38"
Oct 30 01:00:09.155363 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:09.155297    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-8dbe73/test-model-8dbe73-kserve-78dbd9d5cc-5zdxp" podStartSLOduration=3.155276431 podStartE2EDuration="3.155276431s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:00:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:00:09.153776547 +0000 UTC m=+1704.770209421" watchObservedRunningTime="2026-10-30 01:00:09.155276431 +0000 UTC m=+1704.771709310"
Oct 30 01:00:09.943180 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:09.943147    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-8dbe73/test-model-8dbe73-kserve-78dbd9d5cc-5zdxp"
Oct 30 01:00:10.143957 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:10.143927    5002 scope.go:122] "RemoveContainer" containerID="386fa1d5a2c4d3a43aa56a4e7887c0c518d65a7993cf95921b6bc0a401d6e2f8"
Oct 30 01:00:10.144354 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:00:10.144245    5002 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-ca5363-29888700-trbnd_odh-ai-gateway-infra(5a907e40-190d-44fe-89b8-a51e621c04db)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ca5363-29888700-trbnd" podUID="5a907e40-190d-44fe-89b8-a51e621c04db"
Oct 30 01:00:10.145823 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:10.145798    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-84b89f/test-model-84b89f-kserve-d8c475485-5prlt" event={"ID":"00a3e39a-01f0-4d98-9350-aa0b17a6c434","Type":"ContainerStarted","Data":"09c94c5020da7b32f56961507c85a26041e42f9685c73aa2348c2fc0bb9e0e22"}
Oct 30 01:00:10.145939 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:10.145835    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-84b89f/test-model-84b89f-kserve-d8c475485-5prlt" event={"ID":"00a3e39a-01f0-4d98-9350-aa0b17a6c434","Type":"ContainerStarted","Data":"fefc066dec24ba55b9133855ddb925c8d54386a2e8b68b8fddf18ea6e733d6c2"}
Oct 30 01:00:10.233675 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:10.233579    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-84b89f/test-model-84b89f-kserve-d8c475485-5prlt" podStartSLOduration=2.233563365 podStartE2EDuration="2.233563365s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:00:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:00:10.230225132 +0000 UTC m=+1705.846658015" watchObservedRunningTime="2026-10-30 01:00:10.233563365 +0000 UTC m=+1705.849996243"
Oct 30 01:00:10.564016 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:10.563979    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-84b89f/test-model-84b89f-kserve-d8c475485-5prlt"
Oct 30 01:00:11.155761 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:11.155728    5002 generic.go:356] "Generic (PLEG): container finished" podID="4a0578e8-4726-4c93-9c21-d09fac94c229" containerID="3ffeb3906c89d012b7dd3d63ff485e9872aa3b6d2cbb6723d33e9b567a928133" exitCode=0
Oct 30 01:00:11.156212 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:11.155803    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b1f6ed-openshift-default-767786fdd5-5gvj4" event={"ID":"4a0578e8-4726-4c93-9c21-d09fac94c229","Type":"ContainerDied","Data":"3ffeb3906c89d012b7dd3d63ff485e9872aa3b6d2cbb6723d33e9b567a928133"}
Oct 30 01:00:11.156721 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:11.156691    5002 scope.go:122] "RemoveContainer" containerID="3ffeb3906c89d012b7dd3d63ff485e9872aa3b6d2cbb6723d33e9b567a928133"
Oct 30 01:00:11.158486 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:11.158456    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:00:11.158603 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:11.158529    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:00:11.158603 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:11.158562    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:00:12.202195 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:12.202146    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b1f6ed-openshift-default-767786fdd5-5gvj4" event={"ID":"4a0578e8-4726-4c93-9c21-d09fac94c229","Type":"ContainerStarted","Data":"1adb422bfd667770bfc95c1be46853f10ec8122031f8b9cf7ab8ae826c1a740b"}
Oct 30 01:00:12.633231 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:12.633182    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-03d892/test-model-03d892-kserve-7f8c7cb699-kp2mb"
Oct 30 01:00:12.633762 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:12.633742    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-03d892/test-model-03d892-kserve-7f8c7cb699-kp2mb"
Oct 30 01:00:12.673403 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:12.667731    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-03d892/test-model-03d892-kserve-7f8c7cb699-kp2mb"
Oct 30 01:00:12.897460 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:12.896764    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-b1f6ed-openshift-default-767786fdd5-5gvj4"
Oct 30 01:00:12.920508 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:12.919124    5002 patch_prober.go:29] interesting pod/e2e-inf-b-b1f6ed-openshift-default-767786fdd5-5gvj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body=
Oct 30 01:00:12.920508 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:12.919203    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b1f6ed-openshift-default-767786fdd5-5gvj4" podUID="4a0578e8-4726-4c93-9c21-d09fac94c229" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused"
Oct 30 01:00:12.920764 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:00:12.920611    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbjdq" containerName="extractor"
Oct 30 01:00:13.015281 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:13.013303    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-t4s54"
Oct 30 01:00:13.130809 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:13.130765    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a107188-58d7-4c62-a2e0-e81eb181d540-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a107188-58d7-4c62-a2e0-e81eb181d540-workload-socket\") pod \"3a107188-58d7-4c62-a2e0-e81eb181d540\" (UID: \"3a107188-58d7-4c62-a2e0-e81eb181d540\") "
Oct 30 01:00:13.130985 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:13.130880    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a107188-58d7-4c62-a2e0-e81eb181d540-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a107188-58d7-4c62-a2e0-e81eb181d540-istiod-ca-cert\") pod \"3a107188-58d7-4c62-a2e0-e81eb181d540\" (UID: \"3a107188-58d7-4c62-a2e0-e81eb181d540\") "
Oct 30 01:00:13.130985 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:13.130957    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a107188-58d7-4c62-a2e0-e81eb181d540-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a107188-58d7-4c62-a2e0-e81eb181d540-workload-certs\") pod \"3a107188-58d7-4c62-a2e0-e81eb181d540\" (UID: \"3a107188-58d7-4c62-a2e0-e81eb181d540\") "
Oct 30 01:00:13.131141 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:13.131009    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a107188-58d7-4c62-a2e0-e81eb181d540-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a107188-58d7-4c62-a2e0-e81eb181d540-istio-data\") pod \"3a107188-58d7-4c62-a2e0-e81eb181d540\" (UID: \"3a107188-58d7-4c62-a2e0-e81eb181d540\") "
Oct 30 01:00:13.131141 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:13.131093    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3a107188-58d7-4c62-a2e0-e81eb181d540-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a107188-58d7-4c62-a2e0-e81eb181d540-istio-podinfo\") pod \"3a107188-58d7-4c62-a2e0-e81eb181d540\" (UID: \"3a107188-58d7-4c62-a2e0-e81eb181d540\") "
Oct 30 01:00:13.131141 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:13.131106    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a107188-58d7-4c62-a2e0-e81eb181d540-workload-socket" pod "3a107188-58d7-4c62-a2e0-e81eb181d540" (UID: "3a107188-58d7-4c62-a2e0-e81eb181d540"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:00:13.131314 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:13.131188    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a107188-58d7-4c62-a2e0-e81eb181d540-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a107188-58d7-4c62-a2e0-e81eb181d540-istio-envoy\") pod \"3a107188-58d7-4c62-a2e0-e81eb181d540\" (UID: \"3a107188-58d7-4c62-a2e0-e81eb181d540\") "
Oct 30 01:00:13.131314 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:13.131230    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a107188-58d7-4c62-a2e0-e81eb181d540-istio-token\" (UniqueName: \"kubernetes.io/projected/3a107188-58d7-4c62-a2e0-e81eb181d540-istio-token\") pod \"3a107188-58d7-4c62-a2e0-e81eb181d540\" (UID: \"3a107188-58d7-4c62-a2e0-e81eb181d540\") "
Oct 30 01:00:13.131314 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:13.131269    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a107188-58d7-4c62-a2e0-e81eb181d540-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a107188-58d7-4c62-a2e0-e81eb181d540-credential-socket\") pod \"3a107188-58d7-4c62-a2e0-e81eb181d540\" (UID: \"3a107188-58d7-4c62-a2e0-e81eb181d540\") "
Oct 30 01:00:13.131504 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:13.131324    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a107188-58d7-4c62-a2e0-e81eb181d540-istiod-ca-cert" pod "3a107188-58d7-4c62-a2e0-e81eb181d540" (UID: "3a107188-58d7-4c62-a2e0-e81eb181d540"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:00:13.131504 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:13.131346    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a107188-58d7-4c62-a2e0-e81eb181d540-kube-api-access-cl67m\" (UniqueName: \"kubernetes.io/projected/3a107188-58d7-4c62-a2e0-e81eb181d540-kube-api-access-cl67m\") pod \"3a107188-58d7-4c62-a2e0-e81eb181d540\" (UID: \"3a107188-58d7-4c62-a2e0-e81eb181d540\") "
Oct 30 01:00:13.131504 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:13.131349    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a107188-58d7-4c62-a2e0-e81eb181d540-workload-certs" pod "3a107188-58d7-4c62-a2e0-e81eb181d540" (UID: "3a107188-58d7-4c62-a2e0-e81eb181d540"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:00:13.131663 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:13.131612    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a107188-58d7-4c62-a2e0-e81eb181d540-istio-data" pod "3a107188-58d7-4c62-a2e0-e81eb181d540" (UID: "3a107188-58d7-4c62-a2e0-e81eb181d540"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:00:13.132337 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:13.132075    5002 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a107188-58d7-4c62-a2e0-e81eb181d540-istiod-ca-cert\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:00:13.132337 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:13.132094    5002 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a107188-58d7-4c62-a2e0-e81eb181d540-workload-certs\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:00:13.132337 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:13.132104    5002 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a107188-58d7-4c62-a2e0-e81eb181d540-istio-data\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:00:13.132337 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:13.132112    5002 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a107188-58d7-4c62-a2e0-e81eb181d540-workload-socket\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:00:13.132739 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:13.132416    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a107188-58d7-4c62-a2e0-e81eb181d540-credential-socket" pod "3a107188-58d7-4c62-a2e0-e81eb181d540" (UID: "3a107188-58d7-4c62-a2e0-e81eb181d540"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:00:13.133941 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:13.133905    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a107188-58d7-4c62-a2e0-e81eb181d540-istio-envoy" pod "3a107188-58d7-4c62-a2e0-e81eb181d540" (UID: "3a107188-58d7-4c62-a2e0-e81eb181d540"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:00:13.135823 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:13.135757    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3a107188-58d7-4c62-a2e0-e81eb181d540-istio-podinfo" pod "3a107188-58d7-4c62-a2e0-e81eb181d540" (UID: "3a107188-58d7-4c62-a2e0-e81eb181d540"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:00:13.135936 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:13.135914    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a107188-58d7-4c62-a2e0-e81eb181d540-kube-api-access-cl67m" pod "3a107188-58d7-4c62-a2e0-e81eb181d540" (UID: "3a107188-58d7-4c62-a2e0-e81eb181d540"). InnerVolumeSpecName "kube-api-access-cl67m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:00:13.136632 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:13.136605    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a107188-58d7-4c62-a2e0-e81eb181d540-istio-token" pod "3a107188-58d7-4c62-a2e0-e81eb181d540" (UID: "3a107188-58d7-4c62-a2e0-e81eb181d540"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:00:13.220422 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:13.220373    5002 generic.go:356] "Generic (PLEG): container finished" podID="3a107188-58d7-4c62-a2e0-e81eb181d540" containerID="cf7d7f440c1f4124242028ef09533f0122a4dc6177bbc75d06fc326e5c288fe1" exitCode=0
Oct 30 01:00:13.220816 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:13.220425    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-t4s54" event={"ID":"3a107188-58d7-4c62-a2e0-e81eb181d540","Type":"ContainerDied","Data":"cf7d7f440c1f4124242028ef09533f0122a4dc6177bbc75d06fc326e5c288fe1"}
Oct 30 01:00:13.220816 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:13.220466    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-t4s54"
Oct 30 01:00:13.220816 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:13.220483    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-t4s54" event={"ID":"3a107188-58d7-4c62-a2e0-e81eb181d540","Type":"ContainerDied","Data":"e25fd4f1ced7f5c66f4fded538e156709c7bd25391a20e183518522213516415"}
Oct 30 01:00:13.220816 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:13.220507    5002 scope.go:122] "RemoveContainer" containerID="cf7d7f440c1f4124242028ef09533f0122a4dc6177bbc75d06fc326e5c288fe1"
Oct 30 01:00:13.232733 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:13.232690    5002 scope.go:122] "RemoveContainer" containerID="cf7d7f440c1f4124242028ef09533f0122a4dc6177bbc75d06fc326e5c288fe1"
Oct 30 01:00:13.233101 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:00:13.233074    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf7d7f440c1f4124242028ef09533f0122a4dc6177bbc75d06fc326e5c288fe1\": container with ID starting with cf7d7f440c1f4124242028ef09533f0122a4dc6177bbc75d06fc326e5c288fe1 not found: ID does not exist" containerID="cf7d7f440c1f4124242028ef09533f0122a4dc6177bbc75d06fc326e5c288fe1"
Oct 30 01:00:13.233154 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:13.233114    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf7d7f440c1f4124242028ef09533f0122a4dc6177bbc75d06fc326e5c288fe1"} err="failed to get container status \"cf7d7f440c1f4124242028ef09533f0122a4dc6177bbc75d06fc326e5c288fe1\": rpc error: code = NotFound desc = could not find container \"cf7d7f440c1f4124242028ef09533f0122a4dc6177bbc75d06fc326e5c288fe1\": container with ID starting with cf7d7f440c1f4124242028ef09533f0122a4dc6177bbc75d06fc326e5c288fe1 not found: ID does not exist"
Oct 30 01:00:13.233745 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:13.233725    5002 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a107188-58d7-4c62-a2e0-e81eb181d540-istio-podinfo\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:00:13.233808 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:13.233747    5002 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a107188-58d7-4c62-a2e0-e81eb181d540-istio-envoy\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:00:13.233808 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:13.233755    5002 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a107188-58d7-4c62-a2e0-e81eb181d540-istio-token\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:00:13.233808 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:13.233764    5002 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a107188-58d7-4c62-a2e0-e81eb181d540-credential-socket\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:00:13.233808 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:13.233773    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cl67m\" (UniqueName: \"kubernetes.io/projected/3a107188-58d7-4c62-a2e0-e81eb181d540-kube-api-access-cl67m\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:00:13.302151 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:13.302082    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-t4s54"]
Oct 30 01:00:13.306507 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:13.306478    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-t4s54"]
Oct 30 01:00:13.866400 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:13.864178    5002 patch_prober.go:29] interesting pod/e2e-inf-b-b1f6ed-openshift-default-767786fdd5-5gvj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body=
Oct 30 01:00:13.866400 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:13.864241    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b1f6ed-openshift-default-767786fdd5-5gvj4" podUID="4a0578e8-4726-4c93-9c21-d09fac94c229" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused"
Oct 30 01:00:14.499259 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:14.499220    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a107188-58d7-4c62-a2e0-e81eb181d540" path="/var/lib/kubelet/pods/3a107188-58d7-4c62-a2e0-e81eb181d540/volumes"
Oct 30 01:00:14.857451 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:14.855768    5002 patch_prober.go:29] interesting pod/e2e-inf-b-b1f6ed-openshift-default-767786fdd5-5gvj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body=
Oct 30 01:00:14.857451 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:14.855830    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b1f6ed-openshift-default-767786fdd5-5gvj4" podUID="4a0578e8-4726-4c93-9c21-d09fac94c229" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused"
Oct 30 01:00:15.288958 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:15.288483    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w1-48df96/auth-test-model-48df96-kserve-9bb4b5585-cvvln"]
Oct 30 01:00:15.289985 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:15.289953    5002 cpu_manager.go:414] "containerMap: removing container" podUID="3a107188-58d7-4c62-a2e0-e81eb181d540" containerName="istio-proxy"
Oct 30 01:00:15.289985 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:15.289982    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a107188-58d7-4c62-a2e0-e81eb181d540" containerName="istio-proxy"
Oct 30 01:00:15.290261 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:15.290237    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a107188-58d7-4c62-a2e0-e81eb181d540" containerName="istio-proxy"
Oct 30 01:00:15.296013 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:15.295991    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-48df96/auth-test-model-48df96-kserve-9bb4b5585-cvvln"
Oct 30 01:00:15.327180 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:15.320350    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-48df96/auth-test-model-48df96-kserve-9bb4b5585-cvvln"]
Oct 30 01:00:15.335968 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:15.335862    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-48df96\"/\"kube-root-ca.crt\""
Oct 30 01:00:15.337012 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:15.336864    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-48df96\"/\"openshift-service-ca.crt\""
Oct 30 01:00:15.364024 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:15.360760    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-48df96\"/\"default-dockercfg-7f7cd\""
Oct 30 01:00:15.364024 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:15.360865    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-48df96\"/\"auth-test-model-48df96-kserve-self-signed-certs\""
Oct 30 01:00:15.410859 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:15.410814    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22351ddc-9884-4058-b71e-7c320c2da701-tls-certs\") pod \"auth-test-model-48df96-kserve-9bb4b5585-cvvln\" (UID: \"22351ddc-9884-4058-b71e-7c320c2da701\") " pod="ai-tenant-e2e-shared-a-w1-48df96/auth-test-model-48df96-kserve-9bb4b5585-cvvln"
Oct 30 01:00:15.411072 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:15.410905    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22351ddc-9884-4058-b71e-7c320c2da701-dshm\") pod \"auth-test-model-48df96-kserve-9bb4b5585-cvvln\" (UID: \"22351ddc-9884-4058-b71e-7c320c2da701\") " pod="ai-tenant-e2e-shared-a-w1-48df96/auth-test-model-48df96-kserve-9bb4b5585-cvvln"
Oct 30 01:00:15.411072 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:15.410939    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22351ddc-9884-4058-b71e-7c320c2da701-model-cache\") pod \"auth-test-model-48df96-kserve-9bb4b5585-cvvln\" (UID: \"22351ddc-9884-4058-b71e-7c320c2da701\") " pod="ai-tenant-e2e-shared-a-w1-48df96/auth-test-model-48df96-kserve-9bb4b5585-cvvln"
Oct 30 01:00:15.411072 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:15.410998    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4g4k4\" (UniqueName: \"kubernetes.io/projected/22351ddc-9884-4058-b71e-7c320c2da701-kube-api-access-4g4k4\") pod \"auth-test-model-48df96-kserve-9bb4b5585-cvvln\" (UID: \"22351ddc-9884-4058-b71e-7c320c2da701\") " pod="ai-tenant-e2e-shared-a-w1-48df96/auth-test-model-48df96-kserve-9bb4b5585-cvvln"
Oct 30 01:00:15.411262 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:15.411089    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22351ddc-9884-4058-b71e-7c320c2da701-tmp-dir\") pod \"auth-test-model-48df96-kserve-9bb4b5585-cvvln\" (UID: \"22351ddc-9884-4058-b71e-7c320c2da701\") " pod="ai-tenant-e2e-shared-a-w1-48df96/auth-test-model-48df96-kserve-9bb4b5585-cvvln"
Oct 30 01:00:15.411262 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:15.411192    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22351ddc-9884-4058-b71e-7c320c2da701-home\") pod \"auth-test-model-48df96-kserve-9bb4b5585-cvvln\" (UID: \"22351ddc-9884-4058-b71e-7c320c2da701\") " pod="ai-tenant-e2e-shared-a-w1-48df96/auth-test-model-48df96-kserve-9bb4b5585-cvvln"
Oct 30 01:00:15.512258 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:15.512200    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22351ddc-9884-4058-b71e-7c320c2da701-home\") pod \"auth-test-model-48df96-kserve-9bb4b5585-cvvln\" (UID: \"22351ddc-9884-4058-b71e-7c320c2da701\") " pod="ai-tenant-e2e-shared-a-w1-48df96/auth-test-model-48df96-kserve-9bb4b5585-cvvln"
Oct 30 01:00:15.512688 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:15.512367    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22351ddc-9884-4058-b71e-7c320c2da701-tls-certs\") pod \"auth-test-model-48df96-kserve-9bb4b5585-cvvln\" (UID: \"22351ddc-9884-4058-b71e-7c320c2da701\") " pod="ai-tenant-e2e-shared-a-w1-48df96/auth-test-model-48df96-kserve-9bb4b5585-cvvln"
Oct 30 01:00:15.512688 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:15.512435    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22351ddc-9884-4058-b71e-7c320c2da701-dshm\") pod \"auth-test-model-48df96-kserve-9bb4b5585-cvvln\" (UID: \"22351ddc-9884-4058-b71e-7c320c2da701\") " pod="ai-tenant-e2e-shared-a-w1-48df96/auth-test-model-48df96-kserve-9bb4b5585-cvvln"
Oct 30 01:00:15.512688 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:15.512467    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22351ddc-9884-4058-b71e-7c320c2da701-model-cache\") pod \"auth-test-model-48df96-kserve-9bb4b5585-cvvln\" (UID: \"22351ddc-9884-4058-b71e-7c320c2da701\") " pod="ai-tenant-e2e-shared-a-w1-48df96/auth-test-model-48df96-kserve-9bb4b5585-cvvln"
Oct 30 01:00:15.512688 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:15.512538    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4g4k4\" (UniqueName: \"kubernetes.io/projected/22351ddc-9884-4058-b71e-7c320c2da701-kube-api-access-4g4k4\") pod \"auth-test-model-48df96-kserve-9bb4b5585-cvvln\" (UID: \"22351ddc-9884-4058-b71e-7c320c2da701\") " pod="ai-tenant-e2e-shared-a-w1-48df96/auth-test-model-48df96-kserve-9bb4b5585-cvvln"
Oct 30 01:00:15.512688 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:15.512626    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22351ddc-9884-4058-b71e-7c320c2da701-tmp-dir\") pod \"auth-test-model-48df96-kserve-9bb4b5585-cvvln\" (UID: \"22351ddc-9884-4058-b71e-7c320c2da701\") " pod="ai-tenant-e2e-shared-a-w1-48df96/auth-test-model-48df96-kserve-9bb4b5585-cvvln"
Oct 30 01:00:15.513142 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:15.513117    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22351ddc-9884-4058-b71e-7c320c2da701-tmp-dir\") pod \"auth-test-model-48df96-kserve-9bb4b5585-cvvln\" (UID: \"22351ddc-9884-4058-b71e-7c320c2da701\") " pod="ai-tenant-e2e-shared-a-w1-48df96/auth-test-model-48df96-kserve-9bb4b5585-cvvln"
Oct 30 01:00:15.513433 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:15.513368    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22351ddc-9884-4058-b71e-7c320c2da701-model-cache\") pod \"auth-test-model-48df96-kserve-9bb4b5585-cvvln\" (UID: \"22351ddc-9884-4058-b71e-7c320c2da701\") " pod="ai-tenant-e2e-shared-a-w1-48df96/auth-test-model-48df96-kserve-9bb4b5585-cvvln"
Oct 30 01:00:15.513675 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:15.513655    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22351ddc-9884-4058-b71e-7c320c2da701-home\") pod \"auth-test-model-48df96-kserve-9bb4b5585-cvvln\" (UID: \"22351ddc-9884-4058-b71e-7c320c2da701\") " pod="ai-tenant-e2e-shared-a-w1-48df96/auth-test-model-48df96-kserve-9bb4b5585-cvvln"
Oct 30 01:00:15.516219 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:15.516194    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22351ddc-9884-4058-b71e-7c320c2da701-dshm\") pod \"auth-test-model-48df96-kserve-9bb4b5585-cvvln\" (UID: \"22351ddc-9884-4058-b71e-7c320c2da701\") " pod="ai-tenant-e2e-shared-a-w1-48df96/auth-test-model-48df96-kserve-9bb4b5585-cvvln"
Oct 30 01:00:15.516895 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:15.516842    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22351ddc-9884-4058-b71e-7c320c2da701-tls-certs\") pod \"auth-test-model-48df96-kserve-9bb4b5585-cvvln\" (UID: \"22351ddc-9884-4058-b71e-7c320c2da701\") " pod="ai-tenant-e2e-shared-a-w1-48df96/auth-test-model-48df96-kserve-9bb4b5585-cvvln"
Oct 30 01:00:15.611335 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:15.611208    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4g4k4\" (UniqueName: \"kubernetes.io/projected/22351ddc-9884-4058-b71e-7c320c2da701-kube-api-access-4g4k4\") pod \"auth-test-model-48df96-kserve-9bb4b5585-cvvln\" (UID: \"22351ddc-9884-4058-b71e-7c320c2da701\") " pod="ai-tenant-e2e-shared-a-w1-48df96/auth-test-model-48df96-kserve-9bb4b5585-cvvln"
Oct 30 01:00:15.642866 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:15.642824    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-48df96/auth-test-model-48df96-kserve-9bb4b5585-cvvln"
Oct 30 01:00:15.856406 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:15.855479    5002 patch_prober.go:29] interesting pod/e2e-inf-b-b1f6ed-openshift-default-767786fdd5-5gvj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body=
Oct 30 01:00:15.856406 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:15.855544    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b1f6ed-openshift-default-767786fdd5-5gvj4" podUID="4a0578e8-4726-4c93-9c21-d09fac94c229" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused"
Oct 30 01:00:16.200248 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:16.200211    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-48df96/auth-test-model-48df96-kserve-9bb4b5585-cvvln"]
Oct 30 01:00:16.353400 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:16.349537    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-48df96/auth-test-model-48df96-kserve-9bb4b5585-cvvln" event={"ID":"22351ddc-9884-4058-b71e-7c320c2da701","Type":"ContainerStarted","Data":"8ab78a1df6e272ee61949526d60496490e917347d3748d45b4d3aeb97c217a58"}
Oct 30 01:00:16.884529 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:16.873953    5002 patch_prober.go:29] interesting pod/e2e-inf-b-b1f6ed-openshift-default-767786fdd5-5gvj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body=
Oct 30 01:00:16.884529 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:16.874017    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b1f6ed-openshift-default-767786fdd5-5gvj4" podUID="4a0578e8-4726-4c93-9c21-d09fac94c229" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused"
Oct 30 01:00:16.943029 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:16.942997    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-8dbe73/test-model-8dbe73-kserve-78dbd9d5cc-5zdxp"
Oct 30 01:00:16.989874 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:16.989841    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-8dbe73/test-model-8dbe73-kserve-78dbd9d5cc-5zdxp"
Oct 30 01:00:17.388111 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:17.388044    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-48df96/auth-test-model-48df96-kserve-9bb4b5585-cvvln" event={"ID":"22351ddc-9884-4058-b71e-7c320c2da701","Type":"ContainerStarted","Data":"578e94fe330d154d28c00680bf3416b451f2a56bc0672cde0b1e37b779581440"}
Oct 30 01:00:17.400927 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:17.400895    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-8dbe73/test-model-8dbe73-kserve-78dbd9d5cc-5zdxp"
Oct 30 01:00:17.436972 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:17.436908    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w1-48df96/auth-test-model-48df96-kserve-9bb4b5585-cvvln" podStartSLOduration=2.436887214 podStartE2EDuration="2.436887214s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:00:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:00:17.432019447 +0000 UTC m=+1713.048452327" watchObservedRunningTime="2026-10-30 01:00:17.436887214 +0000 UTC m=+1713.053320094"
Oct 30 01:00:17.491403 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:17.488053    5002 scope.go:122] "RemoveContainer" containerID="b09f8bdbe09a4f05a4237a051cf4304c30df663110fceab326eea8510faad135"
Oct 30 01:00:17.643628 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:17.643478    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w1-48df96/auth-test-model-48df96-kserve-9bb4b5585-cvvln"
Oct 30 01:00:17.798405 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:17.797452    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-b1f6ed-openshift-default-767786fdd5-5gvj4"
Oct 30 01:00:18.214430 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:18.212940    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-b1f6ed-openshift-default-767786fdd5-5gvj4"
Oct 30 01:00:18.400446 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:18.400307    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-52650f-29888700-8mvn6" event={"ID":"09fb128c-0670-4260-a77b-ae32de23fa0d","Type":"ContainerStarted","Data":"f7f7c860079a37739b55fcce44de65100f13f349f1863bb6ba96afec8fcfea51"}
Oct 30 01:00:18.402680 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:18.402651    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-b1f6ed-openshift-default-767786fdd5-5gvj4"
Oct 30 01:00:18.487878 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:18.487843    5002 scope.go:122] "RemoveContainer" containerID="54153540b9cbb314851c0da45b9629e291ef335f37fd4fa18b2d95a4c5dec048"
Oct 30 01:00:18.488130 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:18.488109    5002 scope.go:122] "RemoveContainer" containerID="982c36549726383392d577034ce61f2700a7b1e1d242506a82e3064d1b865036"
Oct 30 01:00:18.488318 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:18.488299    5002 scope.go:122] "RemoveContainer" containerID="0643d42215bf6a5d84c1269e24289345566d810fed3f4f2079a63d302be67676"
Oct 30 01:00:18.488478 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:18.488458    5002 scope.go:122] "RemoveContainer" containerID="441e5c13bff6f0f1fb7b82264ceaccf2a98dc69609e4fb64193662b9fcda5160"
Oct 30 01:00:18.489098 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:18.488896    5002 scope.go:122] "RemoveContainer" containerID="a4572a0daa660fcdf452acd07ef154e37825a339040563b851f9b439c994c008"
Oct 30 01:00:18.563880 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:18.563843    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-84b89f/test-model-84b89f-kserve-d8c475485-5prlt"
Oct 30 01:00:18.589128 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:18.589065    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-84b89f/test-model-84b89f-kserve-d8c475485-5prlt"
Oct 30 01:00:19.412228 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:19.412190    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6f4fba-29888700-zvdkx" event={"ID":"6302c577-afc3-4632-a383-cb1fd8159992","Type":"ContainerStarted","Data":"ba1ec04a3a05167d1560d31009c209b3722d400ef52046b07e782b1d24d44f91"}
Oct 30 01:00:19.414452 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:19.414415    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-cce513-29888700-744n2" event={"ID":"743ebbf6-b4db-4774-b836-de5d7c716ca7","Type":"ContainerStarted","Data":"ded5c17024715fcaaffefe84542116d7ff5e57cec48cbcd9853525f58e1ee8b8"}
Oct 30 01:00:19.417002 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:19.416976    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8d2a70-29888700-jlmvv" event={"ID":"d22fc624-e367-49a0-ba47-9a5dcabdde1f","Type":"ContainerStarted","Data":"6ceaac9ce998e813e0a4e98f3a3dc1a1480ac3c07200b11fadb365dc6f42b043"}
Oct 30 01:00:19.420029 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:19.419999    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1a11cf-29888700-d9s5c" event={"ID":"c111db1f-7e6f-44b8-b211-1658aee31337","Type":"ContainerStarted","Data":"9a858720da3de764d91b2029bc1d8a7c476b209a952d4a613ed60a6e45417586"}
Oct 30 01:00:19.422167 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:19.422124    5002 generic.go:356] "Generic (PLEG): container finished" podID="09fb128c-0670-4260-a77b-ae32de23fa0d" containerID="f7f7c860079a37739b55fcce44de65100f13f349f1863bb6ba96afec8fcfea51" exitCode=60
Oct 30 01:00:19.422302 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:19.422205    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-52650f-29888700-8mvn6" event={"ID":"09fb128c-0670-4260-a77b-ae32de23fa0d","Type":"ContainerDied","Data":"f7f7c860079a37739b55fcce44de65100f13f349f1863bb6ba96afec8fcfea51"}
Oct 30 01:00:19.422302 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:19.422240    5002 scope.go:122] "RemoveContainer" containerID="b09f8bdbe09a4f05a4237a051cf4304c30df663110fceab326eea8510faad135"
Oct 30 01:00:19.422682 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:19.422660    5002 scope.go:122] "RemoveContainer" containerID="f7f7c860079a37739b55fcce44de65100f13f349f1863bb6ba96afec8fcfea51"
Oct 30 01:00:19.422995 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:00:19.422969    5002 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-52650f-29888700-8mvn6_odh-ai-gateway-infra(09fb128c-0670-4260-a77b-ae32de23fa0d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-52650f-29888700-8mvn6" podUID="09fb128c-0670-4260-a77b-ae32de23fa0d"
Oct 30 01:00:19.427875 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:19.427847    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-bab19a-29888700-f7tvc" event={"ID":"b6c630bd-ac41-4607-8ed9-5f68cf00be38","Type":"ContainerStarted","Data":"02ad2188c2ee4d393177e35b8e25369a5ab0de1dea76867ea10b85ac113e253b"}
Oct 30 01:00:19.444806 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:19.444780    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-84b89f/test-model-84b89f-kserve-d8c475485-5prlt"
Oct 30 01:00:19.460754 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:19.460230    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-52650f-29888700-8mvn6"]
Oct 30 01:00:19.487715 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:19.487687    5002 scope.go:122] "RemoveContainer" containerID="cf5baf6bf97a9f4e9b95b5efdd8d7180f6328f1e964d0b4cae74b92af25daaf7"
Oct 30 01:00:19.487866 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:19.487794    5002 scope.go:122] "RemoveContainer" containerID="cdc6b212bb47ace68ace90e7eaed307569eb5a305ec64e50bafab9ca50204fbe"
Oct 30 01:00:19.487948 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:19.487926    5002 scope.go:122] "RemoveContainer" containerID="70fa31a4dae40eb87d43802b774fae56d2a381fe759b25005c5751be178b8056"
Oct 30 01:00:19.607294 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:19.607256    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1a11cf-29888700-d9s5c"]
Oct 30 01:00:20.437956 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:20.437922    5002 generic.go:356] "Generic (PLEG): container finished" podID="b6c630bd-ac41-4607-8ed9-5f68cf00be38" containerID="02ad2188c2ee4d393177e35b8e25369a5ab0de1dea76867ea10b85ac113e253b" exitCode=60
Oct 30 01:00:20.438427 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:20.438004    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-bab19a-29888700-f7tvc" event={"ID":"b6c630bd-ac41-4607-8ed9-5f68cf00be38","Type":"ContainerDied","Data":"02ad2188c2ee4d393177e35b8e25369a5ab0de1dea76867ea10b85ac113e253b"}
Oct 30 01:00:20.438427 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:20.438074    5002 scope.go:122] "RemoveContainer" containerID="982c36549726383392d577034ce61f2700a7b1e1d242506a82e3064d1b865036"
Oct 30 01:00:20.438621 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:20.438601    5002 scope.go:122] "RemoveContainer" containerID="02ad2188c2ee4d393177e35b8e25369a5ab0de1dea76867ea10b85ac113e253b"
Oct 30 01:00:20.439321 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:00:20.439187    5002 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-bab19a-29888700-f7tvc_odh-ai-gateway-infra(b6c630bd-ac41-4607-8ed9-5f68cf00be38)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-bab19a-29888700-f7tvc" podUID="b6c630bd-ac41-4607-8ed9-5f68cf00be38"
Oct 30 01:00:20.440455 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:20.440430    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8e0f66-29888700-tnj7q" event={"ID":"5fa9212f-5c4d-47a2-a855-2c18a807533e","Type":"ContainerStarted","Data":"a9abb13a5de9a91d2294f3feeed77547a291fc384876850dbc8e8b3ec59f7645"}
Oct 30 01:00:20.444350 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:20.444326    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888700-k6pvc" event={"ID":"d5279cb1-29b0-494a-b3b3-527e8af641dc","Type":"ContainerStarted","Data":"42265725d58abfe459dc726587ca11c8def903b8e431398183267501da12d3e1"}
Oct 30 01:00:20.446600 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:20.446580    5002 generic.go:356] "Generic (PLEG): container finished" podID="6302c577-afc3-4632-a383-cb1fd8159992" containerID="ba1ec04a3a05167d1560d31009c209b3722d400ef52046b07e782b1d24d44f91" exitCode=60
Oct 30 01:00:20.446698 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:20.446631    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6f4fba-29888700-zvdkx" event={"ID":"6302c577-afc3-4632-a383-cb1fd8159992","Type":"ContainerDied","Data":"ba1ec04a3a05167d1560d31009c209b3722d400ef52046b07e782b1d24d44f91"}
Oct 30 01:00:20.447006 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:20.446989    5002 scope.go:122] "RemoveContainer" containerID="ba1ec04a3a05167d1560d31009c209b3722d400ef52046b07e782b1d24d44f91"
Oct 30 01:00:20.447409 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:00:20.447370    5002 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-6f4fba-29888700-zvdkx_odh-ai-gateway-infra(6302c577-afc3-4632-a383-cb1fd8159992)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6f4fba-29888700-zvdkx" podUID="6302c577-afc3-4632-a383-cb1fd8159992"
Oct 30 01:00:20.448931 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:20.448911    5002 generic.go:356] "Generic (PLEG): container finished" podID="743ebbf6-b4db-4774-b836-de5d7c716ca7" containerID="ded5c17024715fcaaffefe84542116d7ff5e57cec48cbcd9853525f58e1ee8b8" exitCode=60
Oct 30 01:00:20.449029 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:20.448979    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-cce513-29888700-744n2" event={"ID":"743ebbf6-b4db-4774-b836-de5d7c716ca7","Type":"ContainerDied","Data":"ded5c17024715fcaaffefe84542116d7ff5e57cec48cbcd9853525f58e1ee8b8"}
Oct 30 01:00:20.449317 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:20.449297    5002 scope.go:122] "RemoveContainer" containerID="ded5c17024715fcaaffefe84542116d7ff5e57cec48cbcd9853525f58e1ee8b8"
Oct 30 01:00:20.449712 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:00:20.449685    5002 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-cce513-29888700-744n2_odh-ai-gateway-infra(743ebbf6-b4db-4774-b836-de5d7c716ca7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-cce513-29888700-744n2" podUID="743ebbf6-b4db-4774-b836-de5d7c716ca7"
Oct 30 01:00:20.451104 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:20.451084    5002 generic.go:356] "Generic (PLEG): container finished" podID="d22fc624-e367-49a0-ba47-9a5dcabdde1f" containerID="6ceaac9ce998e813e0a4e98f3a3dc1a1480ac3c07200b11fadb365dc6f42b043" exitCode=60
Oct 30 01:00:20.451202 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:20.451159    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8d2a70-29888700-jlmvv" event={"ID":"d22fc624-e367-49a0-ba47-9a5dcabdde1f","Type":"ContainerDied","Data":"6ceaac9ce998e813e0a4e98f3a3dc1a1480ac3c07200b11fadb365dc6f42b043"}
Oct 30 01:00:20.451544 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:20.451521    5002 scope.go:122] "RemoveContainer" containerID="6ceaac9ce998e813e0a4e98f3a3dc1a1480ac3c07200b11fadb365dc6f42b043"
Oct 30 01:00:20.451885 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:00:20.451846    5002 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-8d2a70-29888700-jlmvv_odh-ai-gateway-infra(d22fc624-e367-49a0-ba47-9a5dcabdde1f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8d2a70-29888700-jlmvv" podUID="d22fc624-e367-49a0-ba47-9a5dcabdde1f"
Oct 30 01:00:20.453170 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:20.453145    5002 generic.go:356] "Generic (PLEG): container finished" podID="c111db1f-7e6f-44b8-b211-1658aee31337" containerID="9a858720da3de764d91b2029bc1d8a7c476b209a952d4a613ed60a6e45417586" exitCode=60
Oct 30 01:00:20.453280 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:20.453226    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1a11cf-29888700-d9s5c" event={"ID":"c111db1f-7e6f-44b8-b211-1658aee31337","Type":"ContainerDied","Data":"9a858720da3de764d91b2029bc1d8a7c476b209a952d4a613ed60a6e45417586"}
Oct 30 01:00:20.455427 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:20.455402    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5f387c-29888700-vtvtq" event={"ID":"4af343ba-bc4d-4ec3-92df-1534780d3089","Type":"ContainerStarted","Data":"3365c4917c725676bfe2bc923168fce4ad541db86703a6230cd2ead2d6235395"}
Oct 30 01:00:20.458007 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:20.457994    5002 scope.go:122] "RemoveContainer" containerID="0643d42215bf6a5d84c1269e24289345566d810fed3f4f2079a63d302be67676"
Oct 30 01:00:20.475293 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:20.475264    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6f4fba-29888700-zvdkx"]
Oct 30 01:00:20.475966 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:20.475866    5002 scope.go:122] "RemoveContainer" containerID="a4572a0daa660fcdf452acd07ef154e37825a339040563b851f9b439c994c008"
Oct 30 01:00:20.491444 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:20.487891    5002 scope.go:122] "RemoveContainer" containerID="f42d54614481c6397b7e65761d0312be90c869f167d5624324fe55f0d754bfa1"
Oct 30 01:00:20.491444 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:20.488878    5002 scope.go:122] "RemoveContainer" containerID="3d3fdb0735f79018c13a82512cd7420f9099a1138403ae93a519dcf547e8deee"
Oct 30 01:00:20.514449 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:20.514427    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-bab19a-29888700-f7tvc"]
Oct 30 01:00:20.522004 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:20.521892    5002 scope.go:122] "RemoveContainer" containerID="54153540b9cbb314851c0da45b9629e291ef335f37fd4fa18b2d95a4c5dec048"
Oct 30 01:00:20.582838 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:20.582643    5002 scope.go:122] "RemoveContainer" containerID="441e5c13bff6f0f1fb7b82264ceaccf2a98dc69609e4fb64193662b9fcda5160"
Oct 30 01:00:20.627431 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:20.622239    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8d2a70-29888700-jlmvv"]
Oct 30 01:00:20.695611 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:20.695565    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-cce513-29888700-744n2"]
Oct 30 01:00:20.729359 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:20.729312    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1a11cf-29888700-d9s5c"
Oct 30 01:00:20.785881 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:20.785855    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-52650f-29888700-8mvn6"
Oct 30 01:00:20.907830 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:20.907797    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09fb128c-0670-4260-a77b-ae32de23fa0d-kube-api-access-brqqq\" (UniqueName: \"kubernetes.io/projected/09fb128c-0670-4260-a77b-ae32de23fa0d-kube-api-access-brqqq\") pod \"09fb128c-0670-4260-a77b-ae32de23fa0d\" (UID: \"09fb128c-0670-4260-a77b-ae32de23fa0d\") "
Oct 30 01:00:20.908099 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:20.908080    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09fb128c-0670-4260-a77b-ae32de23fa0d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/09fb128c-0670-4260-a77b-ae32de23fa0d-maas-api-service-ca\") pod \"09fb128c-0670-4260-a77b-ae32de23fa0d\" (UID: \"09fb128c-0670-4260-a77b-ae32de23fa0d\") "
Oct 30 01:00:20.908159 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:20.908149    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c111db1f-7e6f-44b8-b211-1658aee31337-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c111db1f-7e6f-44b8-b211-1658aee31337-maas-api-service-ca\") pod \"c111db1f-7e6f-44b8-b211-1658aee31337\" (UID: \"c111db1f-7e6f-44b8-b211-1658aee31337\") "
Oct 30 01:00:20.908295 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:20.908278    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c111db1f-7e6f-44b8-b211-1658aee31337-kube-api-access-n2xg2\" (UniqueName: \"kubernetes.io/projected/c111db1f-7e6f-44b8-b211-1658aee31337-kube-api-access-n2xg2\") pod \"c111db1f-7e6f-44b8-b211-1658aee31337\" (UID: \"c111db1f-7e6f-44b8-b211-1658aee31337\") "
Oct 30 01:00:20.908459 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:20.908434    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09fb128c-0670-4260-a77b-ae32de23fa0d-maas-api-service-ca" pod "09fb128c-0670-4260-a77b-ae32de23fa0d" (UID: "09fb128c-0670-4260-a77b-ae32de23fa0d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:00:20.908554 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:20.908533    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c111db1f-7e6f-44b8-b211-1658aee31337-maas-api-service-ca" pod "c111db1f-7e6f-44b8-b211-1658aee31337" (UID: "c111db1f-7e6f-44b8-b211-1658aee31337"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:00:20.908955 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:20.908931    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/09fb128c-0670-4260-a77b-ae32de23fa0d-maas-api-service-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:00:20.909046 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:20.908958    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c111db1f-7e6f-44b8-b211-1658aee31337-maas-api-service-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:00:20.910195 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:20.910173    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09fb128c-0670-4260-a77b-ae32de23fa0d-kube-api-access-brqqq" pod "09fb128c-0670-4260-a77b-ae32de23fa0d" (UID: "09fb128c-0670-4260-a77b-ae32de23fa0d"). InnerVolumeSpecName "kube-api-access-brqqq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:00:20.910573 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:20.910551    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c111db1f-7e6f-44b8-b211-1658aee31337-kube-api-access-n2xg2" pod "c111db1f-7e6f-44b8-b211-1658aee31337" (UID: "c111db1f-7e6f-44b8-b211-1658aee31337"). InnerVolumeSpecName "kube-api-access-n2xg2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:00:21.010659 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:21.010627    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n2xg2\" (UniqueName: \"kubernetes.io/projected/c111db1f-7e6f-44b8-b211-1658aee31337-kube-api-access-n2xg2\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:00:21.010659 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:21.010655    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-brqqq\" (UniqueName: \"kubernetes.io/projected/09fb128c-0670-4260-a77b-ae32de23fa0d-kube-api-access-brqqq\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:00:21.469813 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:21.469768    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b3bdf9-29888700-pzr6b" event={"ID":"bd72bc53-220d-40cd-abb8-36bc71b66094","Type":"ContainerStarted","Data":"ee2daee41fa1c1d01a1db3593506d968865634dbc6d72162283d67176d365ac8"}
Oct 30 01:00:21.475162 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:21.475136    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1a11cf-29888700-d9s5c" event={"ID":"c111db1f-7e6f-44b8-b211-1658aee31337","Type":"ContainerDied","Data":"142521fcab562ea0b58b9231fdab8633931b469ee378fc9eafcf72a814912499"}
Oct 30 01:00:21.475280 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:21.475157    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1a11cf-29888700-d9s5c"
Oct 30 01:00:21.475280 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:21.475195    5002 scope.go:122] "RemoveContainer" containerID="9a858720da3de764d91b2029bc1d8a7c476b209a952d4a613ed60a6e45417586"
Oct 30 01:00:21.478871 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:21.478849    5002 generic.go:356] "Generic (PLEG): container finished" podID="4af343ba-bc4d-4ec3-92df-1534780d3089" containerID="3365c4917c725676bfe2bc923168fce4ad541db86703a6230cd2ead2d6235395" exitCode=60
Oct 30 01:00:21.479004 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:21.478935    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5f387c-29888700-vtvtq" event={"ID":"4af343ba-bc4d-4ec3-92df-1534780d3089","Type":"ContainerDied","Data":"3365c4917c725676bfe2bc923168fce4ad541db86703a6230cd2ead2d6235395"}
Oct 30 01:00:21.479306 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:21.479287    5002 scope.go:122] "RemoveContainer" containerID="3365c4917c725676bfe2bc923168fce4ad541db86703a6230cd2ead2d6235395"
Oct 30 01:00:21.479808 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:00:21.479779    5002 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-5f387c-29888700-vtvtq_odh-ai-gateway-infra(4af343ba-bc4d-4ec3-92df-1534780d3089)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5f387c-29888700-vtvtq" podUID="4af343ba-bc4d-4ec3-92df-1534780d3089"
Oct 30 01:00:21.481871 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:21.481848    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-52650f-29888700-8mvn6" event={"ID":"09fb128c-0670-4260-a77b-ae32de23fa0d","Type":"ContainerDied","Data":"a1bb82837f5663136ee41cba1c81976e1b0622b6dc75047cd199f96448ee7ddf"}
Oct 30 01:00:21.481958 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:21.481945    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-52650f-29888700-8mvn6"
Oct 30 01:00:21.487752 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:21.487729    5002 scope.go:122] "RemoveContainer" containerID="1323635381a3d79b74e0afced5180f6f4f2bcc65686aed8d50d6bb29559460d0"
Oct 30 01:00:21.490825 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:21.488610    5002 scope.go:122] "RemoveContainer" containerID="1c5c4fc069b26c0b4d51bf5a0fbd3c16dfa0e1f360ca6195142d58981827b7b6"
Oct 30 01:00:21.494670 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:21.494634    5002 scope.go:122] "RemoveContainer" containerID="70fa31a4dae40eb87d43802b774fae56d2a381fe759b25005c5751be178b8056"
Oct 30 01:00:21.519424 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:21.517034    5002 generic.go:356] "Generic (PLEG): container finished" podID="5fa9212f-5c4d-47a2-a855-2c18a807533e" containerID="a9abb13a5de9a91d2294f3feeed77547a291fc384876850dbc8e8b3ec59f7645" exitCode=60
Oct 30 01:00:21.519424 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:21.517126    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8e0f66-29888700-tnj7q" event={"ID":"5fa9212f-5c4d-47a2-a855-2c18a807533e","Type":"ContainerDied","Data":"a9abb13a5de9a91d2294f3feeed77547a291fc384876850dbc8e8b3ec59f7645"}
Oct 30 01:00:21.519424 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:21.517634    5002 scope.go:122] "RemoveContainer" containerID="a9abb13a5de9a91d2294f3feeed77547a291fc384876850dbc8e8b3ec59f7645"
Oct 30 01:00:21.519424 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:00:21.518084    5002 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-8e0f66-29888700-tnj7q_odh-ai-gateway-infra(5fa9212f-5c4d-47a2-a855-2c18a807533e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8e0f66-29888700-tnj7q" podUID="5fa9212f-5c4d-47a2-a855-2c18a807533e"
Oct 30 01:00:21.537980 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:21.537561    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-481ab7-29888700-8mt68" event={"ID":"b15c8d59-8957-4e66-8e15-998c8a33e690","Type":"ContainerStarted","Data":"91831523ffc822b0ecd7051a1b3f9a5235b5cedcf6703ddb00c1ba656a3b9633"}
Oct 30 01:00:21.543996 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:21.540442    5002 generic.go:356] "Generic (PLEG): container finished" podID="d5279cb1-29b0-494a-b3b3-527e8af641dc" containerID="42265725d58abfe459dc726587ca11c8def903b8e431398183267501da12d3e1" exitCode=60
Oct 30 01:00:21.543996 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:21.540543    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888700-k6pvc" event={"ID":"d5279cb1-29b0-494a-b3b3-527e8af641dc","Type":"ContainerDied","Data":"42265725d58abfe459dc726587ca11c8def903b8e431398183267501da12d3e1"}
Oct 30 01:00:21.560614 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:21.559093    5002 scope.go:122] "RemoveContainer" containerID="f7f7c860079a37739b55fcce44de65100f13f349f1863bb6ba96afec8fcfea51"
Oct 30 01:00:21.578106 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:21.578075    5002 scope.go:122] "RemoveContainer" containerID="42265725d58abfe459dc726587ca11c8def903b8e431398183267501da12d3e1"
Oct 30 01:00:21.579300 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:00:21.579267    5002 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29888700-k6pvc_odh-ai-gateway-infra(d5279cb1-29b0-494a-b3b3-527e8af641dc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888700-k6pvc" podUID="d5279cb1-29b0-494a-b3b3-527e8af641dc"
Oct 30 01:00:21.595863 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:21.595813    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8e0f66-29888700-tnj7q"]
Oct 30 01:00:21.606315 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:21.606258    5002 scope.go:122] "RemoveContainer" containerID="cdc6b212bb47ace68ace90e7eaed307569eb5a305ec64e50bafab9ca50204fbe"
Oct 30 01:00:21.634811 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:21.634785    5002 scope.go:122] "RemoveContainer" containerID="cf5baf6bf97a9f4e9b95b5efdd8d7180f6328f1e964d0b4cae74b92af25daaf7"
Oct 30 01:00:21.658426 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:21.658232    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1a11cf-29888700-d9s5c"]
Oct 30 01:00:21.664742 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:21.664704    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1a11cf-29888700-d9s5c"]
Oct 30 01:00:21.807212 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:21.807185    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-52650f-29888700-8mvn6"]
Oct 30 01:00:21.818916 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:21.818786    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-52650f-29888700-8mvn6"]
Oct 30 01:00:21.824316 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:21.824292    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888700-k6pvc"]
Oct 30 01:00:21.884546 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:21.883092    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-cce513-29888700-744n2"
Oct 30 01:00:21.936354 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:21.936323    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5f387c-29888700-vtvtq"]
Oct 30 01:00:21.940994 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:21.940960    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/743ebbf6-b4db-4774-b836-de5d7c716ca7-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/743ebbf6-b4db-4774-b836-de5d7c716ca7-maas-api-service-ca\") pod \"743ebbf6-b4db-4774-b836-de5d7c716ca7\" (UID: \"743ebbf6-b4db-4774-b836-de5d7c716ca7\") "
Oct 30 01:00:21.941716 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:21.941694    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/743ebbf6-b4db-4774-b836-de5d7c716ca7-kube-api-access-gw4pf\" (UniqueName: \"kubernetes.io/projected/743ebbf6-b4db-4774-b836-de5d7c716ca7-kube-api-access-gw4pf\") pod \"743ebbf6-b4db-4774-b836-de5d7c716ca7\" (UID: \"743ebbf6-b4db-4774-b836-de5d7c716ca7\") "
Oct 30 01:00:21.942256 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:21.941451    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/743ebbf6-b4db-4774-b836-de5d7c716ca7-maas-api-service-ca" pod "743ebbf6-b4db-4774-b836-de5d7c716ca7" (UID: "743ebbf6-b4db-4774-b836-de5d7c716ca7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:00:21.944194 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:21.944173    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/743ebbf6-b4db-4774-b836-de5d7c716ca7-maas-api-service-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:00:21.947982 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:21.947960    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/743ebbf6-b4db-4774-b836-de5d7c716ca7-kube-api-access-gw4pf" pod "743ebbf6-b4db-4774-b836-de5d7c716ca7" (UID: "743ebbf6-b4db-4774-b836-de5d7c716ca7"). InnerVolumeSpecName "kube-api-access-gw4pf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:00:21.964979 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:21.964954    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6f4fba-29888700-zvdkx"
Oct 30 01:00:21.970184 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:21.970164    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-bab19a-29888700-f7tvc"
Oct 30 01:00:21.973632 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:21.973609    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8d2a70-29888700-jlmvv"
Oct 30 01:00:22.013841 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:22.013687    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-481ab7-29888700-8mt68" podStartSLOduration=22.013605488 podStartE2EDuration="22.013605488s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:00:22.010917908 +0000 UTC m=+1717.627350809" watchObservedRunningTime="2026-10-30 01:00:22.013605488 +0000 UTC m=+1717.630038368"
Oct 30 01:00:22.045328 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:22.045297    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d22fc624-e367-49a0-ba47-9a5dcabdde1f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d22fc624-e367-49a0-ba47-9a5dcabdde1f-maas-api-service-ca\") pod \"d22fc624-e367-49a0-ba47-9a5dcabdde1f\" (UID: \"d22fc624-e367-49a0-ba47-9a5dcabdde1f\") "
Oct 30 01:00:22.045574 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:22.045371    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6302c577-afc3-4632-a383-cb1fd8159992-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6302c577-afc3-4632-a383-cb1fd8159992-maas-api-service-ca\") pod \"6302c577-afc3-4632-a383-cb1fd8159992\" (UID: \"6302c577-afc3-4632-a383-cb1fd8159992\") "
Oct 30 01:00:22.045574 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:22.045490    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d22fc624-e367-49a0-ba47-9a5dcabdde1f-kube-api-access-cqn4x\" (UniqueName: \"kubernetes.io/projected/d22fc624-e367-49a0-ba47-9a5dcabdde1f-kube-api-access-cqn4x\") pod \"d22fc624-e367-49a0-ba47-9a5dcabdde1f\" (UID: \"d22fc624-e367-49a0-ba47-9a5dcabdde1f\") "
Oct 30 01:00:22.045751 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:22.045678    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6c630bd-ac41-4607-8ed9-5f68cf00be38-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6c630bd-ac41-4607-8ed9-5f68cf00be38-maas-api-service-ca\") pod \"b6c630bd-ac41-4607-8ed9-5f68cf00be38\" (UID: \"b6c630bd-ac41-4607-8ed9-5f68cf00be38\") "
Oct 30 01:00:22.045816 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:22.045775    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6302c577-afc3-4632-a383-cb1fd8159992-maas-api-service-ca" pod "6302c577-afc3-4632-a383-cb1fd8159992" (UID: "6302c577-afc3-4632-a383-cb1fd8159992"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:00:22.045880 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:22.045819    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d22fc624-e367-49a0-ba47-9a5dcabdde1f-maas-api-service-ca" pod "d22fc624-e367-49a0-ba47-9a5dcabdde1f" (UID: "d22fc624-e367-49a0-ba47-9a5dcabdde1f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:00:22.046070 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:22.046045    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6c630bd-ac41-4607-8ed9-5f68cf00be38-maas-api-service-ca" pod "b6c630bd-ac41-4607-8ed9-5f68cf00be38" (UID: "b6c630bd-ac41-4607-8ed9-5f68cf00be38"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:00:22.046260 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:22.046240    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6302c577-afc3-4632-a383-cb1fd8159992-kube-api-access-st6zv\" (UniqueName: \"kubernetes.io/projected/6302c577-afc3-4632-a383-cb1fd8159992-kube-api-access-st6zv\") pod \"6302c577-afc3-4632-a383-cb1fd8159992\" (UID: \"6302c577-afc3-4632-a383-cb1fd8159992\") "
Oct 30 01:00:22.046792 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:22.046771    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6c630bd-ac41-4607-8ed9-5f68cf00be38-kube-api-access-qp985\" (UniqueName: \"kubernetes.io/projected/b6c630bd-ac41-4607-8ed9-5f68cf00be38-kube-api-access-qp985\") pod \"b6c630bd-ac41-4607-8ed9-5f68cf00be38\" (UID: \"b6c630bd-ac41-4607-8ed9-5f68cf00be38\") "
Oct 30 01:00:22.048066 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:22.048044    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6c630bd-ac41-4607-8ed9-5f68cf00be38-maas-api-service-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:00:22.048197 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:22.048179    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gw4pf\" (UniqueName: \"kubernetes.io/projected/743ebbf6-b4db-4774-b836-de5d7c716ca7-kube-api-access-gw4pf\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:00:22.048258 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:22.048199    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d22fc624-e367-49a0-ba47-9a5dcabdde1f-maas-api-service-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:00:22.048258 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:22.048213    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6302c577-afc3-4632-a383-cb1fd8159992-maas-api-service-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:00:22.048692 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:22.048667    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d22fc624-e367-49a0-ba47-9a5dcabdde1f-kube-api-access-cqn4x" pod "d22fc624-e367-49a0-ba47-9a5dcabdde1f" (UID: "d22fc624-e367-49a0-ba47-9a5dcabdde1f"). InnerVolumeSpecName "kube-api-access-cqn4x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:00:22.049340 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:22.049320    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6c630bd-ac41-4607-8ed9-5f68cf00be38-kube-api-access-qp985" pod "b6c630bd-ac41-4607-8ed9-5f68cf00be38" (UID: "b6c630bd-ac41-4607-8ed9-5f68cf00be38"). InnerVolumeSpecName "kube-api-access-qp985". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:00:22.050781 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:22.050744    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6302c577-afc3-4632-a383-cb1fd8159992-kube-api-access-st6zv" pod "6302c577-afc3-4632-a383-cb1fd8159992" (UID: "6302c577-afc3-4632-a383-cb1fd8159992"). InnerVolumeSpecName "kube-api-access-st6zv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:00:22.150408 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:22.150294    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cqn4x\" (UniqueName: \"kubernetes.io/projected/d22fc624-e367-49a0-ba47-9a5dcabdde1f-kube-api-access-cqn4x\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:00:22.150408 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:22.150333    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-st6zv\" (UniqueName: \"kubernetes.io/projected/6302c577-afc3-4632-a383-cb1fd8159992-kube-api-access-st6zv\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:00:22.150408 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:22.150348    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qp985\" (UniqueName: \"kubernetes.io/projected/b6c630bd-ac41-4607-8ed9-5f68cf00be38-kube-api-access-qp985\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:00:22.488451 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:22.488095    5002 scope.go:122] "RemoveContainer" containerID="1f4025cfa5689b2203a7a9e9de122debe44b072e0f2563a0f56343b9f15bb934"
Oct 30 01:00:22.490671 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:22.490620    5002 scope.go:122] "RemoveContainer" containerID="386fa1d5a2c4d3a43aa56a4e7887c0c518d65a7993cf95921b6bc0a401d6e2f8"
Oct 30 01:00:22.510074 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:22.509480    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09fb128c-0670-4260-a77b-ae32de23fa0d" path="/var/lib/kubelet/pods/09fb128c-0670-4260-a77b-ae32de23fa0d/volumes"
Oct 30 01:00:22.510200 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:22.510142    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c111db1f-7e6f-44b8-b211-1658aee31337" path="/var/lib/kubelet/pods/c111db1f-7e6f-44b8-b211-1658aee31337/volumes"
Oct 30 01:00:22.532515 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:22.532486    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b3bdf9-29888700-pzr6b"]
Oct 30 01:00:22.560695 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:22.560666    5002 generic.go:356] "Generic (PLEG): container finished" podID="b15c8d59-8957-4e66-8e15-998c8a33e690" containerID="91831523ffc822b0ecd7051a1b3f9a5235b5cedcf6703ddb00c1ba656a3b9633" exitCode=60
Oct 30 01:00:22.560818 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:22.560737    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-481ab7-29888700-8mt68" event={"ID":"b15c8d59-8957-4e66-8e15-998c8a33e690","Type":"ContainerDied","Data":"91831523ffc822b0ecd7051a1b3f9a5235b5cedcf6703ddb00c1ba656a3b9633"}
Oct 30 01:00:22.560818 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:22.560794    5002 scope.go:122] "RemoveContainer" containerID="3d3fdb0735f79018c13a82512cd7420f9099a1138403ae93a519dcf547e8deee"
Oct 30 01:00:22.561128 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:22.561109    5002 scope.go:122] "RemoveContainer" containerID="91831523ffc822b0ecd7051a1b3f9a5235b5cedcf6703ddb00c1ba656a3b9633"
Oct 30 01:00:22.561677 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:00:22.561647    5002 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-481ab7-29888700-8mt68_odh-ai-gateway-infra(b15c8d59-8957-4e66-8e15-998c8a33e690)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-481ab7-29888700-8mt68" podUID="b15c8d59-8957-4e66-8e15-998c8a33e690"
Oct 30 01:00:22.564723 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:22.564701    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6f4fba-29888700-zvdkx" event={"ID":"6302c577-afc3-4632-a383-cb1fd8159992","Type":"ContainerDied","Data":"4a8d23dd4f36183583e7ca7ac1a6c372b5e478a9a3d7b54f1e3f521ca34d2227"}
Oct 30 01:00:22.564723 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:22.564717    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6f4fba-29888700-zvdkx"
Oct 30 01:00:22.566964 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:22.566942    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-5007e1-29888700-dvm8j" event={"ID":"38a379b2-37b6-4185-8319-2e62dcec38c8","Type":"ContainerStarted","Data":"ddad04d1e37fb7ab6d902cdf2bfcba87aab46d3cda03d34867111a5f4ac9056c"}
Oct 30 01:00:22.569462 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:22.569444    5002 generic.go:356] "Generic (PLEG): container finished" podID="bd72bc53-220d-40cd-abb8-36bc71b66094" containerID="ee2daee41fa1c1d01a1db3593506d968865634dbc6d72162283d67176d365ac8" exitCode=60
Oct 30 01:00:22.569637 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:22.569534    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b3bdf9-29888700-pzr6b" event={"ID":"bd72bc53-220d-40cd-abb8-36bc71b66094","Type":"ContainerDied","Data":"ee2daee41fa1c1d01a1db3593506d968865634dbc6d72162283d67176d365ac8"}
Oct 30 01:00:22.572081 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:22.571822    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-cce513-29888700-744n2" event={"ID":"743ebbf6-b4db-4774-b836-de5d7c716ca7","Type":"ContainerDied","Data":"93731423201ab10194ea8664832f92c54c4c039bb16056f112d14d83fdb611a4"}
Oct 30 01:00:22.572081 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:22.571863    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-cce513-29888700-744n2"
Oct 30 01:00:22.573255 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:22.573238    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8d2a70-29888700-jlmvv"
Oct 30 01:00:22.573476 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:22.573416    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8d2a70-29888700-jlmvv" event={"ID":"d22fc624-e367-49a0-ba47-9a5dcabdde1f","Type":"ContainerDied","Data":"09eb8897e3e1223c880ca52859273b20820261f9857fe6d90588ce957326b359"}
Oct 30 01:00:22.583861 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:22.580893    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-29be73-29888700-gvwp4" event={"ID":"86bc1f67-d9db-4d33-8bfc-e57494705d80","Type":"ContainerStarted","Data":"2b1c98e49cfafa0b862b5ae3a94ed4b0706ae15ef5be8c21be5d64155a0b7ef0"}
Oct 30 01:00:22.583861 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:22.582179    5002 scope.go:122] "RemoveContainer" containerID="2b1c98e49cfafa0b862b5ae3a94ed4b0706ae15ef5be8c21be5d64155a0b7ef0"
Oct 30 01:00:22.583861 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:00:22.583051    5002 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-29be73-29888700-gvwp4_odh-ai-gateway-infra(86bc1f67-d9db-4d33-8bfc-e57494705d80)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-29be73-29888700-gvwp4" podUID="86bc1f67-d9db-4d33-8bfc-e57494705d80"
Oct 30 01:00:22.588180 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:22.586434    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-bab19a-29888700-f7tvc" event={"ID":"b6c630bd-ac41-4607-8ed9-5f68cf00be38","Type":"ContainerDied","Data":"44c5a3ceda959040a702d560da55c8274a12f3d852664a3f4cda1030ce1e4518"}
Oct 30 01:00:22.588180 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:22.586550    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-bab19a-29888700-f7tvc"
Oct 30 01:00:22.647613 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:22.646587    5002 scope.go:122] "RemoveContainer" containerID="ba1ec04a3a05167d1560d31009c209b3722d400ef52046b07e782b1d24d44f91"
Oct 30 01:00:22.711349 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:22.711315    5002 scope.go:122] "RemoveContainer" containerID="f42d54614481c6397b7e65761d0312be90c869f167d5624324fe55f0d754bfa1"
Oct 30 01:00:22.755082 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:22.755055    5002 scope.go:122] "RemoveContainer" containerID="ded5c17024715fcaaffefe84542116d7ff5e57cec48cbcd9853525f58e1ee8b8"
Oct 30 01:00:22.764892 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:22.764861    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-29be73-29888700-gvwp4"]
Oct 30 01:00:22.863510 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:22.863487    5002 scope.go:122] "RemoveContainer" containerID="6ceaac9ce998e813e0a4e98f3a3dc1a1480ac3c07200b11fadb365dc6f42b043"
Oct 30 01:00:22.878000 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:22.877972    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888700-k6pvc"
Oct 30 01:00:22.880149 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:22.880123    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b3bdf9-29888700-pzr6b"
Oct 30 01:00:22.898402 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:22.898363    5002 scope.go:122] "RemoveContainer" containerID="02ad2188c2ee4d393177e35b8e25369a5ab0de1dea76867ea10b85ac113e253b"
Oct 30 01:00:22.958289 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:22.958244    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5f387c-29888700-vtvtq"
Oct 30 01:00:22.961922 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:22.961896    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8e0f66-29888700-tnj7q"
Oct 30 01:00:22.984970 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:22.984942    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5279cb1-29b0-494a-b3b3-527e8af641dc-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5279cb1-29b0-494a-b3b3-527e8af641dc-maas-api-service-ca\") pod \"d5279cb1-29b0-494a-b3b3-527e8af641dc\" (UID: \"d5279cb1-29b0-494a-b3b3-527e8af641dc\") "
Oct 30 01:00:22.985106 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:22.985012    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd72bc53-220d-40cd-abb8-36bc71b66094-kube-api-access-qrkbd\" (UniqueName: \"kubernetes.io/projected/bd72bc53-220d-40cd-abb8-36bc71b66094-kube-api-access-qrkbd\") pod \"bd72bc53-220d-40cd-abb8-36bc71b66094\" (UID: \"bd72bc53-220d-40cd-abb8-36bc71b66094\") "
Oct 30 01:00:22.985186 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:22.985120    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd72bc53-220d-40cd-abb8-36bc71b66094-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd72bc53-220d-40cd-abb8-36bc71b66094-maas-api-service-ca\") pod \"bd72bc53-220d-40cd-abb8-36bc71b66094\" (UID: \"bd72bc53-220d-40cd-abb8-36bc71b66094\") "
Oct 30 01:00:22.985242 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:22.985185    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5279cb1-29b0-494a-b3b3-527e8af641dc-kube-api-access-h4sfq\" (UniqueName: \"kubernetes.io/projected/d5279cb1-29b0-494a-b3b3-527e8af641dc-kube-api-access-h4sfq\") pod \"d5279cb1-29b0-494a-b3b3-527e8af641dc\" (UID: \"d5279cb1-29b0-494a-b3b3-527e8af641dc\") "
Oct 30 01:00:22.985484 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:22.985431    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5279cb1-29b0-494a-b3b3-527e8af641dc-maas-api-service-ca" pod "d5279cb1-29b0-494a-b3b3-527e8af641dc" (UID: "d5279cb1-29b0-494a-b3b3-527e8af641dc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:00:22.985603 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:22.985481    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd72bc53-220d-40cd-abb8-36bc71b66094-maas-api-service-ca" pod "bd72bc53-220d-40cd-abb8-36bc71b66094" (UID: "bd72bc53-220d-40cd-abb8-36bc71b66094"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:00:22.986221 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:22.986198    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5279cb1-29b0-494a-b3b3-527e8af641dc-maas-api-service-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:00:22.986298 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:22.986229    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd72bc53-220d-40cd-abb8-36bc71b66094-maas-api-service-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:00:22.987607 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:22.987586    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd72bc53-220d-40cd-abb8-36bc71b66094-kube-api-access-qrkbd" pod "bd72bc53-220d-40cd-abb8-36bc71b66094" (UID: "bd72bc53-220d-40cd-abb8-36bc71b66094"). InnerVolumeSpecName "kube-api-access-qrkbd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:00:22.987973 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:22.987956    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5279cb1-29b0-494a-b3b3-527e8af641dc-kube-api-access-h4sfq" pod "d5279cb1-29b0-494a-b3b3-527e8af641dc" (UID: "d5279cb1-29b0-494a-b3b3-527e8af641dc"). InnerVolumeSpecName "kube-api-access-h4sfq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:00:23.018732 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:23.018629    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-481ab7-29888700-8mt68"]
Oct 30 01:00:23.047433 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:23.047355    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-bab19a-29888700-f7tvc"]
Oct 30 01:00:23.052807 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:23.052770    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-bab19a-29888700-f7tvc"]
Oct 30 01:00:23.086882 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:23.086845    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4af343ba-bc4d-4ec3-92df-1534780d3089-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4af343ba-bc4d-4ec3-92df-1534780d3089-maas-api-service-ca\") pod \"4af343ba-bc4d-4ec3-92df-1534780d3089\" (UID: \"4af343ba-bc4d-4ec3-92df-1534780d3089\") "
Oct 30 01:00:23.087042 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:23.087028    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5fa9212f-5c4d-47a2-a855-2c18a807533e-kube-api-access-jv2gh\" (UniqueName: \"kubernetes.io/projected/5fa9212f-5c4d-47a2-a855-2c18a807533e-kube-api-access-jv2gh\") pod \"5fa9212f-5c4d-47a2-a855-2c18a807533e\" (UID: \"5fa9212f-5c4d-47a2-a855-2c18a807533e\") "
Oct 30 01:00:23.087130 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:23.087096    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5fa9212f-5c4d-47a2-a855-2c18a807533e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fa9212f-5c4d-47a2-a855-2c18a807533e-maas-api-service-ca\") pod \"5fa9212f-5c4d-47a2-a855-2c18a807533e\" (UID: \"5fa9212f-5c4d-47a2-a855-2c18a807533e\") "
Oct 30 01:00:23.087196 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:23.087133    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4af343ba-bc4d-4ec3-92df-1534780d3089-kube-api-access-79jrw\" (UniqueName: \"kubernetes.io/projected/4af343ba-bc4d-4ec3-92df-1534780d3089-kube-api-access-79jrw\") pod \"4af343ba-bc4d-4ec3-92df-1534780d3089\" (UID: \"4af343ba-bc4d-4ec3-92df-1534780d3089\") "
Oct 30 01:00:23.087254 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:23.087182    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4af343ba-bc4d-4ec3-92df-1534780d3089-maas-api-service-ca" pod "4af343ba-bc4d-4ec3-92df-1534780d3089" (UID: "4af343ba-bc4d-4ec3-92df-1534780d3089"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:00:23.087638 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:23.087610    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5fa9212f-5c4d-47a2-a855-2c18a807533e-maas-api-service-ca" pod "5fa9212f-5c4d-47a2-a855-2c18a807533e" (UID: "5fa9212f-5c4d-47a2-a855-2c18a807533e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:00:23.087803 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:23.087778    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6f4fba-29888700-zvdkx"]
Oct 30 01:00:23.087905 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:23.087882    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fa9212f-5c4d-47a2-a855-2c18a807533e-maas-api-service-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:00:23.087965 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:23.087913    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qrkbd\" (UniqueName: \"kubernetes.io/projected/bd72bc53-220d-40cd-abb8-36bc71b66094-kube-api-access-qrkbd\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:00:23.087965 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:23.087931    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h4sfq\" (UniqueName: \"kubernetes.io/projected/d5279cb1-29b0-494a-b3b3-527e8af641dc-kube-api-access-h4sfq\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:00:23.087965 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:23.087949    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4af343ba-bc4d-4ec3-92df-1534780d3089-maas-api-service-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:00:23.089471 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:23.089446    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fa9212f-5c4d-47a2-a855-2c18a807533e-kube-api-access-jv2gh" pod "5fa9212f-5c4d-47a2-a855-2c18a807533e" (UID: "5fa9212f-5c4d-47a2-a855-2c18a807533e"). InnerVolumeSpecName "kube-api-access-jv2gh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:00:23.089675 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:23.089652    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4af343ba-bc4d-4ec3-92df-1534780d3089-kube-api-access-79jrw" pod "4af343ba-bc4d-4ec3-92df-1534780d3089" (UID: "4af343ba-bc4d-4ec3-92df-1534780d3089"). InnerVolumeSpecName "kube-api-access-79jrw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:00:23.092541 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:23.092519    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6f4fba-29888700-zvdkx"]
Oct 30 01:00:23.130057 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:23.130019    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8d2a70-29888700-jlmvv"]
Oct 30 01:00:23.146401 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:23.136639    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8d2a70-29888700-jlmvv"]
Oct 30 01:00:23.190987 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:23.190946    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jv2gh\" (UniqueName: \"kubernetes.io/projected/5fa9212f-5c4d-47a2-a855-2c18a807533e-kube-api-access-jv2gh\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:00:23.190987 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:23.190982    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79jrw\" (UniqueName: \"kubernetes.io/projected/4af343ba-bc4d-4ec3-92df-1534780d3089-kube-api-access-79jrw\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:00:23.266282 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:23.266239    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-cce513-29888700-744n2"]
Oct 30 01:00:23.271206 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:23.271124    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-cce513-29888700-744n2"]
Oct 30 01:00:23.604291 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:23.604256    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8e0f66-29888700-tnj7q"
Oct 30 01:00:23.604773 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:23.604251    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8e0f66-29888700-tnj7q" event={"ID":"5fa9212f-5c4d-47a2-a855-2c18a807533e","Type":"ContainerDied","Data":"643b6f3b817905b70040447dc85d07aaf97d0845e7234d7ec03a1ebb9f6666a1"}
Oct 30 01:00:23.604773 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:23.604434    5002 scope.go:122] "RemoveContainer" containerID="a9abb13a5de9a91d2294f3feeed77547a291fc384876850dbc8e8b3ec59f7645"
Oct 30 01:00:23.608401 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:23.608319    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888700-k6pvc" event={"ID":"d5279cb1-29b0-494a-b3b3-527e8af641dc","Type":"ContainerDied","Data":"2b32619ef3a63ad753c8612623a8fc61f2e9d29aebee0628a89d053a2c3e35e0"}
Oct 30 01:00:23.608401 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:23.608358    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888700-k6pvc"
Oct 30 01:00:23.614940 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:23.614913    5002 generic.go:356] "Generic (PLEG): container finished" podID="38a379b2-37b6-4185-8319-2e62dcec38c8" containerID="ddad04d1e37fb7ab6d902cdf2bfcba87aab46d3cda03d34867111a5f4ac9056c" exitCode=60
Oct 30 01:00:23.615139 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:23.614965    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-5007e1-29888700-dvm8j" event={"ID":"38a379b2-37b6-4185-8319-2e62dcec38c8","Type":"ContainerDied","Data":"ddad04d1e37fb7ab6d902cdf2bfcba87aab46d3cda03d34867111a5f4ac9056c"}
Oct 30 01:00:23.615833 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:23.615814    5002 scope.go:122] "RemoveContainer" containerID="ddad04d1e37fb7ab6d902cdf2bfcba87aab46d3cda03d34867111a5f4ac9056c"
Oct 30 01:00:23.616315 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:00:23.616280    5002 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-5007e1-29888700-dvm8j_odh-ai-gateway-infra(38a379b2-37b6-4185-8319-2e62dcec38c8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-5007e1-29888700-dvm8j" podUID="38a379b2-37b6-4185-8319-2e62dcec38c8"
Oct 30 01:00:23.618853 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:23.618830    5002 scope.go:122] "RemoveContainer" containerID="42265725d58abfe459dc726587ca11c8def903b8e431398183267501da12d3e1"
Oct 30 01:00:23.619615 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:23.619031    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b3bdf9-29888700-pzr6b" event={"ID":"bd72bc53-220d-40cd-abb8-36bc71b66094","Type":"ContainerDied","Data":"735dfe0534ef8e9b6e6a2607faa556c025a3be9234b1edd214910f13eee8b2a1"}
Oct 30 01:00:23.619615 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:23.619143    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b3bdf9-29888700-pzr6b"
Oct 30 01:00:23.624132 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:23.624110    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ca5363-29888700-trbnd" event={"ID":"5a907e40-190d-44fe-89b8-a51e621c04db","Type":"ContainerStarted","Data":"5fac5d51930ab47edea15a76d372467b3641fb092006e5eec63567519dcec27c"}
Oct 30 01:00:23.628978 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:23.628948    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5f387c-29888700-vtvtq" event={"ID":"4af343ba-bc4d-4ec3-92df-1534780d3089","Type":"ContainerDied","Data":"d1332ba06dbf8ad951d7b5f26be348b2d3b05ec9905a3f16c7d1b103a4b0433c"}
Oct 30 01:00:23.629081 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:23.629000    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5f387c-29888700-vtvtq"
Oct 30 01:00:23.633539 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:23.633427    5002 generic.go:356] "Generic (PLEG): container finished" podID="86bc1f67-d9db-4d33-8bfc-e57494705d80" containerID="2b1c98e49cfafa0b862b5ae3a94ed4b0706ae15ef5be8c21be5d64155a0b7ef0" exitCode=60
Oct 30 01:00:23.634634 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:23.634053    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-29be73-29888700-gvwp4" event={"ID":"86bc1f67-d9db-4d33-8bfc-e57494705d80","Type":"ContainerDied","Data":"2b1c98e49cfafa0b862b5ae3a94ed4b0706ae15ef5be8c21be5d64155a0b7ef0"}
Oct 30 01:00:23.635340 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:23.635328    5002 scope.go:122] "RemoveContainer" containerID="1323635381a3d79b74e0afced5180f6f4f2bcc65686aed8d50d6bb29559460d0"
Oct 30 01:00:23.639208 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:23.638921    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-11f521-29888700-khq6f" event={"ID":"7f443227-6ffc-40f4-80aa-1485ae41aa02","Type":"ContainerStarted","Data":"3ebf07cd05cbfab22fcec5063c943e4b9a54669db9b312012f54c00d13e52dfe"}
Oct 30 01:00:23.662648 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:23.662069    5002 scope.go:122] "RemoveContainer" containerID="ee2daee41fa1c1d01a1db3593506d968865634dbc6d72162283d67176d365ac8"
Oct 30 01:00:23.663997 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:23.663937    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8e0f66-29888700-tnj7q"]
Oct 30 01:00:23.670548 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:23.670523    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8e0f66-29888700-tnj7q"]
Oct 30 01:00:23.676561 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:23.676541    5002 scope.go:122] "RemoveContainer" containerID="3365c4917c725676bfe2bc923168fce4ad541db86703a6230cd2ead2d6235395"
Oct 30 01:00:23.690686 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:23.690664    5002 scope.go:122] "RemoveContainer" containerID="1c5c4fc069b26c0b4d51bf5a0fbd3c16dfa0e1f360ca6195142d58981827b7b6"
Oct 30 01:00:23.859522 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:23.859483    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888700-k6pvc"]
Oct 30 01:00:23.863697 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:23.863670    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888700-k6pvc"]
Oct 30 01:00:23.873895 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:23.873872    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-5007e1-29888700-dvm8j"]
Oct 30 01:00:23.981496 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:23.981281    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-481ab7-29888700-8mt68"
Oct 30 01:00:23.984178 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:23.984155    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-29be73-29888700-gvwp4"
Oct 30 01:00:23.984668 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:23.984651    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5f387c-29888700-vtvtq"]
Oct 30 01:00:23.988551 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:23.988527    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5f387c-29888700-vtvtq"]
Oct 30 01:00:24.003161 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:24.003131    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b3bdf9-29888700-pzr6b"]
Oct 30 01:00:24.008841 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:24.008813    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b3bdf9-29888700-pzr6b"]
Oct 30 01:00:24.142594 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:24.142501    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b15c8d59-8957-4e66-8e15-998c8a33e690-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b15c8d59-8957-4e66-8e15-998c8a33e690-maas-api-service-ca\") pod \"b15c8d59-8957-4e66-8e15-998c8a33e690\" (UID: \"b15c8d59-8957-4e66-8e15-998c8a33e690\") "
Oct 30 01:00:24.142594 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:24.142566    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86bc1f67-d9db-4d33-8bfc-e57494705d80-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86bc1f67-d9db-4d33-8bfc-e57494705d80-maas-api-service-ca\") pod \"86bc1f67-d9db-4d33-8bfc-e57494705d80\" (UID: \"86bc1f67-d9db-4d33-8bfc-e57494705d80\") "
Oct 30 01:00:24.142819 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:24.142746    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b15c8d59-8957-4e66-8e15-998c8a33e690-kube-api-access-tt8fq\" (UniqueName: \"kubernetes.io/projected/b15c8d59-8957-4e66-8e15-998c8a33e690-kube-api-access-tt8fq\") pod \"b15c8d59-8957-4e66-8e15-998c8a33e690\" (UID: \"b15c8d59-8957-4e66-8e15-998c8a33e690\") "
Oct 30 01:00:24.142880 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:24.142853    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b15c8d59-8957-4e66-8e15-998c8a33e690-maas-api-service-ca" pod "b15c8d59-8957-4e66-8e15-998c8a33e690" (UID: "b15c8d59-8957-4e66-8e15-998c8a33e690"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:00:24.142965 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:24.142947    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86bc1f67-d9db-4d33-8bfc-e57494705d80-kube-api-access-7l5xn\" (UniqueName: \"kubernetes.io/projected/86bc1f67-d9db-4d33-8bfc-e57494705d80-kube-api-access-7l5xn\") pod \"86bc1f67-d9db-4d33-8bfc-e57494705d80\" (UID: \"86bc1f67-d9db-4d33-8bfc-e57494705d80\") "
Oct 30 01:00:24.143142 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:24.143125    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86bc1f67-d9db-4d33-8bfc-e57494705d80-maas-api-service-ca" pod "86bc1f67-d9db-4d33-8bfc-e57494705d80" (UID: "86bc1f67-d9db-4d33-8bfc-e57494705d80"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:00:24.152510 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:24.145774    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86bc1f67-d9db-4d33-8bfc-e57494705d80-kube-api-access-7l5xn" pod "86bc1f67-d9db-4d33-8bfc-e57494705d80" (UID: "86bc1f67-d9db-4d33-8bfc-e57494705d80"). InnerVolumeSpecName "kube-api-access-7l5xn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:00:24.153092 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:24.153065    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b15c8d59-8957-4e66-8e15-998c8a33e690-kube-api-access-tt8fq" pod "b15c8d59-8957-4e66-8e15-998c8a33e690" (UID: "b15c8d59-8957-4e66-8e15-998c8a33e690"). InnerVolumeSpecName "kube-api-access-tt8fq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:00:24.163830 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:24.163794    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b15c8d59-8957-4e66-8e15-998c8a33e690-maas-api-service-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:00:24.163830 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:24.163829    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86bc1f67-d9db-4d33-8bfc-e57494705d80-maas-api-service-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:00:24.164008 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:24.163847    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tt8fq\" (UniqueName: \"kubernetes.io/projected/b15c8d59-8957-4e66-8e15-998c8a33e690-kube-api-access-tt8fq\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:00:24.164008 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:24.163861    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7l5xn\" (UniqueName: \"kubernetes.io/projected/86bc1f67-d9db-4d33-8bfc-e57494705d80-kube-api-access-7l5xn\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:00:24.504294 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:24.504255    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4af343ba-bc4d-4ec3-92df-1534780d3089" path="/var/lib/kubelet/pods/4af343ba-bc4d-4ec3-92df-1534780d3089/volumes"
Oct 30 01:00:24.505319 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:24.505294    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5fa9212f-5c4d-47a2-a855-2c18a807533e" path="/var/lib/kubelet/pods/5fa9212f-5c4d-47a2-a855-2c18a807533e/volumes"
Oct 30 01:00:24.506159 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:24.506138    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6302c577-afc3-4632-a383-cb1fd8159992" path="/var/lib/kubelet/pods/6302c577-afc3-4632-a383-cb1fd8159992/volumes"
Oct 30 01:00:24.507394 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:24.507358    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="743ebbf6-b4db-4774-b836-de5d7c716ca7" path="/var/lib/kubelet/pods/743ebbf6-b4db-4774-b836-de5d7c716ca7/volumes"
Oct 30 01:00:24.508263 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:24.508228    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6c630bd-ac41-4607-8ed9-5f68cf00be38" path="/var/lib/kubelet/pods/b6c630bd-ac41-4607-8ed9-5f68cf00be38/volumes"
Oct 30 01:00:24.508979 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:24.508954    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd72bc53-220d-40cd-abb8-36bc71b66094" path="/var/lib/kubelet/pods/bd72bc53-220d-40cd-abb8-36bc71b66094/volumes"
Oct 30 01:00:24.509784 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:24.509759    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d22fc624-e367-49a0-ba47-9a5dcabdde1f" path="/var/lib/kubelet/pods/d22fc624-e367-49a0-ba47-9a5dcabdde1f/volumes"
Oct 30 01:00:24.510704 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:24.510682    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5279cb1-29b0-494a-b3b3-527e8af641dc" path="/var/lib/kubelet/pods/d5279cb1-29b0-494a-b3b3-527e8af641dc/volumes"
Oct 30 01:00:24.650631 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:24.650599    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-29be73-29888700-gvwp4"
Oct 30 01:00:24.651045 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:24.650597    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-29be73-29888700-gvwp4" event={"ID":"86bc1f67-d9db-4d33-8bfc-e57494705d80","Type":"ContainerDied","Data":"4072e33018d75cc2ff780267df1f19570810527919cca1d70810556f99fcad80"}
Oct 30 01:00:24.651045 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:24.650759    5002 scope.go:122] "RemoveContainer" containerID="2b1c98e49cfafa0b862b5ae3a94ed4b0706ae15ef5be8c21be5d64155a0b7ef0"
Oct 30 01:00:24.653070 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:24.653042    5002 generic.go:356] "Generic (PLEG): container finished" podID="7f443227-6ffc-40f4-80aa-1485ae41aa02" containerID="3ebf07cd05cbfab22fcec5063c943e4b9a54669db9b312012f54c00d13e52dfe" exitCode=60
Oct 30 01:00:24.653196 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:24.653179    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-11f521-29888700-khq6f" event={"ID":"7f443227-6ffc-40f4-80aa-1485ae41aa02","Type":"ContainerDied","Data":"3ebf07cd05cbfab22fcec5063c943e4b9a54669db9b312012f54c00d13e52dfe"}
Oct 30 01:00:24.653711 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:24.653689    5002 scope.go:122] "RemoveContainer" containerID="3ebf07cd05cbfab22fcec5063c943e4b9a54669db9b312012f54c00d13e52dfe"
Oct 30 01:00:24.654391 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:00:24.654347    5002 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-11f521-29888700-khq6f_odh-ai-gateway-infra(7f443227-6ffc-40f4-80aa-1485ae41aa02)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-11f521-29888700-khq6f" podUID="7f443227-6ffc-40f4-80aa-1485ae41aa02"
Oct 30 01:00:24.656586 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:24.656566    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-481ab7-29888700-8mt68"
Oct 30 01:00:24.656653 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:24.656585    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-481ab7-29888700-8mt68" event={"ID":"b15c8d59-8957-4e66-8e15-998c8a33e690","Type":"ContainerDied","Data":"581327734d6adbc745024694636fbc74dfc6e7dce3652db02111447ed747b59a"}
Oct 30 01:00:24.662757 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:24.662738    5002 scope.go:122] "RemoveContainer" containerID="1f4025cfa5689b2203a7a9e9de122debe44b072e0f2563a0f56343b9f15bb934"
Oct 30 01:00:24.665772 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:24.665750    5002 generic.go:356] "Generic (PLEG): container finished" podID="5a907e40-190d-44fe-89b8-a51e621c04db" containerID="5fac5d51930ab47edea15a76d372467b3641fb092006e5eec63567519dcec27c" exitCode=60
Oct 30 01:00:24.665847 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:24.665788    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ca5363-29888700-trbnd" event={"ID":"5a907e40-190d-44fe-89b8-a51e621c04db","Type":"ContainerDied","Data":"5fac5d51930ab47edea15a76d372467b3641fb092006e5eec63567519dcec27c"}
Oct 30 01:00:24.666156 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:24.666141    5002 scope.go:122] "RemoveContainer" containerID="5fac5d51930ab47edea15a76d372467b3641fb092006e5eec63567519dcec27c"
Oct 30 01:00:24.666475 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:00:24.666455    5002 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-ca5363-29888700-trbnd_odh-ai-gateway-infra(5a907e40-190d-44fe-89b8-a51e621c04db)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ca5363-29888700-trbnd" podUID="5a907e40-190d-44fe-89b8-a51e621c04db"
Oct 30 01:00:24.679076 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:24.679050    5002 scope.go:122] "RemoveContainer" containerID="91831523ffc822b0ecd7051a1b3f9a5235b5cedcf6703ddb00c1ba656a3b9633"
Oct 30 01:00:24.694802 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:24.694779    5002 scope.go:122] "RemoveContainer" containerID="386fa1d5a2c4d3a43aa56a4e7887c0c518d65a7993cf95921b6bc0a401d6e2f8"
Oct 30 01:00:24.850221 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:24.850188    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-481ab7-29888700-8mt68"]
Oct 30 01:00:24.856629 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:24.856601    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-481ab7-29888700-8mt68"]
Oct 30 01:00:24.878253 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:24.878223    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-5007e1-29888700-dvm8j"
Oct 30 01:00:24.898734 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:24.898703    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-29be73-29888700-gvwp4"]
Oct 30 01:00:24.902182 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:24.902154    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-29be73-29888700-gvwp4"]
Oct 30 01:00:24.975800 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:24.975769    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38a379b2-37b6-4185-8319-2e62dcec38c8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38a379b2-37b6-4185-8319-2e62dcec38c8-maas-api-service-ca\") pod \"38a379b2-37b6-4185-8319-2e62dcec38c8\" (UID: \"38a379b2-37b6-4185-8319-2e62dcec38c8\") "
Oct 30 01:00:24.975996 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:24.975809    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38a379b2-37b6-4185-8319-2e62dcec38c8-kube-api-access-dzwmv\" (UniqueName: \"kubernetes.io/projected/38a379b2-37b6-4185-8319-2e62dcec38c8-kube-api-access-dzwmv\") pod \"38a379b2-37b6-4185-8319-2e62dcec38c8\" (UID: \"38a379b2-37b6-4185-8319-2e62dcec38c8\") "
Oct 30 01:00:24.977065 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:24.976165    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38a379b2-37b6-4185-8319-2e62dcec38c8-maas-api-service-ca" pod "38a379b2-37b6-4185-8319-2e62dcec38c8" (UID: "38a379b2-37b6-4185-8319-2e62dcec38c8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:00:24.977065 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:24.976605    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38a379b2-37b6-4185-8319-2e62dcec38c8-maas-api-service-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:00:24.978784 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:24.978759    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38a379b2-37b6-4185-8319-2e62dcec38c8-kube-api-access-dzwmv" pod "38a379b2-37b6-4185-8319-2e62dcec38c8" (UID: "38a379b2-37b6-4185-8319-2e62dcec38c8"). InnerVolumeSpecName "kube-api-access-dzwmv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:00:25.056421 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:25.056374    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-11f521-29888700-khq6f"]
Oct 30 01:00:25.078390 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:25.078346    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dzwmv\" (UniqueName: \"kubernetes.io/projected/38a379b2-37b6-4185-8319-2e62dcec38c8-kube-api-access-dzwmv\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:00:25.097952 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:25.097878    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ca5363-29888700-trbnd"]
Oct 30 01:00:25.660895 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:25.659828    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w1-48df96/auth-test-model-48df96-kserve-9bb4b5585-cvvln"
Oct 30 01:00:25.719232 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:25.718606    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w1-48df96/auth-test-model-48df96-kserve-9bb4b5585-cvvln"
Oct 30 01:00:25.726798 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:25.726762    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-5007e1-29888700-dvm8j"
Oct 30 01:00:25.727002 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:25.726789    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-5007e1-29888700-dvm8j" event={"ID":"38a379b2-37b6-4185-8319-2e62dcec38c8","Type":"ContainerDied","Data":"9b99a1f2a2e06266a062454ab57913512ef59c758eb1ad3c8eb89c28c4b6c2e4"}
Oct 30 01:00:25.727002 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:25.726865    5002 scope.go:122] "RemoveContainer" containerID="ddad04d1e37fb7ab6d902cdf2bfcba87aab46d3cda03d34867111a5f4ac9056c"
Oct 30 01:00:25.750448 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:25.750027    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w1-48df96/auth-test-model-48df96-kserve-9bb4b5585-cvvln"
Oct 30 01:00:25.856761 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:25.856729    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-5007e1-29888700-dvm8j"]
Oct 30 01:00:25.868669 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:25.868187    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-5007e1-29888700-dvm8j"]
Oct 30 01:00:26.038353 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:26.038325    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ca5363-29888700-trbnd"
Oct 30 01:00:26.040829 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:26.040812    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-11f521-29888700-khq6f"
Oct 30 01:00:26.103523 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:26.103488    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f443227-6ffc-40f4-80aa-1485ae41aa02-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f443227-6ffc-40f4-80aa-1485ae41aa02-maas-api-service-ca\") pod \"7f443227-6ffc-40f4-80aa-1485ae41aa02\" (UID: \"7f443227-6ffc-40f4-80aa-1485ae41aa02\") "
Oct 30 01:00:26.103778 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:26.103617    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f443227-6ffc-40f4-80aa-1485ae41aa02-kube-api-access-ns8b9\" (UniqueName: \"kubernetes.io/projected/7f443227-6ffc-40f4-80aa-1485ae41aa02-kube-api-access-ns8b9\") pod \"7f443227-6ffc-40f4-80aa-1485ae41aa02\" (UID: \"7f443227-6ffc-40f4-80aa-1485ae41aa02\") "
Oct 30 01:00:26.103778 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:26.103667    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a907e40-190d-44fe-89b8-a51e621c04db-kube-api-access-5bjnc\" (UniqueName: \"kubernetes.io/projected/5a907e40-190d-44fe-89b8-a51e621c04db-kube-api-access-5bjnc\") pod \"5a907e40-190d-44fe-89b8-a51e621c04db\" (UID: \"5a907e40-190d-44fe-89b8-a51e621c04db\") "
Oct 30 01:00:26.103778 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:26.103723    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a907e40-190d-44fe-89b8-a51e621c04db-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a907e40-190d-44fe-89b8-a51e621c04db-maas-api-service-ca\") pod \"5a907e40-190d-44fe-89b8-a51e621c04db\" (UID: \"5a907e40-190d-44fe-89b8-a51e621c04db\") "
Oct 30 01:00:26.104639 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:26.104607    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f443227-6ffc-40f4-80aa-1485ae41aa02-maas-api-service-ca" pod "7f443227-6ffc-40f4-80aa-1485ae41aa02" (UID: "7f443227-6ffc-40f4-80aa-1485ae41aa02"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:00:26.105758 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:26.105729    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a907e40-190d-44fe-89b8-a51e621c04db-maas-api-service-ca" pod "5a907e40-190d-44fe-89b8-a51e621c04db" (UID: "5a907e40-190d-44fe-89b8-a51e621c04db"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:00:26.106208 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:26.106180    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f443227-6ffc-40f4-80aa-1485ae41aa02-kube-api-access-ns8b9" pod "7f443227-6ffc-40f4-80aa-1485ae41aa02" (UID: "7f443227-6ffc-40f4-80aa-1485ae41aa02"). InnerVolumeSpecName "kube-api-access-ns8b9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:00:26.107870 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:26.107849    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f443227-6ffc-40f4-80aa-1485ae41aa02-maas-api-service-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:00:26.107945 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:26.107873    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ns8b9\" (UniqueName: \"kubernetes.io/projected/7f443227-6ffc-40f4-80aa-1485ae41aa02-kube-api-access-ns8b9\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:00:26.107945 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:26.107890    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a907e40-190d-44fe-89b8-a51e621c04db-maas-api-service-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:00:26.108724 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:26.108693    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a907e40-190d-44fe-89b8-a51e621c04db-kube-api-access-5bjnc" pod "5a907e40-190d-44fe-89b8-a51e621c04db" (UID: "5a907e40-190d-44fe-89b8-a51e621c04db"). InnerVolumeSpecName "kube-api-access-5bjnc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:00:26.209222 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:26.209184    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5bjnc\" (UniqueName: \"kubernetes.io/projected/5a907e40-190d-44fe-89b8-a51e621c04db-kube-api-access-5bjnc\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:00:26.381215 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:26.381135    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c72c8308-xgd5x"]
Oct 30 01:00:26.381864 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:26.381843    5002 cpu_manager.go:414] "containerMap: removing container" podUID="b15c8d59-8957-4e66-8e15-998c8a33e690" containerName="cleanup"
Oct 30 01:00:26.381864 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:26.381863    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b15c8d59-8957-4e66-8e15-998c8a33e690" containerName="cleanup"
Oct 30 01:00:26.382016 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:26.381874    5002 cpu_manager.go:414] "containerMap: removing container" podUID="4af343ba-bc4d-4ec3-92df-1534780d3089" containerName="cleanup"
Oct 30 01:00:26.382016 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:26.381879    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4af343ba-bc4d-4ec3-92df-1534780d3089" containerName="cleanup"
Oct 30 01:00:26.382016 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:26.381889    5002 cpu_manager.go:414] "containerMap: removing container" podUID="5fa9212f-5c4d-47a2-a855-2c18a807533e" containerName="cleanup"
Oct 30 01:00:26.382016 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:26.381895    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5fa9212f-5c4d-47a2-a855-2c18a807533e" containerName="cleanup"
Oct 30 01:00:26.382016 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:26.381902    5002 cpu_manager.go:414] "containerMap: removing container" podUID="c111db1f-7e6f-44b8-b211-1658aee31337" containerName="cleanup"
Oct 30 01:00:26.382016 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:26.381907    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c111db1f-7e6f-44b8-b211-1658aee31337" containerName="cleanup"
Oct 30 01:00:26.382016 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:26.381913    5002 cpu_manager.go:414] "containerMap: removing container" podUID="09fb128c-0670-4260-a77b-ae32de23fa0d" containerName="cleanup"
Oct 30 01:00:26.382016 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:26.381918    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09fb128c-0670-4260-a77b-ae32de23fa0d" containerName="cleanup"
Oct 30 01:00:26.382016 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:26.381925    5002 cpu_manager.go:414] "containerMap: removing container" podUID="d5279cb1-29b0-494a-b3b3-527e8af641dc" containerName="cleanup"
Oct 30 01:00:26.382016 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:26.381930    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5279cb1-29b0-494a-b3b3-527e8af641dc" containerName="cleanup"
Oct 30 01:00:26.382016 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:26.381937    5002 cpu_manager.go:414] "containerMap: removing container" podUID="7f443227-6ffc-40f4-80aa-1485ae41aa02" containerName="cleanup"
Oct 30 01:00:26.382016 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:26.381941    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f443227-6ffc-40f4-80aa-1485ae41aa02" containerName="cleanup"
Oct 30 01:00:26.382016 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:26.381951    5002 cpu_manager.go:414] "containerMap: removing container" podUID="7f443227-6ffc-40f4-80aa-1485ae41aa02" containerName="cleanup"
Oct 30 01:00:26.382016 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:26.381955    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f443227-6ffc-40f4-80aa-1485ae41aa02" containerName="cleanup"
Oct 30 01:00:26.382016 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:26.381966    5002 cpu_manager.go:414] "containerMap: removing container" podUID="38a379b2-37b6-4185-8319-2e62dcec38c8" containerName="cleanup"
Oct 30 01:00:26.382016 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:26.381971    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38a379b2-37b6-4185-8319-2e62dcec38c8" containerName="cleanup"
Oct 30 01:00:26.382016 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:26.381978    5002 cpu_manager.go:414] "containerMap: removing container" podUID="d22fc624-e367-49a0-ba47-9a5dcabdde1f" containerName="cleanup"
Oct 30 01:00:26.382016 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:26.381983    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d22fc624-e367-49a0-ba47-9a5dcabdde1f" containerName="cleanup"
Oct 30 01:00:26.382016 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:26.381991    5002 cpu_manager.go:414] "containerMap: removing container" podUID="09fb128c-0670-4260-a77b-ae32de23fa0d" containerName="cleanup"
Oct 30 01:00:26.382016 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:26.381995    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09fb128c-0670-4260-a77b-ae32de23fa0d" containerName="cleanup"
Oct 30 01:00:26.382016 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:26.382004    5002 cpu_manager.go:414] "containerMap: removing container" podUID="d5279cb1-29b0-494a-b3b3-527e8af641dc" containerName="cleanup"
Oct 30 01:00:26.382016 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:26.382009    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5279cb1-29b0-494a-b3b3-527e8af641dc" containerName="cleanup"
Oct 30 01:00:26.382016 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:26.382015    5002 cpu_manager.go:414] "containerMap: removing container" podUID="bd72bc53-220d-40cd-abb8-36bc71b66094" containerName="cleanup"
Oct 30 01:00:26.382016 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:26.382019    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd72bc53-220d-40cd-abb8-36bc71b66094" containerName="cleanup"
Oct 30 01:00:26.382016 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:26.382025    5002 cpu_manager.go:414] "containerMap: removing container" podUID="5a907e40-190d-44fe-89b8-a51e621c04db" containerName="cleanup"
Oct 30 01:00:26.382760 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:26.382029    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a907e40-190d-44fe-89b8-a51e621c04db" containerName="cleanup"
Oct 30 01:00:26.382760 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:26.382036    5002 cpu_manager.go:414] "containerMap: removing container" podUID="b6c630bd-ac41-4607-8ed9-5f68cf00be38" containerName="cleanup"
Oct 30 01:00:26.382760 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:26.382040    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6c630bd-ac41-4607-8ed9-5f68cf00be38" containerName="cleanup"
Oct 30 01:00:26.382760 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:26.382048    5002 cpu_manager.go:414] "containerMap: removing container" podUID="4af343ba-bc4d-4ec3-92df-1534780d3089" containerName="cleanup"
Oct 30 01:00:26.382760 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:26.382052    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4af343ba-bc4d-4ec3-92df-1534780d3089" containerName="cleanup"
Oct 30 01:00:26.382760 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:26.382059    5002 cpu_manager.go:414] "containerMap: removing container" podUID="38a379b2-37b6-4185-8319-2e62dcec38c8" containerName="cleanup"
Oct 30 01:00:26.382760 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:26.382063    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38a379b2-37b6-4185-8319-2e62dcec38c8" containerName="cleanup"
Oct 30 01:00:26.382760 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:26.382075    5002 cpu_manager.go:414] "containerMap: removing container" podUID="86bc1f67-d9db-4d33-8bfc-e57494705d80" containerName="cleanup"
Oct 30 01:00:26.382760 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:26.382079    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86bc1f67-d9db-4d33-8bfc-e57494705d80" containerName="cleanup"
Oct 30 01:00:26.382760 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:26.382088    5002 cpu_manager.go:414] "containerMap: removing container" podUID="bd72bc53-220d-40cd-abb8-36bc71b66094" containerName="cleanup"
Oct 30 01:00:26.382760 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:26.382092    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd72bc53-220d-40cd-abb8-36bc71b66094" containerName="cleanup"
Oct 30 01:00:26.382760 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:26.382098    5002 cpu_manager.go:414] "containerMap: removing container" podUID="38a379b2-37b6-4185-8319-2e62dcec38c8" containerName="cleanup"
Oct 30 01:00:26.382760 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:26.382103    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38a379b2-37b6-4185-8319-2e62dcec38c8" containerName="cleanup"
Oct 30 01:00:26.382760 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:26.382109    5002 cpu_manager.go:414] "containerMap: removing container" podUID="743ebbf6-b4db-4774-b836-de5d7c716ca7" containerName="cleanup"
Oct 30 01:00:26.382760 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:26.382114    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="743ebbf6-b4db-4774-b836-de5d7c716ca7" containerName="cleanup"
Oct 30 01:00:26.382760 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:26.382121    5002 cpu_manager.go:414] "containerMap: removing container" podUID="4af343ba-bc4d-4ec3-92df-1534780d3089" containerName="cleanup"
Oct 30 01:00:26.382760 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:26.382125    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4af343ba-bc4d-4ec3-92df-1534780d3089" containerName="cleanup"
Oct 30 01:00:26.382760 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:26.382132    5002 cpu_manager.go:414] "containerMap: removing container" podUID="6302c577-afc3-4632-a383-cb1fd8159992" containerName="cleanup"
Oct 30 01:00:26.382760 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:26.382136    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6302c577-afc3-4632-a383-cb1fd8159992" containerName="cleanup"
Oct 30 01:00:26.382760 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:26.382141    5002 cpu_manager.go:414] "containerMap: removing container" podUID="6302c577-afc3-4632-a383-cb1fd8159992" containerName="cleanup"
Oct 30 01:00:26.382760 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:26.382145    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6302c577-afc3-4632-a383-cb1fd8159992" containerName="cleanup"
Oct 30 01:00:26.382760 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:26.382152    5002 cpu_manager.go:414] "containerMap: removing container" podUID="86bc1f67-d9db-4d33-8bfc-e57494705d80" containerName="cleanup"
Oct 30 01:00:26.382760 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:26.382156    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86bc1f67-d9db-4d33-8bfc-e57494705d80" containerName="cleanup"
Oct 30 01:00:26.382760 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:26.382164    5002 cpu_manager.go:414] "containerMap: removing container" podUID="743ebbf6-b4db-4774-b836-de5d7c716ca7" containerName="cleanup"
Oct 30 01:00:26.382760 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:26.382168    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="743ebbf6-b4db-4774-b836-de5d7c716ca7" containerName="cleanup"
Oct 30 01:00:26.382760 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:26.382176    5002 cpu_manager.go:414] "containerMap: removing container" podUID="743ebbf6-b4db-4774-b836-de5d7c716ca7" containerName="cleanup"
Oct 30 01:00:26.382760 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:26.382181    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="743ebbf6-b4db-4774-b836-de5d7c716ca7" containerName="cleanup"
Oct 30 01:00:26.382760 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:26.382188    5002 cpu_manager.go:414] "containerMap: removing container" podUID="5a907e40-190d-44fe-89b8-a51e621c04db" containerName="cleanup"
Oct 30 01:00:26.382760 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:26.382192    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a907e40-190d-44fe-89b8-a51e621c04db" containerName="cleanup"
Oct 30 01:00:26.382760 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:26.382199    5002 cpu_manager.go:414] "containerMap: removing container" podUID="d22fc624-e367-49a0-ba47-9a5dcabdde1f" containerName="cleanup"
Oct 30 01:00:26.382760 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:26.382203    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d22fc624-e367-49a0-ba47-9a5dcabdde1f" containerName="cleanup"
Oct 30 01:00:26.382760 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:26.382211    5002 cpu_manager.go:414] "containerMap: removing container" podUID="c111db1f-7e6f-44b8-b211-1658aee31337" containerName="cleanup"
Oct 30 01:00:26.382760 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:26.382215    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c111db1f-7e6f-44b8-b211-1658aee31337" containerName="cleanup"
Oct 30 01:00:26.382760 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:26.382222    5002 cpu_manager.go:414] "containerMap: removing container" podUID="b6c630bd-ac41-4607-8ed9-5f68cf00be38" containerName="cleanup"
Oct 30 01:00:26.382760 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:26.382226    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6c630bd-ac41-4607-8ed9-5f68cf00be38" containerName="cleanup"
Oct 30 01:00:26.382760 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:26.382237    5002 cpu_manager.go:414] "containerMap: removing container" podUID="b15c8d59-8957-4e66-8e15-998c8a33e690" containerName="cleanup"
Oct 30 01:00:26.382760 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:26.382241    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b15c8d59-8957-4e66-8e15-998c8a33e690" containerName="cleanup"
Oct 30 01:00:26.382760 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:26.382249    5002 cpu_manager.go:414] "containerMap: removing container" podUID="d22fc624-e367-49a0-ba47-9a5dcabdde1f" containerName="cleanup"
Oct 30 01:00:26.382760 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:26.382253    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d22fc624-e367-49a0-ba47-9a5dcabdde1f" containerName="cleanup"
Oct 30 01:00:26.382760 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:26.382259    5002 cpu_manager.go:414] "containerMap: removing container" podUID="5fa9212f-5c4d-47a2-a855-2c18a807533e" containerName="cleanup"
Oct 30 01:00:26.382760 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:26.382264    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5fa9212f-5c4d-47a2-a855-2c18a807533e" containerName="cleanup"
Oct 30 01:00:26.382760 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:26.382273    5002 cpu_manager.go:414] "containerMap: removing container" podUID="86bc1f67-d9db-4d33-8bfc-e57494705d80" containerName="cleanup"
Oct 30 01:00:26.382760 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:26.382278    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86bc1f67-d9db-4d33-8bfc-e57494705d80" containerName="cleanup"
Oct 30 01:00:26.382760 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:26.382290    5002 cpu_manager.go:414] "containerMap: removing container" podUID="5fa9212f-5c4d-47a2-a855-2c18a807533e" containerName="cleanup"
Oct 30 01:00:26.382760 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:26.382297    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5fa9212f-5c4d-47a2-a855-2c18a807533e" containerName="cleanup"
Oct 30 01:00:26.382760 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:26.382415    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6c630bd-ac41-4607-8ed9-5f68cf00be38" containerName="cleanup"
Oct 30 01:00:26.382760 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:26.382423    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="4af343ba-bc4d-4ec3-92df-1534780d3089" containerName="cleanup"
Oct 30 01:00:26.382760 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:26.382428    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="38a379b2-37b6-4185-8319-2e62dcec38c8" containerName="cleanup"
Oct 30 01:00:26.382760 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:26.382435    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a907e40-190d-44fe-89b8-a51e621c04db" containerName="cleanup"
Oct 30 01:00:26.382760 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:26.382441    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="86bc1f67-d9db-4d33-8bfc-e57494705d80" containerName="cleanup"
Oct 30 01:00:26.382760 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:26.382446    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="86bc1f67-d9db-4d33-8bfc-e57494705d80" containerName="cleanup"
Oct 30 01:00:26.382760 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:26.382452    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd72bc53-220d-40cd-abb8-36bc71b66094" containerName="cleanup"
Oct 30 01:00:26.382760 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:26.382458    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="38a379b2-37b6-4185-8319-2e62dcec38c8" containerName="cleanup"
Oct 30 01:00:26.382760 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:26.382465    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="743ebbf6-b4db-4774-b836-de5d7c716ca7" containerName="cleanup"
Oct 30 01:00:26.382760 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:26.382471    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="4af343ba-bc4d-4ec3-92df-1534780d3089" containerName="cleanup"
Oct 30 01:00:26.382760 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:26.382477    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="6302c577-afc3-4632-a383-cb1fd8159992" containerName="cleanup"
Oct 30 01:00:26.382760 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:26.382482    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="6302c577-afc3-4632-a383-cb1fd8159992" containerName="cleanup"
Oct 30 01:00:26.382760 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:26.382487    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="743ebbf6-b4db-4774-b836-de5d7c716ca7" containerName="cleanup"
Oct 30 01:00:26.382760 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:26.382492    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="743ebbf6-b4db-4774-b836-de5d7c716ca7" containerName="cleanup"
Oct 30 01:00:26.382760 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:26.382498    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a907e40-190d-44fe-89b8-a51e621c04db" containerName="cleanup"
Oct 30 01:00:26.382760 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:26.382504    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="d22fc624-e367-49a0-ba47-9a5dcabdde1f" containerName="cleanup"
Oct 30 01:00:26.382760 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:26.382509    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="09fb128c-0670-4260-a77b-ae32de23fa0d" containerName="cleanup"
Oct 30 01:00:26.382760 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:26.382513    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="c111db1f-7e6f-44b8-b211-1658aee31337" containerName="cleanup"
Oct 30 01:00:26.382760 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:26.382517    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="b15c8d59-8957-4e66-8e15-998c8a33e690" containerName="cleanup"
Oct 30 01:00:26.382760 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:26.382523    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6c630bd-ac41-4607-8ed9-5f68cf00be38" containerName="cleanup"
Oct 30 01:00:26.382760 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:26.382529    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="b15c8d59-8957-4e66-8e15-998c8a33e690" containerName="cleanup"
Oct 30 01:00:26.382760 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:26.382535    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="d22fc624-e367-49a0-ba47-9a5dcabdde1f" containerName="cleanup"
Oct 30 01:00:26.382760 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:26.382540    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="5fa9212f-5c4d-47a2-a855-2c18a807533e" containerName="cleanup"
Oct 30 01:00:26.382760 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:26.382545    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f443227-6ffc-40f4-80aa-1485ae41aa02" containerName="cleanup"
Oct 30 01:00:26.382760 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:26.382550    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5279cb1-29b0-494a-b3b3-527e8af641dc" containerName="cleanup"
Oct 30 01:00:26.382760 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:26.382554    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="b15c8d59-8957-4e66-8e15-998c8a33e690" containerName="cleanup"
Oct 30 01:00:26.382760 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:26.382561    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="5fa9212f-5c4d-47a2-a855-2c18a807533e" containerName="cleanup"
Oct 30 01:00:26.382760 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:26.382567    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="c111db1f-7e6f-44b8-b211-1658aee31337" containerName="cleanup"
Oct 30 01:00:26.382760 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:26.382571    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="09fb128c-0670-4260-a77b-ae32de23fa0d" containerName="cleanup"
Oct 30 01:00:26.382760 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:26.382577    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5279cb1-29b0-494a-b3b3-527e8af641dc" containerName="cleanup"
Oct 30 01:00:26.382760 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:26.382582    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f443227-6ffc-40f4-80aa-1485ae41aa02" containerName="cleanup"
Oct 30 01:00:26.382760 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:26.382589    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5279cb1-29b0-494a-b3b3-527e8af641dc" containerName="cleanup"
Oct 30 01:00:26.382760 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:26.382594    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd72bc53-220d-40cd-abb8-36bc71b66094" containerName="cleanup"
Oct 30 01:00:26.386444 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:26.386424    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c72c8308-xgd5x"
Oct 30 01:00:26.402933 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:26.402908    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:00:26.405046 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:26.405021    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c72c8308-xgd5x"]
Oct 30 01:00:26.412093 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:26.412057    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58s2c\" (UniqueName: \"kubernetes.io/projected/d76781c5-3cb1-48c5-81d0-8c4d09c26a82-kube-api-access-58s2c\") pod \"maas-api-revoke-sub-c72c8308-xgd5x\" (UID: \"d76781c5-3cb1-48c5-81d0-8c4d09c26a82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c72c8308-xgd5x"
Oct 30 01:00:26.412197 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:26.412145    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d76781c5-3cb1-48c5-81d0-8c4d09c26a82-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c72c8308-xgd5x\" (UID: \"d76781c5-3cb1-48c5-81d0-8c4d09c26a82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c72c8308-xgd5x"
Oct 30 01:00:26.412335 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:26.412310    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d76781c5-3cb1-48c5-81d0-8c4d09c26a82-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c72c8308-xgd5x\" (UID: \"d76781c5-3cb1-48c5-81d0-8c4d09c26a82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c72c8308-xgd5x"
Oct 30 01:00:26.505017 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:26.504981    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38a379b2-37b6-4185-8319-2e62dcec38c8" path="/var/lib/kubelet/pods/38a379b2-37b6-4185-8319-2e62dcec38c8/volumes"
Oct 30 01:00:26.505619 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:26.505593    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86bc1f67-d9db-4d33-8bfc-e57494705d80" path="/var/lib/kubelet/pods/86bc1f67-d9db-4d33-8bfc-e57494705d80/volumes"
Oct 30 01:00:26.506029 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:26.506013    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b15c8d59-8957-4e66-8e15-998c8a33e690" path="/var/lib/kubelet/pods/b15c8d59-8957-4e66-8e15-998c8a33e690/volumes"
Oct 30 01:00:26.513422 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:26.513391    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d76781c5-3cb1-48c5-81d0-8c4d09c26a82-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c72c8308-xgd5x\" (UID: \"d76781c5-3cb1-48c5-81d0-8c4d09c26a82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c72c8308-xgd5x"
Oct 30 01:00:26.513585 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:26.513568    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58s2c\" (UniqueName: \"kubernetes.io/projected/d76781c5-3cb1-48c5-81d0-8c4d09c26a82-kube-api-access-58s2c\") pod \"maas-api-revoke-sub-c72c8308-xgd5x\" (UID: \"d76781c5-3cb1-48c5-81d0-8c4d09c26a82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c72c8308-xgd5x"
Oct 30 01:00:26.513665 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:26.513628    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d76781c5-3cb1-48c5-81d0-8c4d09c26a82-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c72c8308-xgd5x\" (UID: \"d76781c5-3cb1-48c5-81d0-8c4d09c26a82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c72c8308-xgd5x"
Oct 30 01:00:26.515182 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:26.515153    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d76781c5-3cb1-48c5-81d0-8c4d09c26a82-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c72c8308-xgd5x\" (UID: \"d76781c5-3cb1-48c5-81d0-8c4d09c26a82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c72c8308-xgd5x"
Oct 30 01:00:26.516790 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:26.516768    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d76781c5-3cb1-48c5-81d0-8c4d09c26a82-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c72c8308-xgd5x\" (UID: \"d76781c5-3cb1-48c5-81d0-8c4d09c26a82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c72c8308-xgd5x"
Oct 30 01:00:26.539639 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:26.539611    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-58s2c\" (UniqueName: \"kubernetes.io/projected/d76781c5-3cb1-48c5-81d0-8c4d09c26a82-kube-api-access-58s2c\") pod \"maas-api-revoke-sub-c72c8308-xgd5x\" (UID: \"d76781c5-3cb1-48c5-81d0-8c4d09c26a82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c72c8308-xgd5x"
Oct 30 01:00:26.703244 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:26.703206    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c72c8308-xgd5x"
Oct 30 01:00:26.747644 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:26.745509    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-11f521-29888700-khq6f"
Oct 30 01:00:26.747644 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:26.745521    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-11f521-29888700-khq6f" event={"ID":"7f443227-6ffc-40f4-80aa-1485ae41aa02","Type":"ContainerDied","Data":"6080d5850f8b58432aba64ab7f83c20f4392b3f9d229b27c062521c90284f67f"}
Oct 30 01:00:26.747644 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:26.745601    5002 scope.go:122] "RemoveContainer" containerID="3ebf07cd05cbfab22fcec5063c943e4b9a54669db9b312012f54c00d13e52dfe"
Oct 30 01:00:26.749487 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:26.749453    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ca5363-29888700-trbnd" event={"ID":"5a907e40-190d-44fe-89b8-a51e621c04db","Type":"ContainerDied","Data":"595e74c00f33e815ccc45c5ac91f8a591834b4e3cd2e92c84b4b2a6c45a8b522"}
Oct 30 01:00:26.749692 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:26.749568    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ca5363-29888700-trbnd"
Oct 30 01:00:26.759794 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:26.759775    5002 scope.go:122] "RemoveContainer" containerID="5fac5d51930ab47edea15a76d372467b3641fb092006e5eec63567519dcec27c"
Oct 30 01:00:26.810178 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:26.810138    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-11f521-29888700-khq6f"]
Oct 30 01:00:26.816287 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:26.816256    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-11f521-29888700-khq6f"]
Oct 30 01:00:26.849398 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:26.849341    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ca5363-29888700-trbnd"]
Oct 30 01:00:26.853228 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:26.853199    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ca5363-29888700-trbnd"]
Oct 30 01:00:27.021205 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:27.021173    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c72c8308-xgd5x"]
Oct 30 01:00:27.021832 ip-10-0-128-7 kubenswrapper[5002]: W1030 01:00:27.021809    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd76781c5_3cb1_48c5_81d0_8c4d09c26a82.slice/crio-2627159e8d0d25eb7fd90d74fd4e7ba8f5beb3ac040838ffa1017b606e5fbe0e WatchSource:0}: Error finding container 2627159e8d0d25eb7fd90d74fd4e7ba8f5beb3ac040838ffa1017b606e5fbe0e: Status 404 returned error can't find the container with id 2627159e8d0d25eb7fd90d74fd4e7ba8f5beb3ac040838ffa1017b606e5fbe0e
Oct 30 01:00:27.761288 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:27.761153    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c72c8308-xgd5x" event={"ID":"d76781c5-3cb1-48c5-81d0-8c4d09c26a82","Type":"ContainerStarted","Data":"20aa1961d48dcad93eb5f84e2cda6e191ad8e191d5bf163c93253bd5888f44e2"}
Oct 30 01:00:27.761805 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:27.761494    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c72c8308-xgd5x" event={"ID":"d76781c5-3cb1-48c5-81d0-8c4d09c26a82","Type":"ContainerStarted","Data":"2627159e8d0d25eb7fd90d74fd4e7ba8f5beb3ac040838ffa1017b606e5fbe0e"}
Oct 30 01:00:28.500214 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:28.500181    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a907e40-190d-44fe-89b8-a51e621c04db" path="/var/lib/kubelet/pods/5a907e40-190d-44fe-89b8-a51e621c04db/volumes"
Oct 30 01:00:28.500754 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:28.500738    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f443227-6ffc-40f4-80aa-1485ae41aa02" path="/var/lib/kubelet/pods/7f443227-6ffc-40f4-80aa-1485ae41aa02/volumes"
Oct 30 01:00:28.772492 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:28.772362    5002 generic.go:356] "Generic (PLEG): container finished" podID="d76781c5-3cb1-48c5-81d0-8c4d09c26a82" containerID="20aa1961d48dcad93eb5f84e2cda6e191ad8e191d5bf163c93253bd5888f44e2" exitCode=0
Oct 30 01:00:28.772492 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:28.772458    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c72c8308-xgd5x" event={"ID":"d76781c5-3cb1-48c5-81d0-8c4d09c26a82","Type":"ContainerDied","Data":"20aa1961d48dcad93eb5f84e2cda6e191ad8e191d5bf163c93253bd5888f44e2"}
Oct 30 01:00:29.964038 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:29.964008    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c72c8308-xgd5x"
Oct 30 01:00:30.068761 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:30.068664    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d76781c5-3cb1-48c5-81d0-8c4d09c26a82-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d76781c5-3cb1-48c5-81d0-8c4d09c26a82-maas-api-openshift-ca\") pod \"d76781c5-3cb1-48c5-81d0-8c4d09c26a82\" (UID: \"d76781c5-3cb1-48c5-81d0-8c4d09c26a82\") "
Oct 30 01:00:30.068922 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:30.068787    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d76781c5-3cb1-48c5-81d0-8c4d09c26a82-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d76781c5-3cb1-48c5-81d0-8c4d09c26a82-maas-api-xks-ca\") pod \"d76781c5-3cb1-48c5-81d0-8c4d09c26a82\" (UID: \"d76781c5-3cb1-48c5-81d0-8c4d09c26a82\") "
Oct 30 01:00:30.068922 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:30.068899    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d76781c5-3cb1-48c5-81d0-8c4d09c26a82-kube-api-access-58s2c\" (UniqueName: \"kubernetes.io/projected/d76781c5-3cb1-48c5-81d0-8c4d09c26a82-kube-api-access-58s2c\") pod \"d76781c5-3cb1-48c5-81d0-8c4d09c26a82\" (UID: \"d76781c5-3cb1-48c5-81d0-8c4d09c26a82\") "
Oct 30 01:00:30.069154 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:30.069104    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d76781c5-3cb1-48c5-81d0-8c4d09c26a82-maas-api-openshift-ca" pod "d76781c5-3cb1-48c5-81d0-8c4d09c26a82" (UID: "d76781c5-3cb1-48c5-81d0-8c4d09c26a82"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:00:30.069705 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:30.069679    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d76781c5-3cb1-48c5-81d0-8c4d09c26a82-maas-api-openshift-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:00:30.071943 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:30.071916    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d76781c5-3cb1-48c5-81d0-8c4d09c26a82-kube-api-access-58s2c" pod "d76781c5-3cb1-48c5-81d0-8c4d09c26a82" (UID: "d76781c5-3cb1-48c5-81d0-8c4d09c26a82"). InnerVolumeSpecName "kube-api-access-58s2c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:00:30.072042 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:30.071996    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d76781c5-3cb1-48c5-81d0-8c4d09c26a82-maas-api-xks-ca" pod "d76781c5-3cb1-48c5-81d0-8c4d09c26a82" (UID: "d76781c5-3cb1-48c5-81d0-8c4d09c26a82"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:00:30.172129 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:30.172091    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d76781c5-3cb1-48c5-81d0-8c4d09c26a82-maas-api-xks-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:00:30.172129 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:30.172130    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-58s2c\" (UniqueName: \"kubernetes.io/projected/d76781c5-3cb1-48c5-81d0-8c4d09c26a82-kube-api-access-58s2c\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:00:30.790483 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:30.790399    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c72c8308-xgd5x"
Oct 30 01:00:30.790483 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:30.790415    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c72c8308-xgd5x" event={"ID":"d76781c5-3cb1-48c5-81d0-8c4d09c26a82","Type":"ContainerDied","Data":"2627159e8d0d25eb7fd90d74fd4e7ba8f5beb3ac040838ffa1017b606e5fbe0e"}
Oct 30 01:00:30.790483 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:30.790474    5002 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2627159e8d0d25eb7fd90d74fd4e7ba8f5beb3ac040838ffa1017b606e5fbe0e"
Oct 30 01:00:36.971264 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:36.971230    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bab19a-nrvmd"]
Oct 30 01:00:36.984799 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:36.984767    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bab19a-nrvmd"]
Oct 30 01:00:38.499425 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:38.499349    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b780a91d-c4f6-4c5a-834b-ec5a686bae51" path="/var/lib/kubelet/pods/b780a91d-c4f6-4c5a-834b-ec5a686bae51/volumes"
Oct 30 01:00:42.866226 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:00:42.866188    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbjdq" containerName="extractor"
Oct 30 01:00:44.872862 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:44.872798    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7d8z4"]
Oct 30 01:00:45.534224 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:45.534190    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-ef1084-openshift-default-68989c6b9f-lrmtq"]
Oct 30 01:00:45.534879 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:45.534860    5002 cpu_manager.go:414] "containerMap: removing container" podUID="b15c8d59-8957-4e66-8e15-998c8a33e690" containerName="cleanup"
Oct 30 01:00:45.534949 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:45.534880    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b15c8d59-8957-4e66-8e15-998c8a33e690" containerName="cleanup"
Oct 30 01:00:45.534949 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:45.534900    5002 cpu_manager.go:414] "containerMap: removing container" podUID="7f443227-6ffc-40f4-80aa-1485ae41aa02" containerName="cleanup"
Oct 30 01:00:45.534949 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:45.534905    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f443227-6ffc-40f4-80aa-1485ae41aa02" containerName="cleanup"
Oct 30 01:00:45.534949 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:45.534918    5002 cpu_manager.go:414] "containerMap: removing container" podUID="09fb128c-0670-4260-a77b-ae32de23fa0d" containerName="cleanup"
Oct 30 01:00:45.534949 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:45.534923    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09fb128c-0670-4260-a77b-ae32de23fa0d" containerName="cleanup"
Oct 30 01:00:45.534949 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:45.534930    5002 cpu_manager.go:414] "containerMap: removing container" podUID="d5279cb1-29b0-494a-b3b3-527e8af641dc" containerName="cleanup"
Oct 30 01:00:45.534949 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:45.534935    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5279cb1-29b0-494a-b3b3-527e8af641dc" containerName="cleanup"
Oct 30 01:00:45.534949 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:45.534943    5002 cpu_manager.go:414] "containerMap: removing container" podUID="bd72bc53-220d-40cd-abb8-36bc71b66094" containerName="cleanup"
Oct 30 01:00:45.534949 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:45.534947    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd72bc53-220d-40cd-abb8-36bc71b66094" containerName="cleanup"
Oct 30 01:00:45.535224 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:45.534955    5002 cpu_manager.go:414] "containerMap: removing container" podUID="5a907e40-190d-44fe-89b8-a51e621c04db" containerName="cleanup"
Oct 30 01:00:45.535224 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:45.534960    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a907e40-190d-44fe-89b8-a51e621c04db" containerName="cleanup"
Oct 30 01:00:45.535224 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:45.534972    5002 cpu_manager.go:414] "containerMap: removing container" podUID="d76781c5-3cb1-48c5-81d0-8c4d09c26a82" containerName="revoke-keys"
Oct 30 01:00:45.535224 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:45.534976    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d76781c5-3cb1-48c5-81d0-8c4d09c26a82" containerName="revoke-keys"
Oct 30 01:00:45.535224 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:45.534987    5002 cpu_manager.go:414] "containerMap: removing container" podUID="6302c577-afc3-4632-a383-cb1fd8159992" containerName="cleanup"
Oct 30 01:00:45.535224 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:45.534992    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6302c577-afc3-4632-a383-cb1fd8159992" containerName="cleanup"
Oct 30 01:00:45.535224 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:45.535007    5002 cpu_manager.go:414] "containerMap: removing container" podUID="c111db1f-7e6f-44b8-b211-1658aee31337" containerName="cleanup"
Oct 30 01:00:45.535224 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:45.535012    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c111db1f-7e6f-44b8-b211-1658aee31337" containerName="cleanup"
Oct 30 01:00:45.535224 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:45.535019    5002 cpu_manager.go:414] "containerMap: removing container" podUID="b6c630bd-ac41-4607-8ed9-5f68cf00be38" containerName="cleanup"
Oct 30 01:00:45.535224 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:45.535023    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6c630bd-ac41-4607-8ed9-5f68cf00be38" containerName="cleanup"
Oct 30 01:00:45.535224 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:45.535115    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="d76781c5-3cb1-48c5-81d0-8c4d09c26a82" containerName="revoke-keys"
Oct 30 01:00:45.535224 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:45.535122    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="86bc1f67-d9db-4d33-8bfc-e57494705d80" containerName="cleanup"
Oct 30 01:00:45.535224 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:45.535131    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="38a379b2-37b6-4185-8319-2e62dcec38c8" containerName="cleanup"
Oct 30 01:00:45.535224 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:45.535137    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="4af343ba-bc4d-4ec3-92df-1534780d3089" containerName="cleanup"
Oct 30 01:00:45.535224 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:45.535142    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="6302c577-afc3-4632-a383-cb1fd8159992" containerName="cleanup"
Oct 30 01:00:45.535224 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:45.535150    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a907e40-190d-44fe-89b8-a51e621c04db" containerName="cleanup"
Oct 30 01:00:45.535224 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:45.535156    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="c111db1f-7e6f-44b8-b211-1658aee31337" containerName="cleanup"
Oct 30 01:00:45.535224 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:45.535163    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="d22fc624-e367-49a0-ba47-9a5dcabdde1f" containerName="cleanup"
Oct 30 01:00:45.535224 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:45.535169    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f443227-6ffc-40f4-80aa-1485ae41aa02" containerName="cleanup"
Oct 30 01:00:45.535224 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:45.535176    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="5fa9212f-5c4d-47a2-a855-2c18a807533e" containerName="cleanup"
Oct 30 01:00:45.535224 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:45.535181    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="09fb128c-0670-4260-a77b-ae32de23fa0d" containerName="cleanup"
Oct 30 01:00:45.535224 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:45.535190    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd72bc53-220d-40cd-abb8-36bc71b66094" containerName="cleanup"
Oct 30 01:00:45.535224 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:45.535196    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6c630bd-ac41-4607-8ed9-5f68cf00be38" containerName="cleanup"
Oct 30 01:00:45.540285 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:45.540264    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ef1084-openshift-default-68989c6b9f-lrmtq"
Oct 30 01:00:45.570324 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:45.567998    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-ef1084-openshift-default-68989c6b9f-lrmtq"]
Oct 30 01:00:45.575432 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:45.575369    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-ef1084-openshift-default-dockercfg-57vpk\""
Oct 30 01:00:45.637892 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:45.637854    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-51b95e/test-model-51b95e-kserve-5c4d8899f7-hc4c2"]
Oct 30 01:00:45.642470 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:45.642431    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-51b95e/test-model-51b95e-kserve-5c4d8899f7-hc4c2"
Oct 30 01:00:45.661617 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:45.661588    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-51b95e\"/\"openshift-service-ca.crt\""
Oct 30 01:00:45.668029 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:45.667998    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-51b95e/test-model-51b95e-kserve-5c4d8899f7-hc4c2"]
Oct 30 01:00:45.670213 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:45.668656    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-51b95e\"/\"test-model-51b95e-kserve-self-signed-certs\""
Oct 30 01:00:45.670447 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:45.669699    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-51b95e\"/\"kube-root-ca.crt\""
Oct 30 01:00:45.711762 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:45.711711    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1618b1d3-81ce-4def-9498-43b3fdbc46a4-istio-podinfo\") pod \"e2e-inf-a-ef1084-openshift-default-68989c6b9f-lrmtq\" (UID: \"1618b1d3-81ce-4def-9498-43b3fdbc46a4\") " pod="openshift-ingress/e2e-inf-a-ef1084-openshift-default-68989c6b9f-lrmtq"
Oct 30 01:00:45.711973 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:45.711853    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1618b1d3-81ce-4def-9498-43b3fdbc46a4-workload-socket\") pod \"e2e-inf-a-ef1084-openshift-default-68989c6b9f-lrmtq\" (UID: \"1618b1d3-81ce-4def-9498-43b3fdbc46a4\") " pod="openshift-ingress/e2e-inf-a-ef1084-openshift-default-68989c6b9f-lrmtq"
Oct 30 01:00:45.711973 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:45.711914    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1618b1d3-81ce-4def-9498-43b3fdbc46a4-istio-token\") pod \"e2e-inf-a-ef1084-openshift-default-68989c6b9f-lrmtq\" (UID: \"1618b1d3-81ce-4def-9498-43b3fdbc46a4\") " pod="openshift-ingress/e2e-inf-a-ef1084-openshift-default-68989c6b9f-lrmtq"
Oct 30 01:00:45.711973 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:45.711968    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1618b1d3-81ce-4def-9498-43b3fdbc46a4-istiod-ca-cert\") pod \"e2e-inf-a-ef1084-openshift-default-68989c6b9f-lrmtq\" (UID: \"1618b1d3-81ce-4def-9498-43b3fdbc46a4\") " pod="openshift-ingress/e2e-inf-a-ef1084-openshift-default-68989c6b9f-lrmtq"
Oct 30 01:00:45.712153 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:45.712045    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1618b1d3-81ce-4def-9498-43b3fdbc46a4-credential-socket\") pod \"e2e-inf-a-ef1084-openshift-default-68989c6b9f-lrmtq\" (UID: \"1618b1d3-81ce-4def-9498-43b3fdbc46a4\") " pod="openshift-ingress/e2e-inf-a-ef1084-openshift-default-68989c6b9f-lrmtq"
Oct 30 01:00:45.712153 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:45.712130    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txd2n\" (UniqueName: \"kubernetes.io/projected/1618b1d3-81ce-4def-9498-43b3fdbc46a4-kube-api-access-txd2n\") pod \"e2e-inf-a-ef1084-openshift-default-68989c6b9f-lrmtq\" (UID: \"1618b1d3-81ce-4def-9498-43b3fdbc46a4\") " pod="openshift-ingress/e2e-inf-a-ef1084-openshift-default-68989c6b9f-lrmtq"
Oct 30 01:00:45.712255 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:45.712176    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1618b1d3-81ce-4def-9498-43b3fdbc46a4-istio-envoy\") pod \"e2e-inf-a-ef1084-openshift-default-68989c6b9f-lrmtq\" (UID: \"1618b1d3-81ce-4def-9498-43b3fdbc46a4\") " pod="openshift-ingress/e2e-inf-a-ef1084-openshift-default-68989c6b9f-lrmtq"
Oct 30 01:00:45.712302 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:45.712253    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1618b1d3-81ce-4def-9498-43b3fdbc46a4-istio-data\") pod \"e2e-inf-a-ef1084-openshift-default-68989c6b9f-lrmtq\" (UID: \"1618b1d3-81ce-4def-9498-43b3fdbc46a4\") " pod="openshift-ingress/e2e-inf-a-ef1084-openshift-default-68989c6b9f-lrmtq"
Oct 30 01:00:45.712346 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:45.712330    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1618b1d3-81ce-4def-9498-43b3fdbc46a4-workload-certs\") pod \"e2e-inf-a-ef1084-openshift-default-68989c6b9f-lrmtq\" (UID: \"1618b1d3-81ce-4def-9498-43b3fdbc46a4\") " pod="openshift-ingress/e2e-inf-a-ef1084-openshift-default-68989c6b9f-lrmtq"
Oct 30 01:00:45.720741 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:45.720698    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-51b95e\"/\"default-dockercfg-dsfqn\""
Oct 30 01:00:45.813998 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:45.813906    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1618b1d3-81ce-4def-9498-43b3fdbc46a4-credential-socket\") pod \"e2e-inf-a-ef1084-openshift-default-68989c6b9f-lrmtq\" (UID: \"1618b1d3-81ce-4def-9498-43b3fdbc46a4\") " pod="openshift-ingress/e2e-inf-a-ef1084-openshift-default-68989c6b9f-lrmtq"
Oct 30 01:00:45.813998 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:45.813948    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/87ebc816-fbe5-4a3e-b650-39f2a72e9037-home\") pod \"test-model-51b95e-kserve-5c4d8899f7-hc4c2\" (UID: \"87ebc816-fbe5-4a3e-b650-39f2a72e9037\") " pod="ai-tenant-e2e-inf-a-51b95e/test-model-51b95e-kserve-5c4d8899f7-hc4c2"
Oct 30 01:00:45.813998 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:45.813995    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/87ebc816-fbe5-4a3e-b650-39f2a72e9037-dshm\") pod \"test-model-51b95e-kserve-5c4d8899f7-hc4c2\" (UID: \"87ebc816-fbe5-4a3e-b650-39f2a72e9037\") " pod="ai-tenant-e2e-inf-a-51b95e/test-model-51b95e-kserve-5c4d8899f7-hc4c2"
Oct 30 01:00:45.814284 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:45.814015    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87ebc816-fbe5-4a3e-b650-39f2a72e9037-tmp-dir\") pod \"test-model-51b95e-kserve-5c4d8899f7-hc4c2\" (UID: \"87ebc816-fbe5-4a3e-b650-39f2a72e9037\") " pod="ai-tenant-e2e-inf-a-51b95e/test-model-51b95e-kserve-5c4d8899f7-hc4c2"
Oct 30 01:00:45.814284 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:45.814042    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txd2n\" (UniqueName: \"kubernetes.io/projected/1618b1d3-81ce-4def-9498-43b3fdbc46a4-kube-api-access-txd2n\") pod \"e2e-inf-a-ef1084-openshift-default-68989c6b9f-lrmtq\" (UID: \"1618b1d3-81ce-4def-9498-43b3fdbc46a4\") " pod="openshift-ingress/e2e-inf-a-ef1084-openshift-default-68989c6b9f-lrmtq"
Oct 30 01:00:45.814284 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:45.814265    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1618b1d3-81ce-4def-9498-43b3fdbc46a4-istio-envoy\") pod \"e2e-inf-a-ef1084-openshift-default-68989c6b9f-lrmtq\" (UID: \"1618b1d3-81ce-4def-9498-43b3fdbc46a4\") " pod="openshift-ingress/e2e-inf-a-ef1084-openshift-default-68989c6b9f-lrmtq"
Oct 30 01:00:45.814469 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:45.814359    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1618b1d3-81ce-4def-9498-43b3fdbc46a4-istio-data\") pod \"e2e-inf-a-ef1084-openshift-default-68989c6b9f-lrmtq\" (UID: \"1618b1d3-81ce-4def-9498-43b3fdbc46a4\") " pod="openshift-ingress/e2e-inf-a-ef1084-openshift-default-68989c6b9f-lrmtq"
Oct 30 01:00:45.814469 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:45.814400    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1618b1d3-81ce-4def-9498-43b3fdbc46a4-credential-socket\") pod \"e2e-inf-a-ef1084-openshift-default-68989c6b9f-lrmtq\" (UID: \"1618b1d3-81ce-4def-9498-43b3fdbc46a4\") " pod="openshift-ingress/e2e-inf-a-ef1084-openshift-default-68989c6b9f-lrmtq"
Oct 30 01:00:45.814469 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:45.814462    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1618b1d3-81ce-4def-9498-43b3fdbc46a4-workload-certs\") pod \"e2e-inf-a-ef1084-openshift-default-68989c6b9f-lrmtq\" (UID: \"1618b1d3-81ce-4def-9498-43b3fdbc46a4\") " pod="openshift-ingress/e2e-inf-a-ef1084-openshift-default-68989c6b9f-lrmtq"
Oct 30 01:00:45.814590 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:45.814547    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1618b1d3-81ce-4def-9498-43b3fdbc46a4-istio-podinfo\") pod \"e2e-inf-a-ef1084-openshift-default-68989c6b9f-lrmtq\" (UID: \"1618b1d3-81ce-4def-9498-43b3fdbc46a4\") " pod="openshift-ingress/e2e-inf-a-ef1084-openshift-default-68989c6b9f-lrmtq"
Oct 30 01:00:45.814644 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:45.814588    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87ebc816-fbe5-4a3e-b650-39f2a72e9037-model-cache\") pod \"test-model-51b95e-kserve-5c4d8899f7-hc4c2\" (UID: \"87ebc816-fbe5-4a3e-b650-39f2a72e9037\") " pod="ai-tenant-e2e-inf-a-51b95e/test-model-51b95e-kserve-5c4d8899f7-hc4c2"
Oct 30 01:00:45.814699 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:45.814670    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1618b1d3-81ce-4def-9498-43b3fdbc46a4-istio-data\") pod \"e2e-inf-a-ef1084-openshift-default-68989c6b9f-lrmtq\" (UID: \"1618b1d3-81ce-4def-9498-43b3fdbc46a4\") " pod="openshift-ingress/e2e-inf-a-ef1084-openshift-default-68989c6b9f-lrmtq"
Oct 30 01:00:45.814754 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:45.814678    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1618b1d3-81ce-4def-9498-43b3fdbc46a4-workload-socket\") pod \"e2e-inf-a-ef1084-openshift-default-68989c6b9f-lrmtq\" (UID: \"1618b1d3-81ce-4def-9498-43b3fdbc46a4\") " pod="openshift-ingress/e2e-inf-a-ef1084-openshift-default-68989c6b9f-lrmtq"
Oct 30 01:00:45.814813 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:45.814774    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1618b1d3-81ce-4def-9498-43b3fdbc46a4-istio-token\") pod \"e2e-inf-a-ef1084-openshift-default-68989c6b9f-lrmtq\" (UID: \"1618b1d3-81ce-4def-9498-43b3fdbc46a4\") " pod="openshift-ingress/e2e-inf-a-ef1084-openshift-default-68989c6b9f-lrmtq"
Oct 30 01:00:45.814874 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:45.814814    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87ebc816-fbe5-4a3e-b650-39f2a72e9037-tls-certs\") pod \"test-model-51b95e-kserve-5c4d8899f7-hc4c2\" (UID: \"87ebc816-fbe5-4a3e-b650-39f2a72e9037\") " pod="ai-tenant-e2e-inf-a-51b95e/test-model-51b95e-kserve-5c4d8899f7-hc4c2"
Oct 30 01:00:45.814874 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:45.814850    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1618b1d3-81ce-4def-9498-43b3fdbc46a4-istiod-ca-cert\") pod \"e2e-inf-a-ef1084-openshift-default-68989c6b9f-lrmtq\" (UID: \"1618b1d3-81ce-4def-9498-43b3fdbc46a4\") " pod="openshift-ingress/e2e-inf-a-ef1084-openshift-default-68989c6b9f-lrmtq"
Oct 30 01:00:45.814874 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:45.814862    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1618b1d3-81ce-4def-9498-43b3fdbc46a4-workload-certs\") pod \"e2e-inf-a-ef1084-openshift-default-68989c6b9f-lrmtq\" (UID: \"1618b1d3-81ce-4def-9498-43b3fdbc46a4\") " pod="openshift-ingress/e2e-inf-a-ef1084-openshift-default-68989c6b9f-lrmtq"
Oct 30 01:00:45.815032 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:45.814896    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdxlq\" (UniqueName: \"kubernetes.io/projected/87ebc816-fbe5-4a3e-b650-39f2a72e9037-kube-api-access-xdxlq\") pod \"test-model-51b95e-kserve-5c4d8899f7-hc4c2\" (UID: \"87ebc816-fbe5-4a3e-b650-39f2a72e9037\") " pod="ai-tenant-e2e-inf-a-51b95e/test-model-51b95e-kserve-5c4d8899f7-hc4c2"
Oct 30 01:00:45.815134 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:45.815068    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1618b1d3-81ce-4def-9498-43b3fdbc46a4-workload-socket\") pod \"e2e-inf-a-ef1084-openshift-default-68989c6b9f-lrmtq\" (UID: \"1618b1d3-81ce-4def-9498-43b3fdbc46a4\") " pod="openshift-ingress/e2e-inf-a-ef1084-openshift-default-68989c6b9f-lrmtq"
Oct 30 01:00:45.816265 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:45.816201    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1618b1d3-81ce-4def-9498-43b3fdbc46a4-istiod-ca-cert\") pod \"e2e-inf-a-ef1084-openshift-default-68989c6b9f-lrmtq\" (UID: \"1618b1d3-81ce-4def-9498-43b3fdbc46a4\") " pod="openshift-ingress/e2e-inf-a-ef1084-openshift-default-68989c6b9f-lrmtq"
Oct 30 01:00:45.817148 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:45.817121    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1618b1d3-81ce-4def-9498-43b3fdbc46a4-istio-envoy\") pod \"e2e-inf-a-ef1084-openshift-default-68989c6b9f-lrmtq\" (UID: \"1618b1d3-81ce-4def-9498-43b3fdbc46a4\") " pod="openshift-ingress/e2e-inf-a-ef1084-openshift-default-68989c6b9f-lrmtq"
Oct 30 01:00:45.817871 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:45.817828    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1618b1d3-81ce-4def-9498-43b3fdbc46a4-istio-podinfo\") pod \"e2e-inf-a-ef1084-openshift-default-68989c6b9f-lrmtq\" (UID: \"1618b1d3-81ce-4def-9498-43b3fdbc46a4\") " pod="openshift-ingress/e2e-inf-a-ef1084-openshift-default-68989c6b9f-lrmtq"
Oct 30 01:00:45.840314 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:45.840292    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1618b1d3-81ce-4def-9498-43b3fdbc46a4-istio-token\") pod \"e2e-inf-a-ef1084-openshift-default-68989c6b9f-lrmtq\" (UID: \"1618b1d3-81ce-4def-9498-43b3fdbc46a4\") " pod="openshift-ingress/e2e-inf-a-ef1084-openshift-default-68989c6b9f-lrmtq"
Oct 30 01:00:45.840654 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:45.840630    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-txd2n\" (UniqueName: \"kubernetes.io/projected/1618b1d3-81ce-4def-9498-43b3fdbc46a4-kube-api-access-txd2n\") pod \"e2e-inf-a-ef1084-openshift-default-68989c6b9f-lrmtq\" (UID: \"1618b1d3-81ce-4def-9498-43b3fdbc46a4\") " pod="openshift-ingress/e2e-inf-a-ef1084-openshift-default-68989c6b9f-lrmtq"
Oct 30 01:00:45.868651 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:45.868621    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ef1084-openshift-default-68989c6b9f-lrmtq"
Oct 30 01:00:45.917401 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:45.916584    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87ebc816-fbe5-4a3e-b650-39f2a72e9037-tls-certs\") pod \"test-model-51b95e-kserve-5c4d8899f7-hc4c2\" (UID: \"87ebc816-fbe5-4a3e-b650-39f2a72e9037\") " pod="ai-tenant-e2e-inf-a-51b95e/test-model-51b95e-kserve-5c4d8899f7-hc4c2"
Oct 30 01:00:45.917401 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:45.916664    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdxlq\" (UniqueName: \"kubernetes.io/projected/87ebc816-fbe5-4a3e-b650-39f2a72e9037-kube-api-access-xdxlq\") pod \"test-model-51b95e-kserve-5c4d8899f7-hc4c2\" (UID: \"87ebc816-fbe5-4a3e-b650-39f2a72e9037\") " pod="ai-tenant-e2e-inf-a-51b95e/test-model-51b95e-kserve-5c4d8899f7-hc4c2"
Oct 30 01:00:45.917401 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:45.916720    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/87ebc816-fbe5-4a3e-b650-39f2a72e9037-home\") pod \"test-model-51b95e-kserve-5c4d8899f7-hc4c2\" (UID: \"87ebc816-fbe5-4a3e-b650-39f2a72e9037\") " pod="ai-tenant-e2e-inf-a-51b95e/test-model-51b95e-kserve-5c4d8899f7-hc4c2"
Oct 30 01:00:45.917401 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:45.916781    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/87ebc816-fbe5-4a3e-b650-39f2a72e9037-dshm\") pod \"test-model-51b95e-kserve-5c4d8899f7-hc4c2\" (UID: \"87ebc816-fbe5-4a3e-b650-39f2a72e9037\") " pod="ai-tenant-e2e-inf-a-51b95e/test-model-51b95e-kserve-5c4d8899f7-hc4c2"
Oct 30 01:00:45.917401 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:45.916811    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87ebc816-fbe5-4a3e-b650-39f2a72e9037-tmp-dir\") pod \"test-model-51b95e-kserve-5c4d8899f7-hc4c2\" (UID: \"87ebc816-fbe5-4a3e-b650-39f2a72e9037\") " pod="ai-tenant-e2e-inf-a-51b95e/test-model-51b95e-kserve-5c4d8899f7-hc4c2"
Oct 30 01:00:45.917401 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:45.917028    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87ebc816-fbe5-4a3e-b650-39f2a72e9037-model-cache\") pod \"test-model-51b95e-kserve-5c4d8899f7-hc4c2\" (UID: \"87ebc816-fbe5-4a3e-b650-39f2a72e9037\") " pod="ai-tenant-e2e-inf-a-51b95e/test-model-51b95e-kserve-5c4d8899f7-hc4c2"
Oct 30 01:00:45.917926 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:45.917613    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87ebc816-fbe5-4a3e-b650-39f2a72e9037-model-cache\") pod \"test-model-51b95e-kserve-5c4d8899f7-hc4c2\" (UID: \"87ebc816-fbe5-4a3e-b650-39f2a72e9037\") " pod="ai-tenant-e2e-inf-a-51b95e/test-model-51b95e-kserve-5c4d8899f7-hc4c2"
Oct 30 01:00:45.917966 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:45.917952    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/87ebc816-fbe5-4a3e-b650-39f2a72e9037-home\") pod \"test-model-51b95e-kserve-5c4d8899f7-hc4c2\" (UID: \"87ebc816-fbe5-4a3e-b650-39f2a72e9037\") " pod="ai-tenant-e2e-inf-a-51b95e/test-model-51b95e-kserve-5c4d8899f7-hc4c2"
Oct 30 01:00:45.918773 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:45.918581    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87ebc816-fbe5-4a3e-b650-39f2a72e9037-tmp-dir\") pod \"test-model-51b95e-kserve-5c4d8899f7-hc4c2\" (UID: \"87ebc816-fbe5-4a3e-b650-39f2a72e9037\") " pod="ai-tenant-e2e-inf-a-51b95e/test-model-51b95e-kserve-5c4d8899f7-hc4c2"
Oct 30 01:00:45.921767 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:45.921744    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87ebc816-fbe5-4a3e-b650-39f2a72e9037-tls-certs\") pod \"test-model-51b95e-kserve-5c4d8899f7-hc4c2\" (UID: \"87ebc816-fbe5-4a3e-b650-39f2a72e9037\") " pod="ai-tenant-e2e-inf-a-51b95e/test-model-51b95e-kserve-5c4d8899f7-hc4c2"
Oct 30 01:00:45.932418 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:45.928192    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/87ebc816-fbe5-4a3e-b650-39f2a72e9037-dshm\") pod \"test-model-51b95e-kserve-5c4d8899f7-hc4c2\" (UID: \"87ebc816-fbe5-4a3e-b650-39f2a72e9037\") " pod="ai-tenant-e2e-inf-a-51b95e/test-model-51b95e-kserve-5c4d8899f7-hc4c2"
Oct 30 01:00:45.998415 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:45.987119    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdxlq\" (UniqueName: \"kubernetes.io/projected/87ebc816-fbe5-4a3e-b650-39f2a72e9037-kube-api-access-xdxlq\") pod \"test-model-51b95e-kserve-5c4d8899f7-hc4c2\" (UID: \"87ebc816-fbe5-4a3e-b650-39f2a72e9037\") " pod="ai-tenant-e2e-inf-a-51b95e/test-model-51b95e-kserve-5c4d8899f7-hc4c2"
Oct 30 01:00:46.298503 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:46.283206    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-51b95e/test-model-51b95e-kserve-5c4d8899f7-hc4c2"
Oct 30 01:00:46.608205 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:46.608168    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-ef1084-openshift-default-68989c6b9f-lrmtq"]
Oct 30 01:00:46.720415 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:46.718892    5002 scope.go:122] "RemoveContainer" containerID="75b34039a14a9c761bf08a16681aa616314578152997eb8a3d36095952eddb15"
Oct 30 01:00:46.745948 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:46.744783    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:00:46.745948 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:46.744883    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:00:46.745948 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:46.744926    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:00:46.897496 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:46.868095    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-51b95e/test-model-51b95e-kserve-5c4d8899f7-hc4c2"]
Oct 30 01:00:47.078101 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:47.078068    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-51b95e/test-model-51b95e-kserve-5c4d8899f7-hc4c2" event={"ID":"87ebc816-fbe5-4a3e-b650-39f2a72e9037","Type":"ContainerStarted","Data":"843570c81ee3123d849a9a65337207d0b39140932a41429192e14930d72e36f8"}
Oct 30 01:00:47.083800 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:47.083278    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ef1084-openshift-default-68989c6b9f-lrmtq" event={"ID":"1618b1d3-81ce-4def-9498-43b3fdbc46a4","Type":"ContainerStarted","Data":"af116c343b1d7035544afdf077769fae2d76cf81a278e0ae944ab7b9a02adfc9"}
Oct 30 01:00:48.151819 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:48.151778    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-51b95e/test-model-51b95e-kserve-5c4d8899f7-hc4c2" event={"ID":"87ebc816-fbe5-4a3e-b650-39f2a72e9037","Type":"ContainerStarted","Data":"fff95db193bc2b29a8faf558fbbdf9e1e9b14c6e7b149d8a10051bba847bf629"}
Oct 30 01:00:48.156688 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:48.156654    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ef1084-openshift-default-68989c6b9f-lrmtq" event={"ID":"1618b1d3-81ce-4def-9498-43b3fdbc46a4","Type":"ContainerStarted","Data":"b9bb212d7256f99518451eacffb2043a7ef5781b2b01367473460ad8117c5575"}
Oct 30 01:00:48.237646 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:48.237587    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-51b95e/test-model-51b95e-kserve-5c4d8899f7-hc4c2" podStartSLOduration=3.2375666770000002 podStartE2EDuration="3.237566677s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:00:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:00:48.211227836 +0000 UTC m=+1743.827660716" watchObservedRunningTime="2026-10-30 01:00:48.237566677 +0000 UTC m=+1743.853999557"
Oct 30 01:00:48.264330 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:48.264269    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-ef1084-openshift-default-68989c6b9f-lrmtq" podStartSLOduration=3.264254312 podStartE2EDuration="3.264254312s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:00:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:00:48.262759668 +0000 UTC m=+1743.879192547" watchObservedRunningTime="2026-10-30 01:00:48.264254312 +0000 UTC m=+1743.880687191"
Oct 30 01:00:48.283903 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:48.283853    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-51b95e/test-model-51b95e-kserve-5c4d8899f7-hc4c2"
Oct 30 01:00:48.869885 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:48.869846    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-ef1084-openshift-default-68989c6b9f-lrmtq"
Oct 30 01:00:48.876995 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:48.876966    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-ef1084-openshift-default-68989c6b9f-lrmtq"
Oct 30 01:00:49.166220 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:49.166141    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-ef1084-openshift-default-68989c6b9f-lrmtq"
Oct 30 01:00:49.167285 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:49.167259    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-ef1084-openshift-default-68989c6b9f-lrmtq"
Oct 30 01:00:52.678093 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:52.678045    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ef1084-64c6bbc5ff-rrjq8"]
Oct 30 01:00:52.685582 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:52.685555    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ef1084-64c6bbc5ff-rrjq8"
Oct 30 01:00:52.699237 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:52.699208    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ef1084-64c6bbc5ff-rrjq8"]
Oct 30 01:00:52.707713 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:52.707632    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-ef1084\""
Oct 30 01:00:52.708436 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:52.708412    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-ef1084-dockercfg-d5zv7\""
Oct 30 01:00:52.797399 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:52.788035    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ef1084-66c68bf584-58x9z"]
Oct 30 01:00:52.821408 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:52.817815    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdpfz\" (UniqueName: \"kubernetes.io/projected/d405389a-c3e3-4aa9-a7db-ec4f32166b46-kube-api-access-xdpfz\") pod \"payload-pre-processing-e2e-inf-a-ef1084-64c6bbc5ff-rrjq8\" (UID: \"d405389a-c3e3-4aa9-a7db-ec4f32166b46\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ef1084-64c6bbc5ff-rrjq8"
Oct 30 01:00:52.821408 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:52.818000    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d405389a-c3e3-4aa9-a7db-ec4f32166b46-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-ef1084-64c6bbc5ff-rrjq8\" (UID: \"d405389a-c3e3-4aa9-a7db-ec4f32166b46\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ef1084-64c6bbc5ff-rrjq8"
Oct 30 01:00:52.968404 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:52.960005    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d405389a-c3e3-4aa9-a7db-ec4f32166b46-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-ef1084-64c6bbc5ff-rrjq8\" (UID: \"d405389a-c3e3-4aa9-a7db-ec4f32166b46\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ef1084-64c6bbc5ff-rrjq8"
Oct 30 01:00:52.968404 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:52.962324    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d405389a-c3e3-4aa9-a7db-ec4f32166b46-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-ef1084-64c6bbc5ff-rrjq8\" (UID: \"d405389a-c3e3-4aa9-a7db-ec4f32166b46\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ef1084-64c6bbc5ff-rrjq8"
Oct 30 01:00:52.968404 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:52.967877    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdpfz\" (UniqueName: \"kubernetes.io/projected/d405389a-c3e3-4aa9-a7db-ec4f32166b46-kube-api-access-xdpfz\") pod \"payload-pre-processing-e2e-inf-a-ef1084-64c6bbc5ff-rrjq8\" (UID: \"d405389a-c3e3-4aa9-a7db-ec4f32166b46\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ef1084-64c6bbc5ff-rrjq8"
Oct 30 01:00:52.986398 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:52.975362    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ef1084-66c68bf584-58x9z"]
Oct 30 01:00:52.986398 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:52.975503    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-ef1084-66c68bf584-58x9z"
Oct 30 01:00:53.064458 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:53.062049    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdpfz\" (UniqueName: \"kubernetes.io/projected/d405389a-c3e3-4aa9-a7db-ec4f32166b46-kube-api-access-xdpfz\") pod \"payload-pre-processing-e2e-inf-a-ef1084-64c6bbc5ff-rrjq8\" (UID: \"d405389a-c3e3-4aa9-a7db-ec4f32166b46\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ef1084-64c6bbc5ff-rrjq8"
Oct 30 01:00:53.085117 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:53.084607    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bd9ph\" (UniqueName: \"kubernetes.io/projected/fbe86602-c69c-4fff-b440-5e1ce9dd8e29-kube-api-access-bd9ph\") pod \"payload-processing-e2e-inf-a-ef1084-66c68bf584-58x9z\" (UID: \"fbe86602-c69c-4fff-b440-5e1ce9dd8e29\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ef1084-66c68bf584-58x9z"
Oct 30 01:00:53.085117 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:53.084779    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbe86602-c69c-4fff-b440-5e1ce9dd8e29-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-ef1084-66c68bf584-58x9z\" (UID: \"fbe86602-c69c-4fff-b440-5e1ce9dd8e29\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ef1084-66c68bf584-58x9z"
Oct 30 01:00:53.186936 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:53.186889    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbe86602-c69c-4fff-b440-5e1ce9dd8e29-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-ef1084-66c68bf584-58x9z\" (UID: \"fbe86602-c69c-4fff-b440-5e1ce9dd8e29\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ef1084-66c68bf584-58x9z"
Oct 30 01:00:53.187219 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:53.187198    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bd9ph\" (UniqueName: \"kubernetes.io/projected/fbe86602-c69c-4fff-b440-5e1ce9dd8e29-kube-api-access-bd9ph\") pod \"payload-processing-e2e-inf-a-ef1084-66c68bf584-58x9z\" (UID: \"fbe86602-c69c-4fff-b440-5e1ce9dd8e29\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ef1084-66c68bf584-58x9z"
Oct 30 01:00:53.188024 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:53.187993    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbe86602-c69c-4fff-b440-5e1ce9dd8e29-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-ef1084-66c68bf584-58x9z\" (UID: \"fbe86602-c69c-4fff-b440-5e1ce9dd8e29\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ef1084-66c68bf584-58x9z"
Oct 30 01:00:53.213440 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:53.213414    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bd9ph\" (UniqueName: \"kubernetes.io/projected/fbe86602-c69c-4fff-b440-5e1ce9dd8e29-kube-api-access-bd9ph\") pod \"payload-processing-e2e-inf-a-ef1084-66c68bf584-58x9z\" (UID: \"fbe86602-c69c-4fff-b440-5e1ce9dd8e29\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ef1084-66c68bf584-58x9z"
Oct 30 01:00:53.291752 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:53.291670    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-21ccafef-sr46p"]
Oct 30 01:00:53.296246 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:53.296219    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21ccafef-sr46p"
Oct 30 01:00:53.304198 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:53.304176    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ef1084-64c6bbc5ff-rrjq8"
Oct 30 01:00:53.314232 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:53.314211    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:00:53.314431 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:53.314257    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:00:53.314543 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:53.314350    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5phsh\""
Oct 30 01:00:53.314729 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:53.314699    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:00:53.316480 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:53.316341    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-21ccafef-sr46p"]
Oct 30 01:00:53.397454 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:53.396198    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a5c1b79-e06c-4fdc-ae81-694a1368e991-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-21ccafef-sr46p\" (UID: \"0a5c1b79-e06c-4fdc-ae81-694a1368e991\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21ccafef-sr46p"
Oct 30 01:00:53.397454 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:53.396513    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a5c1b79-e06c-4fdc-ae81-694a1368e991-maas-api-xks-ca\") pod \"maas-api-revoke-sub-21ccafef-sr46p\" (UID: \"0a5c1b79-e06c-4fdc-ae81-694a1368e991\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21ccafef-sr46p"
Oct 30 01:00:53.397454 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:53.396643    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbl87\" (UniqueName: \"kubernetes.io/projected/0a5c1b79-e06c-4fdc-ae81-694a1368e991-kube-api-access-cbl87\") pod \"maas-api-revoke-sub-21ccafef-sr46p\" (UID: \"0a5c1b79-e06c-4fdc-ae81-694a1368e991\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21ccafef-sr46p"
Oct 30 01:00:53.433514 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:53.431361    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-ef1084-66c68bf584-58x9z"
Oct 30 01:00:53.500418 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:53.498430    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a5c1b79-e06c-4fdc-ae81-694a1368e991-maas-api-xks-ca\") pod \"maas-api-revoke-sub-21ccafef-sr46p\" (UID: \"0a5c1b79-e06c-4fdc-ae81-694a1368e991\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21ccafef-sr46p"
Oct 30 01:00:53.500418 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:53.498561    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbl87\" (UniqueName: \"kubernetes.io/projected/0a5c1b79-e06c-4fdc-ae81-694a1368e991-kube-api-access-cbl87\") pod \"maas-api-revoke-sub-21ccafef-sr46p\" (UID: \"0a5c1b79-e06c-4fdc-ae81-694a1368e991\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21ccafef-sr46p"
Oct 30 01:00:53.500418 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:53.498656    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a5c1b79-e06c-4fdc-ae81-694a1368e991-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-21ccafef-sr46p\" (UID: \"0a5c1b79-e06c-4fdc-ae81-694a1368e991\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21ccafef-sr46p"
Oct 30 01:00:53.500418 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:53.499547    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a5c1b79-e06c-4fdc-ae81-694a1368e991-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-21ccafef-sr46p\" (UID: \"0a5c1b79-e06c-4fdc-ae81-694a1368e991\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21ccafef-sr46p"
Oct 30 01:00:53.502466 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:53.501706    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a5c1b79-e06c-4fdc-ae81-694a1368e991-maas-api-xks-ca\") pod \"maas-api-revoke-sub-21ccafef-sr46p\" (UID: \"0a5c1b79-e06c-4fdc-ae81-694a1368e991\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21ccafef-sr46p"
Oct 30 01:00:53.525606 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:53.525538    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbl87\" (UniqueName: \"kubernetes.io/projected/0a5c1b79-e06c-4fdc-ae81-694a1368e991-kube-api-access-cbl87\") pod \"maas-api-revoke-sub-21ccafef-sr46p\" (UID: \"0a5c1b79-e06c-4fdc-ae81-694a1368e991\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21ccafef-sr46p"
Oct 30 01:00:53.615910 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:53.615841    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21ccafef-sr46p"
Oct 30 01:00:53.658480 ip-10-0-128-7 kubenswrapper[5002]: W1030 01:00:53.658450    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd405389a_c3e3_4aa9_a7db_ec4f32166b46.slice/crio-299840d781d8a8468828ef6861986fb0af5e02f7b5c7599a8b8560274a72cf1b WatchSource:0}: Error finding container 299840d781d8a8468828ef6861986fb0af5e02f7b5c7599a8b8560274a72cf1b: Status 404 returned error can't find the container with id 299840d781d8a8468828ef6861986fb0af5e02f7b5c7599a8b8560274a72cf1b
Oct 30 01:00:53.658681 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:53.658656    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ef1084-64c6bbc5ff-rrjq8"]
Oct 30 01:00:53.760474 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:53.760440    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ef1084-66c68bf584-58x9z"]
Oct 30 01:00:53.933063 ip-10-0-128-7 kubenswrapper[5002]: W1030 01:00:53.933027    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a5c1b79_e06c_4fdc_ae81_694a1368e991.slice/crio-92c303c3e8406d7fe7e5ad0b65e99050212acb4a903e858485b11d4a1d241362 WatchSource:0}: Error finding container 92c303c3e8406d7fe7e5ad0b65e99050212acb4a903e858485b11d4a1d241362: Status 404 returned error can't find the container with id 92c303c3e8406d7fe7e5ad0b65e99050212acb4a903e858485b11d4a1d241362
Oct 30 01:00:53.934252 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:53.934223    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-21ccafef-sr46p"]
Oct 30 01:00:54.209919 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:54.209869    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ef1084-64c6bbc5ff-rrjq8" event={"ID":"d405389a-c3e3-4aa9-a7db-ec4f32166b46","Type":"ContainerStarted","Data":"3674e1f28451e5276b398306dd73863073fdddf1e620e90074a493f0754b2013"}
Oct 30 01:00:54.210134 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:54.209977    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ef1084-64c6bbc5ff-rrjq8"
Oct 30 01:00:54.210134 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:54.210003    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ef1084-64c6bbc5ff-rrjq8" event={"ID":"d405389a-c3e3-4aa9-a7db-ec4f32166b46","Type":"ContainerStarted","Data":"299840d781d8a8468828ef6861986fb0af5e02f7b5c7599a8b8560274a72cf1b"}
Oct 30 01:00:54.211549 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:54.211506    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21ccafef-sr46p" event={"ID":"0a5c1b79-e06c-4fdc-ae81-694a1368e991","Type":"ContainerStarted","Data":"92c303c3e8406d7fe7e5ad0b65e99050212acb4a903e858485b11d4a1d241362"}
Oct 30 01:00:54.213247 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:54.213220    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-ef1084-66c68bf584-58x9z" event={"ID":"fbe86602-c69c-4fff-b440-5e1ce9dd8e29","Type":"ContainerStarted","Data":"c9975e41fb734905a892f2391dacea87e78d1d17f742f146a44d1b91646a06b6"}
Oct 30 01:00:54.213404 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:54.213262    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-ef1084-66c68bf584-58x9z" event={"ID":"fbe86602-c69c-4fff-b440-5e1ce9dd8e29","Type":"ContainerStarted","Data":"892943be9b7d98e1085b9f6485ef884d7bbc6eff4fb195da5ac492c00a0395cc"}
Oct 30 01:00:54.213404 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:54.213352    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-ef1084-66c68bf584-58x9z"
Oct 30 01:00:54.254421 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:54.254355    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ef1084-64c6bbc5ff-rrjq8" podStartSLOduration=2.254338687 podStartE2EDuration="2.254338687s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:00:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:00:54.252886608 +0000 UTC m=+1749.869319489" watchObservedRunningTime="2026-10-30 01:00:54.254338687 +0000 UTC m=+1749.870771567"
Oct 30 01:00:54.299174 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:54.299116    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-ef1084-66c68bf584-58x9z" podStartSLOduration=2.299099277 podStartE2EDuration="2.299099277s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:00:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:00:54.296039335 +0000 UTC m=+1749.912472215" watchObservedRunningTime="2026-10-30 01:00:54.299099277 +0000 UTC m=+1749.915532157"
Oct 30 01:00:55.224010 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:55.223912    5002 generic.go:356] "Generic (PLEG): container finished" podID="0a5c1b79-e06c-4fdc-ae81-694a1368e991" containerID="43ca09206436b3afbd96407256020597f97989fd0be50f5bbc366cd58e853eee" exitCode=0
Oct 30 01:00:55.224010 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:55.223953    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21ccafef-sr46p" event={"ID":"0a5c1b79-e06c-4fdc-ae81-694a1368e991","Type":"ContainerDied","Data":"43ca09206436b3afbd96407256020597f97989fd0be50f5bbc366cd58e853eee"}
Oct 30 01:00:56.284053 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:56.284006    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-51b95e/test-model-51b95e-kserve-5c4d8899f7-hc4c2"
Oct 30 01:00:56.317837 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:56.301504    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-51b95e/test-model-51b95e-kserve-5c4d8899f7-hc4c2"
Oct 30 01:00:56.420636 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:56.420606    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21ccafef-sr46p"
Oct 30 01:00:56.580248 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:56.580138    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a5c1b79-e06c-4fdc-ae81-694a1368e991-kube-api-access-cbl87\" (UniqueName: \"kubernetes.io/projected/0a5c1b79-e06c-4fdc-ae81-694a1368e991-kube-api-access-cbl87\") pod \"0a5c1b79-e06c-4fdc-ae81-694a1368e991\" (UID: \"0a5c1b79-e06c-4fdc-ae81-694a1368e991\") "
Oct 30 01:00:56.580457 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:56.580249    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0a5c1b79-e06c-4fdc-ae81-694a1368e991-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a5c1b79-e06c-4fdc-ae81-694a1368e991-maas-api-xks-ca\") pod \"0a5c1b79-e06c-4fdc-ae81-694a1368e991\" (UID: \"0a5c1b79-e06c-4fdc-ae81-694a1368e991\") "
Oct 30 01:00:56.580457 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:56.580303    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a5c1b79-e06c-4fdc-ae81-694a1368e991-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a5c1b79-e06c-4fdc-ae81-694a1368e991-maas-api-openshift-ca\") pod \"0a5c1b79-e06c-4fdc-ae81-694a1368e991\" (UID: \"0a5c1b79-e06c-4fdc-ae81-694a1368e991\") "
Oct 30 01:00:56.580813 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:56.580785    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a5c1b79-e06c-4fdc-ae81-694a1368e991-maas-api-openshift-ca" pod "0a5c1b79-e06c-4fdc-ae81-694a1368e991" (UID: "0a5c1b79-e06c-4fdc-ae81-694a1368e991"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:00:56.581327 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:56.581303    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a5c1b79-e06c-4fdc-ae81-694a1368e991-maas-api-openshift-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:00:56.582806 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:56.582773    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a5c1b79-e06c-4fdc-ae81-694a1368e991-maas-api-xks-ca" pod "0a5c1b79-e06c-4fdc-ae81-694a1368e991" (UID: "0a5c1b79-e06c-4fdc-ae81-694a1368e991"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:00:56.582930 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:56.582905    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a5c1b79-e06c-4fdc-ae81-694a1368e991-kube-api-access-cbl87" pod "0a5c1b79-e06c-4fdc-ae81-694a1368e991" (UID: "0a5c1b79-e06c-4fdc-ae81-694a1368e991"). InnerVolumeSpecName "kube-api-access-cbl87". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:00:56.683162 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:56.683122    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cbl87\" (UniqueName: \"kubernetes.io/projected/0a5c1b79-e06c-4fdc-ae81-694a1368e991-kube-api-access-cbl87\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:00:56.683162 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:56.683161    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a5c1b79-e06c-4fdc-ae81-694a1368e991-maas-api-xks-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:00:57.241423 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:57.241280    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21ccafef-sr46p"
Oct 30 01:00:57.241423 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:57.241316    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21ccafef-sr46p" event={"ID":"0a5c1b79-e06c-4fdc-ae81-694a1368e991","Type":"ContainerDied","Data":"92c303c3e8406d7fe7e5ad0b65e99050212acb4a903e858485b11d4a1d241362"}
Oct 30 01:00:57.241423 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:57.241391    5002 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="92c303c3e8406d7fe7e5ad0b65e99050212acb4a903e858485b11d4a1d241362"
Oct 30 01:00:57.253927 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:00:57.253903    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-51b95e/test-model-51b95e-kserve-5c4d8899f7-hc4c2"
Oct 30 01:01:05.228228 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:05.228197    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ef1084-64c6bbc5ff-rrjq8"
Oct 30 01:01:05.228716 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:05.228282    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-ef1084-66c68bf584-58x9z"
Oct 30 01:01:09.029624 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:09.029587    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-8dbe73-openshift-default-6c44d95ddf-5hz6q"]
Oct 30 01:01:09.030338 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:09.030320    5002 cpu_manager.go:414] "containerMap: removing container" podUID="0a5c1b79-e06c-4fdc-ae81-694a1368e991" containerName="revoke-keys"
Oct 30 01:01:09.030444 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:09.030339    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a5c1b79-e06c-4fdc-ae81-694a1368e991" containerName="revoke-keys"
Oct 30 01:01:09.030502 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:09.030493    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a5c1b79-e06c-4fdc-ae81-694a1368e991" containerName="revoke-keys"
Oct 30 01:01:09.034189 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:09.034172    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-8dbe73-openshift-default-6c44d95ddf-5hz6q"
Oct 30 01:01:09.076556 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:09.053682    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-8dbe73-openshift-default-dockercfg-hjfpm\""
Oct 30 01:01:09.078736 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:09.078705    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-8dbe73-openshift-default-6c44d95ddf-5hz6q"]
Oct 30 01:01:09.177329 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:09.177266    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dsqj6\" (UniqueName: \"kubernetes.io/projected/1383e696-ad5a-4b3f-900c-a987399f6bec-kube-api-access-dsqj6\") pod \"e2e-inf-a-8dbe73-openshift-default-6c44d95ddf-5hz6q\" (UID: \"1383e696-ad5a-4b3f-900c-a987399f6bec\") " pod="openshift-ingress/e2e-inf-a-8dbe73-openshift-default-6c44d95ddf-5hz6q"
Oct 30 01:01:09.177533 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:09.177440    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1383e696-ad5a-4b3f-900c-a987399f6bec-istio-podinfo\") pod \"e2e-inf-a-8dbe73-openshift-default-6c44d95ddf-5hz6q\" (UID: \"1383e696-ad5a-4b3f-900c-a987399f6bec\") " pod="openshift-ingress/e2e-inf-a-8dbe73-openshift-default-6c44d95ddf-5hz6q"
Oct 30 01:01:09.177633 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:09.177608    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1383e696-ad5a-4b3f-900c-a987399f6bec-workload-certs\") pod \"e2e-inf-a-8dbe73-openshift-default-6c44d95ddf-5hz6q\" (UID: \"1383e696-ad5a-4b3f-900c-a987399f6bec\") " pod="openshift-ingress/e2e-inf-a-8dbe73-openshift-default-6c44d95ddf-5hz6q"
Oct 30 01:01:09.177697 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:09.177661    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1383e696-ad5a-4b3f-900c-a987399f6bec-credential-socket\") pod \"e2e-inf-a-8dbe73-openshift-default-6c44d95ddf-5hz6q\" (UID: \"1383e696-ad5a-4b3f-900c-a987399f6bec\") " pod="openshift-ingress/e2e-inf-a-8dbe73-openshift-default-6c44d95ddf-5hz6q"
Oct 30 01:01:09.177768 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:09.177747    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1383e696-ad5a-4b3f-900c-a987399f6bec-istio-envoy\") pod \"e2e-inf-a-8dbe73-openshift-default-6c44d95ddf-5hz6q\" (UID: \"1383e696-ad5a-4b3f-900c-a987399f6bec\") " pod="openshift-ingress/e2e-inf-a-8dbe73-openshift-default-6c44d95ddf-5hz6q"
Oct 30 01:01:09.177869 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:09.177849    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1383e696-ad5a-4b3f-900c-a987399f6bec-istiod-ca-cert\") pod \"e2e-inf-a-8dbe73-openshift-default-6c44d95ddf-5hz6q\" (UID: \"1383e696-ad5a-4b3f-900c-a987399f6bec\") " pod="openshift-ingress/e2e-inf-a-8dbe73-openshift-default-6c44d95ddf-5hz6q"
Oct 30 01:01:09.177917 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:09.177879    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1383e696-ad5a-4b3f-900c-a987399f6bec-workload-socket\") pod \"e2e-inf-a-8dbe73-openshift-default-6c44d95ddf-5hz6q\" (UID: \"1383e696-ad5a-4b3f-900c-a987399f6bec\") " pod="openshift-ingress/e2e-inf-a-8dbe73-openshift-default-6c44d95ddf-5hz6q"
Oct 30 01:01:09.177917 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:09.177905    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1383e696-ad5a-4b3f-900c-a987399f6bec-istio-data\") pod \"e2e-inf-a-8dbe73-openshift-default-6c44d95ddf-5hz6q\" (UID: \"1383e696-ad5a-4b3f-900c-a987399f6bec\") " pod="openshift-ingress/e2e-inf-a-8dbe73-openshift-default-6c44d95ddf-5hz6q"
Oct 30 01:01:09.178135 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:09.178110    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1383e696-ad5a-4b3f-900c-a987399f6bec-istio-token\") pod \"e2e-inf-a-8dbe73-openshift-default-6c44d95ddf-5hz6q\" (UID: \"1383e696-ad5a-4b3f-900c-a987399f6bec\") " pod="openshift-ingress/e2e-inf-a-8dbe73-openshift-default-6c44d95ddf-5hz6q"
Oct 30 01:01:09.279885 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:09.279776    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dsqj6\" (UniqueName: \"kubernetes.io/projected/1383e696-ad5a-4b3f-900c-a987399f6bec-kube-api-access-dsqj6\") pod \"e2e-inf-a-8dbe73-openshift-default-6c44d95ddf-5hz6q\" (UID: \"1383e696-ad5a-4b3f-900c-a987399f6bec\") " pod="openshift-ingress/e2e-inf-a-8dbe73-openshift-default-6c44d95ddf-5hz6q"
Oct 30 01:01:09.279885 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:09.279864    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1383e696-ad5a-4b3f-900c-a987399f6bec-istio-podinfo\") pod \"e2e-inf-a-8dbe73-openshift-default-6c44d95ddf-5hz6q\" (UID: \"1383e696-ad5a-4b3f-900c-a987399f6bec\") " pod="openshift-ingress/e2e-inf-a-8dbe73-openshift-default-6c44d95ddf-5hz6q"
Oct 30 01:01:09.280114 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:09.279984    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1383e696-ad5a-4b3f-900c-a987399f6bec-workload-certs\") pod \"e2e-inf-a-8dbe73-openshift-default-6c44d95ddf-5hz6q\" (UID: \"1383e696-ad5a-4b3f-900c-a987399f6bec\") " pod="openshift-ingress/e2e-inf-a-8dbe73-openshift-default-6c44d95ddf-5hz6q"
Oct 30 01:01:09.280114 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:09.280022    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1383e696-ad5a-4b3f-900c-a987399f6bec-credential-socket\") pod \"e2e-inf-a-8dbe73-openshift-default-6c44d95ddf-5hz6q\" (UID: \"1383e696-ad5a-4b3f-900c-a987399f6bec\") " pod="openshift-ingress/e2e-inf-a-8dbe73-openshift-default-6c44d95ddf-5hz6q"
Oct 30 01:01:09.280114 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:09.280057    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1383e696-ad5a-4b3f-900c-a987399f6bec-istio-envoy\") pod \"e2e-inf-a-8dbe73-openshift-default-6c44d95ddf-5hz6q\" (UID: \"1383e696-ad5a-4b3f-900c-a987399f6bec\") " pod="openshift-ingress/e2e-inf-a-8dbe73-openshift-default-6c44d95ddf-5hz6q"
Oct 30 01:01:09.280292 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:09.280154    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1383e696-ad5a-4b3f-900c-a987399f6bec-istiod-ca-cert\") pod \"e2e-inf-a-8dbe73-openshift-default-6c44d95ddf-5hz6q\" (UID: \"1383e696-ad5a-4b3f-900c-a987399f6bec\") " pod="openshift-ingress/e2e-inf-a-8dbe73-openshift-default-6c44d95ddf-5hz6q"
Oct 30 01:01:09.280292 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:09.280180    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1383e696-ad5a-4b3f-900c-a987399f6bec-workload-socket\") pod \"e2e-inf-a-8dbe73-openshift-default-6c44d95ddf-5hz6q\" (UID: \"1383e696-ad5a-4b3f-900c-a987399f6bec\") " pod="openshift-ingress/e2e-inf-a-8dbe73-openshift-default-6c44d95ddf-5hz6q"
Oct 30 01:01:09.280292 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:09.280215    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1383e696-ad5a-4b3f-900c-a987399f6bec-istio-data\") pod \"e2e-inf-a-8dbe73-openshift-default-6c44d95ddf-5hz6q\" (UID: \"1383e696-ad5a-4b3f-900c-a987399f6bec\") " pod="openshift-ingress/e2e-inf-a-8dbe73-openshift-default-6c44d95ddf-5hz6q"
Oct 30 01:01:09.280505 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:09.280337    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1383e696-ad5a-4b3f-900c-a987399f6bec-istio-token\") pod \"e2e-inf-a-8dbe73-openshift-default-6c44d95ddf-5hz6q\" (UID: \"1383e696-ad5a-4b3f-900c-a987399f6bec\") " pod="openshift-ingress/e2e-inf-a-8dbe73-openshift-default-6c44d95ddf-5hz6q"
Oct 30 01:01:09.280920 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:09.280886    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1383e696-ad5a-4b3f-900c-a987399f6bec-workload-socket\") pod \"e2e-inf-a-8dbe73-openshift-default-6c44d95ddf-5hz6q\" (UID: \"1383e696-ad5a-4b3f-900c-a987399f6bec\") " pod="openshift-ingress/e2e-inf-a-8dbe73-openshift-default-6c44d95ddf-5hz6q"
Oct 30 01:01:09.281045 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:09.280572    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1383e696-ad5a-4b3f-900c-a987399f6bec-credential-socket\") pod \"e2e-inf-a-8dbe73-openshift-default-6c44d95ddf-5hz6q\" (UID: \"1383e696-ad5a-4b3f-900c-a987399f6bec\") " pod="openshift-ingress/e2e-inf-a-8dbe73-openshift-default-6c44d95ddf-5hz6q"
Oct 30 01:01:09.281045 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:09.280900    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1383e696-ad5a-4b3f-900c-a987399f6bec-istio-data\") pod \"e2e-inf-a-8dbe73-openshift-default-6c44d95ddf-5hz6q\" (UID: \"1383e696-ad5a-4b3f-900c-a987399f6bec\") " pod="openshift-ingress/e2e-inf-a-8dbe73-openshift-default-6c44d95ddf-5hz6q"
Oct 30 01:01:09.281184 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:09.281044    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1383e696-ad5a-4b3f-900c-a987399f6bec-istiod-ca-cert\") pod \"e2e-inf-a-8dbe73-openshift-default-6c44d95ddf-5hz6q\" (UID: \"1383e696-ad5a-4b3f-900c-a987399f6bec\") " pod="openshift-ingress/e2e-inf-a-8dbe73-openshift-default-6c44d95ddf-5hz6q"
Oct 30 01:01:09.281262 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:09.281237    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1383e696-ad5a-4b3f-900c-a987399f6bec-workload-certs\") pod \"e2e-inf-a-8dbe73-openshift-default-6c44d95ddf-5hz6q\" (UID: \"1383e696-ad5a-4b3f-900c-a987399f6bec\") " pod="openshift-ingress/e2e-inf-a-8dbe73-openshift-default-6c44d95ddf-5hz6q"
Oct 30 01:01:09.282713 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:09.282689    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1383e696-ad5a-4b3f-900c-a987399f6bec-istio-envoy\") pod \"e2e-inf-a-8dbe73-openshift-default-6c44d95ddf-5hz6q\" (UID: \"1383e696-ad5a-4b3f-900c-a987399f6bec\") " pod="openshift-ingress/e2e-inf-a-8dbe73-openshift-default-6c44d95ddf-5hz6q"
Oct 30 01:01:09.282821 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:09.282757    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1383e696-ad5a-4b3f-900c-a987399f6bec-istio-podinfo\") pod \"e2e-inf-a-8dbe73-openshift-default-6c44d95ddf-5hz6q\" (UID: \"1383e696-ad5a-4b3f-900c-a987399f6bec\") " pod="openshift-ingress/e2e-inf-a-8dbe73-openshift-default-6c44d95ddf-5hz6q"
Oct 30 01:01:09.305248 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:09.305213    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dsqj6\" (UniqueName: \"kubernetes.io/projected/1383e696-ad5a-4b3f-900c-a987399f6bec-kube-api-access-dsqj6\") pod \"e2e-inf-a-8dbe73-openshift-default-6c44d95ddf-5hz6q\" (UID: \"1383e696-ad5a-4b3f-900c-a987399f6bec\") " pod="openshift-ingress/e2e-inf-a-8dbe73-openshift-default-6c44d95ddf-5hz6q"
Oct 30 01:01:09.305446 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:09.305399    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1383e696-ad5a-4b3f-900c-a987399f6bec-istio-token\") pod \"e2e-inf-a-8dbe73-openshift-default-6c44d95ddf-5hz6q\" (UID: \"1383e696-ad5a-4b3f-900c-a987399f6bec\") " pod="openshift-ingress/e2e-inf-a-8dbe73-openshift-default-6c44d95ddf-5hz6q"
Oct 30 01:01:09.349102 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:09.349067    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-8dbe73-openshift-default-6c44d95ddf-5hz6q"
Oct 30 01:01:09.698753 ip-10-0-128-7 kubenswrapper[5002]: W1030 01:01:09.698718    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1383e696_ad5a_4b3f_900c_a987399f6bec.slice/crio-470bcad9733b4a16022f43deb6b5d8a2dda925800a8bd10ebe24591f7f9cc865 WatchSource:0}: Error finding container 470bcad9733b4a16022f43deb6b5d8a2dda925800a8bd10ebe24591f7f9cc865: Status 404 returned error can't find the container with id 470bcad9733b4a16022f43deb6b5d8a2dda925800a8bd10ebe24591f7f9cc865
Oct 30 01:01:09.701446 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:09.701407    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-8dbe73-openshift-default-6c44d95ddf-5hz6q"]
Oct 30 01:01:09.701748 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:09.701707    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:01:09.701797 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:09.701785    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:01:09.701830 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:09.701816    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:01:10.404014 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:10.403971    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8dbe73-openshift-default-6c44d95ddf-5hz6q" event={"ID":"1383e696-ad5a-4b3f-900c-a987399f6bec","Type":"ContainerStarted","Data":"fda3e2253aa699de01587ab3951226395d7f6cecaa8da8b485e032a93e57be3b"}
Oct 30 01:01:10.404478 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:10.404037    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8dbe73-openshift-default-6c44d95ddf-5hz6q" event={"ID":"1383e696-ad5a-4b3f-900c-a987399f6bec","Type":"ContainerStarted","Data":"470bcad9733b4a16022f43deb6b5d8a2dda925800a8bd10ebe24591f7f9cc865"}
Oct 30 01:01:11.370404 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:11.369998    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-8dbe73-openshift-default-6c44d95ddf-5hz6q"
Oct 30 01:01:11.383644 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:11.375500    5002 patch_prober.go:29] interesting pod/e2e-inf-a-8dbe73-openshift-default-6c44d95ddf-5hz6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body=
Oct 30 01:01:11.383644 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:11.375565    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8dbe73-openshift-default-6c44d95ddf-5hz6q" podUID="1383e696-ad5a-4b3f-900c-a987399f6bec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused"
Oct 30 01:01:12.360462 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:12.358480    5002 patch_prober.go:29] interesting pod/e2e-inf-a-8dbe73-openshift-default-6c44d95ddf-5hz6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body=
Oct 30 01:01:12.360462 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:12.358545    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8dbe73-openshift-default-6c44d95ddf-5hz6q" podUID="1383e696-ad5a-4b3f-900c-a987399f6bec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused"
Oct 30 01:01:12.891913 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:01:12.891872    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbjdq" containerName="extractor"
Oct 30 01:01:13.353757 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:13.353718    5002 patch_prober.go:29] interesting pod/e2e-inf-a-8dbe73-openshift-default-6c44d95ddf-5hz6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body=
Oct 30 01:01:13.354091 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:13.354060    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8dbe73-openshift-default-6c44d95ddf-5hz6q" podUID="1383e696-ad5a-4b3f-900c-a987399f6bec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused"
Oct 30 01:01:13.683298 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:13.683235    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-8dbe73-openshift-default-6c44d95ddf-5hz6q" podStartSLOduration=5.683212636 podStartE2EDuration="5.683212636s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:01:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:01:10.453659885 +0000 UTC m=+1766.070092763" watchObservedRunningTime="2026-10-30 01:01:13.683212636 +0000 UTC m=+1769.299645518"
Oct 30 01:01:13.690416 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:13.690371    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-03d892-openshift-default-6d4cdfbfd9-qs6xc"]
Oct 30 01:01:13.695914 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:13.695892    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-03d892-openshift-default-6d4cdfbfd9-qs6xc"
Oct 30 01:01:13.710781 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:13.710758    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-03d892-openshift-default-6d4cdfbfd9-qs6xc"]
Oct 30 01:01:13.719314 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:13.719176    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-03d892-openshift-default-dockercfg-crhvb\""
Oct 30 01:01:13.793136 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:13.793094    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d24afeb-7f16-4e65-b159-9e29b3f32c24-credential-socket\") pod \"e2e-inf-b-03d892-openshift-default-6d4cdfbfd9-qs6xc\" (UID: \"7d24afeb-7f16-4e65-b159-9e29b3f32c24\") " pod="openshift-ingress/e2e-inf-b-03d892-openshift-default-6d4cdfbfd9-qs6xc"
Oct 30 01:01:13.793351 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:13.793147    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d24afeb-7f16-4e65-b159-9e29b3f32c24-istio-envoy\") pod \"e2e-inf-b-03d892-openshift-default-6d4cdfbfd9-qs6xc\" (UID: \"7d24afeb-7f16-4e65-b159-9e29b3f32c24\") " pod="openshift-ingress/e2e-inf-b-03d892-openshift-default-6d4cdfbfd9-qs6xc"
Oct 30 01:01:13.793351 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:13.793237    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d24afeb-7f16-4e65-b159-9e29b3f32c24-workload-certs\") pod \"e2e-inf-b-03d892-openshift-default-6d4cdfbfd9-qs6xc\" (UID: \"7d24afeb-7f16-4e65-b159-9e29b3f32c24\") " pod="openshift-ingress/e2e-inf-b-03d892-openshift-default-6d4cdfbfd9-qs6xc"
Oct 30 01:01:13.793351 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:13.793308    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d24afeb-7f16-4e65-b159-9e29b3f32c24-istio-token\") pod \"e2e-inf-b-03d892-openshift-default-6d4cdfbfd9-qs6xc\" (UID: \"7d24afeb-7f16-4e65-b159-9e29b3f32c24\") " pod="openshift-ingress/e2e-inf-b-03d892-openshift-default-6d4cdfbfd9-qs6xc"
Oct 30 01:01:13.793548 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:13.793457    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d24afeb-7f16-4e65-b159-9e29b3f32c24-istio-data\") pod \"e2e-inf-b-03d892-openshift-default-6d4cdfbfd9-qs6xc\" (UID: \"7d24afeb-7f16-4e65-b159-9e29b3f32c24\") " pod="openshift-ingress/e2e-inf-b-03d892-openshift-default-6d4cdfbfd9-qs6xc"
Oct 30 01:01:13.793548 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:13.793513    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjql8\" (UniqueName: \"kubernetes.io/projected/7d24afeb-7f16-4e65-b159-9e29b3f32c24-kube-api-access-zjql8\") pod \"e2e-inf-b-03d892-openshift-default-6d4cdfbfd9-qs6xc\" (UID: \"7d24afeb-7f16-4e65-b159-9e29b3f32c24\") " pod="openshift-ingress/e2e-inf-b-03d892-openshift-default-6d4cdfbfd9-qs6xc"
Oct 30 01:01:13.793665 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:13.793545    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d24afeb-7f16-4e65-b159-9e29b3f32c24-workload-socket\") pod \"e2e-inf-b-03d892-openshift-default-6d4cdfbfd9-qs6xc\" (UID: \"7d24afeb-7f16-4e65-b159-9e29b3f32c24\") " pod="openshift-ingress/e2e-inf-b-03d892-openshift-default-6d4cdfbfd9-qs6xc"
Oct 30 01:01:13.793665 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:13.793655    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d24afeb-7f16-4e65-b159-9e29b3f32c24-istio-podinfo\") pod \"e2e-inf-b-03d892-openshift-default-6d4cdfbfd9-qs6xc\" (UID: \"7d24afeb-7f16-4e65-b159-9e29b3f32c24\") " pod="openshift-ingress/e2e-inf-b-03d892-openshift-default-6d4cdfbfd9-qs6xc"
Oct 30 01:01:13.793772 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:13.793679    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d24afeb-7f16-4e65-b159-9e29b3f32c24-istiod-ca-cert\") pod \"e2e-inf-b-03d892-openshift-default-6d4cdfbfd9-qs6xc\" (UID: \"7d24afeb-7f16-4e65-b159-9e29b3f32c24\") " pod="openshift-ingress/e2e-inf-b-03d892-openshift-default-6d4cdfbfd9-qs6xc"
Oct 30 01:01:13.895226 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:13.895180    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d24afeb-7f16-4e65-b159-9e29b3f32c24-istio-data\") pod \"e2e-inf-b-03d892-openshift-default-6d4cdfbfd9-qs6xc\" (UID: \"7d24afeb-7f16-4e65-b159-9e29b3f32c24\") " pod="openshift-ingress/e2e-inf-b-03d892-openshift-default-6d4cdfbfd9-qs6xc"
Oct 30 01:01:13.895483 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:13.895245    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjql8\" (UniqueName: \"kubernetes.io/projected/7d24afeb-7f16-4e65-b159-9e29b3f32c24-kube-api-access-zjql8\") pod \"e2e-inf-b-03d892-openshift-default-6d4cdfbfd9-qs6xc\" (UID: \"7d24afeb-7f16-4e65-b159-9e29b3f32c24\") " pod="openshift-ingress/e2e-inf-b-03d892-openshift-default-6d4cdfbfd9-qs6xc"
Oct 30 01:01:13.895483 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:13.895268    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d24afeb-7f16-4e65-b159-9e29b3f32c24-workload-socket\") pod \"e2e-inf-b-03d892-openshift-default-6d4cdfbfd9-qs6xc\" (UID: \"7d24afeb-7f16-4e65-b159-9e29b3f32c24\") " pod="openshift-ingress/e2e-inf-b-03d892-openshift-default-6d4cdfbfd9-qs6xc"
Oct 30 01:01:13.895483 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:13.895339    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d24afeb-7f16-4e65-b159-9e29b3f32c24-istio-podinfo\") pod \"e2e-inf-b-03d892-openshift-default-6d4cdfbfd9-qs6xc\" (UID: \"7d24afeb-7f16-4e65-b159-9e29b3f32c24\") " pod="openshift-ingress/e2e-inf-b-03d892-openshift-default-6d4cdfbfd9-qs6xc"
Oct 30 01:01:13.895483 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:13.895356    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d24afeb-7f16-4e65-b159-9e29b3f32c24-istiod-ca-cert\") pod \"e2e-inf-b-03d892-openshift-default-6d4cdfbfd9-qs6xc\" (UID: \"7d24afeb-7f16-4e65-b159-9e29b3f32c24\") " pod="openshift-ingress/e2e-inf-b-03d892-openshift-default-6d4cdfbfd9-qs6xc"
Oct 30 01:01:13.895483 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:13.895425    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d24afeb-7f16-4e65-b159-9e29b3f32c24-credential-socket\") pod \"e2e-inf-b-03d892-openshift-default-6d4cdfbfd9-qs6xc\" (UID: \"7d24afeb-7f16-4e65-b159-9e29b3f32c24\") " pod="openshift-ingress/e2e-inf-b-03d892-openshift-default-6d4cdfbfd9-qs6xc"
Oct 30 01:01:13.895483 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:13.895443    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d24afeb-7f16-4e65-b159-9e29b3f32c24-istio-envoy\") pod \"e2e-inf-b-03d892-openshift-default-6d4cdfbfd9-qs6xc\" (UID: \"7d24afeb-7f16-4e65-b159-9e29b3f32c24\") " pod="openshift-ingress/e2e-inf-b-03d892-openshift-default-6d4cdfbfd9-qs6xc"
Oct 30 01:01:13.895810 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:13.895515    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d24afeb-7f16-4e65-b159-9e29b3f32c24-workload-certs\") pod \"e2e-inf-b-03d892-openshift-default-6d4cdfbfd9-qs6xc\" (UID: \"7d24afeb-7f16-4e65-b159-9e29b3f32c24\") " pod="openshift-ingress/e2e-inf-b-03d892-openshift-default-6d4cdfbfd9-qs6xc"
Oct 30 01:01:13.895810 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:13.895569    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d24afeb-7f16-4e65-b159-9e29b3f32c24-istio-token\") pod \"e2e-inf-b-03d892-openshift-default-6d4cdfbfd9-qs6xc\" (UID: \"7d24afeb-7f16-4e65-b159-9e29b3f32c24\") " pod="openshift-ingress/e2e-inf-b-03d892-openshift-default-6d4cdfbfd9-qs6xc"
Oct 30 01:01:13.895810 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:13.895672    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d24afeb-7f16-4e65-b159-9e29b3f32c24-istio-data\") pod \"e2e-inf-b-03d892-openshift-default-6d4cdfbfd9-qs6xc\" (UID: \"7d24afeb-7f16-4e65-b159-9e29b3f32c24\") " pod="openshift-ingress/e2e-inf-b-03d892-openshift-default-6d4cdfbfd9-qs6xc"
Oct 30 01:01:13.895984 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:13.895962    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d24afeb-7f16-4e65-b159-9e29b3f32c24-credential-socket\") pod \"e2e-inf-b-03d892-openshift-default-6d4cdfbfd9-qs6xc\" (UID: \"7d24afeb-7f16-4e65-b159-9e29b3f32c24\") " pod="openshift-ingress/e2e-inf-b-03d892-openshift-default-6d4cdfbfd9-qs6xc"
Oct 30 01:01:13.896630 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:13.896581    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d24afeb-7f16-4e65-b159-9e29b3f32c24-istiod-ca-cert\") pod \"e2e-inf-b-03d892-openshift-default-6d4cdfbfd9-qs6xc\" (UID: \"7d24afeb-7f16-4e65-b159-9e29b3f32c24\") " pod="openshift-ingress/e2e-inf-b-03d892-openshift-default-6d4cdfbfd9-qs6xc"
Oct 30 01:01:13.897052 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:13.897019    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d24afeb-7f16-4e65-b159-9e29b3f32c24-workload-certs\") pod \"e2e-inf-b-03d892-openshift-default-6d4cdfbfd9-qs6xc\" (UID: \"7d24afeb-7f16-4e65-b159-9e29b3f32c24\") " pod="openshift-ingress/e2e-inf-b-03d892-openshift-default-6d4cdfbfd9-qs6xc"
Oct 30 01:01:13.897302 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:13.897289    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d24afeb-7f16-4e65-b159-9e29b3f32c24-workload-socket\") pod \"e2e-inf-b-03d892-openshift-default-6d4cdfbfd9-qs6xc\" (UID: \"7d24afeb-7f16-4e65-b159-9e29b3f32c24\") " pod="openshift-ingress/e2e-inf-b-03d892-openshift-default-6d4cdfbfd9-qs6xc"
Oct 30 01:01:13.898790 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:13.898774    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d24afeb-7f16-4e65-b159-9e29b3f32c24-istio-envoy\") pod \"e2e-inf-b-03d892-openshift-default-6d4cdfbfd9-qs6xc\" (UID: \"7d24afeb-7f16-4e65-b159-9e29b3f32c24\") " pod="openshift-ingress/e2e-inf-b-03d892-openshift-default-6d4cdfbfd9-qs6xc"
Oct 30 01:01:13.900154 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:13.900105    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d24afeb-7f16-4e65-b159-9e29b3f32c24-istio-podinfo\") pod \"e2e-inf-b-03d892-openshift-default-6d4cdfbfd9-qs6xc\" (UID: \"7d24afeb-7f16-4e65-b159-9e29b3f32c24\") " pod="openshift-ingress/e2e-inf-b-03d892-openshift-default-6d4cdfbfd9-qs6xc"
Oct 30 01:01:13.921171 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:13.921142    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d24afeb-7f16-4e65-b159-9e29b3f32c24-istio-token\") pod \"e2e-inf-b-03d892-openshift-default-6d4cdfbfd9-qs6xc\" (UID: \"7d24afeb-7f16-4e65-b159-9e29b3f32c24\") " pod="openshift-ingress/e2e-inf-b-03d892-openshift-default-6d4cdfbfd9-qs6xc"
Oct 30 01:01:13.921886 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:13.921845    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjql8\" (UniqueName: \"kubernetes.io/projected/7d24afeb-7f16-4e65-b159-9e29b3f32c24-kube-api-access-zjql8\") pod \"e2e-inf-b-03d892-openshift-default-6d4cdfbfd9-qs6xc\" (UID: \"7d24afeb-7f16-4e65-b159-9e29b3f32c24\") " pod="openshift-ingress/e2e-inf-b-03d892-openshift-default-6d4cdfbfd9-qs6xc"
Oct 30 01:01:14.016280 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:14.016192    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-03d892-openshift-default-6d4cdfbfd9-qs6xc"
Oct 30 01:01:14.388777 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:14.380504    5002 patch_prober.go:29] interesting pod/e2e-inf-a-8dbe73-openshift-default-6c44d95ddf-5hz6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body=
Oct 30 01:01:14.388777 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:14.380564    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8dbe73-openshift-default-6c44d95ddf-5hz6q" podUID="1383e696-ad5a-4b3f-900c-a987399f6bec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused"
Oct 30 01:01:14.388777 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:14.388486    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-03d892-openshift-default-6d4cdfbfd9-qs6xc"]
Oct 30 01:01:14.390997 ip-10-0-128-7 kubenswrapper[5002]: W1030 01:01:14.390840    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d24afeb_7f16_4e65_b159_9e29b3f32c24.slice/crio-e006575e708dddff37a3ccb7520b472cfcc8c3987a1427e71d62d7bf24b4f5bb WatchSource:0}: Error finding container e006575e708dddff37a3ccb7520b472cfcc8c3987a1427e71d62d7bf24b4f5bb: Status 404 returned error can't find the container with id e006575e708dddff37a3ccb7520b472cfcc8c3987a1427e71d62d7bf24b4f5bb
Oct 30 01:01:14.395177 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:14.395137    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:01:14.395275 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:14.395231    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:01:14.395313 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:14.395291    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:01:14.439182 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:14.439157    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-03d892-openshift-default-6d4cdfbfd9-qs6xc" event={"ID":"7d24afeb-7f16-4e65-b159-9e29b3f32c24","Type":"ContainerStarted","Data":"e006575e708dddff37a3ccb7520b472cfcc8c3987a1427e71d62d7bf24b4f5bb"}
Oct 30 01:01:15.352407 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:15.349376    5002 patch_prober.go:29] interesting pod/e2e-inf-a-8dbe73-openshift-default-6c44d95ddf-5hz6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body=
Oct 30 01:01:15.352407 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:15.349462    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8dbe73-openshift-default-6c44d95ddf-5hz6q" podUID="1383e696-ad5a-4b3f-900c-a987399f6bec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused"
Oct 30 01:01:15.497404 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:15.496568    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-03d892-openshift-default-6d4cdfbfd9-qs6xc" event={"ID":"7d24afeb-7f16-4e65-b159-9e29b3f32c24","Type":"ContainerStarted","Data":"a493c7d467e528da883b3ba7767ad707f50543074882fee550e4567563a8bda6"}
Oct 30 01:01:15.624530 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:15.624403    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-03d892-openshift-default-6d4cdfbfd9-qs6xc" podStartSLOduration=2.624365682 podStartE2EDuration="2.624365682s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:01:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:01:15.574344541 +0000 UTC m=+1771.190777444" watchObservedRunningTime="2026-10-30 01:01:15.624365682 +0000 UTC m=+1771.240798562"
Oct 30 01:01:16.017963 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:16.017914    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-03d892-openshift-default-6d4cdfbfd9-qs6xc"
Oct 30 01:01:16.018186 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:16.018055    5002 patch_prober.go:29] interesting pod/e2e-inf-b-03d892-openshift-default-6d4cdfbfd9-qs6xc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body=
Oct 30 01:01:16.018186 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:16.018093    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-03d892-openshift-default-6d4cdfbfd9-qs6xc" podUID="7d24afeb-7f16-4e65-b159-9e29b3f32c24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused"
Oct 30 01:01:16.196615 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:16.196577    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-jwx9q"]
Oct 30 01:01:16.197188 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:16.197131    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-jwx9q" podUID="a4f0d6bc-42a1-4f97-8713-daad565c8d7f" containerName="istio-proxy" containerID="cri-o://603aaf5acff7be23073e0383dd80cc2cf4e78f778ca57de14f8771ba8b16790f" gracePeriod=30
Oct 30 01:01:16.202892 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:16.202835    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-fwrpg"]
Oct 30 01:01:16.203581 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:16.203550    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-fwrpg" podUID="a498076a-ee51-4409-b7d4-a0fabd0f385e" containerName="istio-proxy" containerID="cri-o://6a4bc3dbb2ca630ef750fb9f407640b45b1313836cc71723ba752f8e79dbb471" gracePeriod=30
Oct 30 01:01:16.373610 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:16.373483    5002 patch_prober.go:29] interesting pod/e2e-inf-a-8dbe73-openshift-default-6c44d95ddf-5hz6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body=
Oct 30 01:01:16.373610 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:16.373554    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8dbe73-openshift-default-6c44d95ddf-5hz6q" podUID="1383e696-ad5a-4b3f-900c-a987399f6bec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused"
Oct 30 01:01:16.596007 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:16.595961    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-8s7j6"]
Oct 30 01:01:16.601805 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:16.601775    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-8s7j6"
Oct 30 01:01:16.606453 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:16.606415    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-8s7j6"]
Oct 30 01:01:16.622213 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:16.621521    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-48df96-openshift-default-dockercfg-h6qzs\""
Oct 30 01:01:16.694411 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:16.690008    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70b897af-fd89-4e91-b0f4-e6adff33cfdd-istio-podinfo\") pod \"e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-8s7j6\" (UID: \"70b897af-fd89-4e91-b0f4-e6adff33cfdd\") " pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-8s7j6"
Oct 30 01:01:16.694411 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:16.690087    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70b897af-fd89-4e91-b0f4-e6adff33cfdd-workload-certs\") pod \"e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-8s7j6\" (UID: \"70b897af-fd89-4e91-b0f4-e6adff33cfdd\") " pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-8s7j6"
Oct 30 01:01:16.694411 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:16.690133    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkb6b\" (UniqueName: \"kubernetes.io/projected/70b897af-fd89-4e91-b0f4-e6adff33cfdd-kube-api-access-dkb6b\") pod \"e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-8s7j6\" (UID: \"70b897af-fd89-4e91-b0f4-e6adff33cfdd\") " pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-8s7j6"
Oct 30 01:01:16.706443 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:16.690245    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70b897af-fd89-4e91-b0f4-e6adff33cfdd-workload-socket\") pod \"e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-8s7j6\" (UID: \"70b897af-fd89-4e91-b0f4-e6adff33cfdd\") " pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-8s7j6"
Oct 30 01:01:16.706443 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:16.705694    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70b897af-fd89-4e91-b0f4-e6adff33cfdd-istio-token\") pod \"e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-8s7j6\" (UID: \"70b897af-fd89-4e91-b0f4-e6adff33cfdd\") " pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-8s7j6"
Oct 30 01:01:16.706443 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:16.705878    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70b897af-fd89-4e91-b0f4-e6adff33cfdd-istiod-ca-cert\") pod \"e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-8s7j6\" (UID: \"70b897af-fd89-4e91-b0f4-e6adff33cfdd\") " pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-8s7j6"
Oct 30 01:01:16.706443 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:16.706006    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70b897af-fd89-4e91-b0f4-e6adff33cfdd-istio-data\") pod \"e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-8s7j6\" (UID: \"70b897af-fd89-4e91-b0f4-e6adff33cfdd\") " pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-8s7j6"
Oct 30 01:01:16.706443 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:16.706062    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70b897af-fd89-4e91-b0f4-e6adff33cfdd-credential-socket\") pod \"e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-8s7j6\" (UID: \"70b897af-fd89-4e91-b0f4-e6adff33cfdd\") " pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-8s7j6"
Oct 30 01:01:16.706443 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:16.706124    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70b897af-fd89-4e91-b0f4-e6adff33cfdd-istio-envoy\") pod \"e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-8s7j6\" (UID: \"70b897af-fd89-4e91-b0f4-e6adff33cfdd\") " pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-8s7j6"
Oct 30 01:01:16.807545 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:16.807498    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70b897af-fd89-4e91-b0f4-e6adff33cfdd-workload-certs\") pod \"e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-8s7j6\" (UID: \"70b897af-fd89-4e91-b0f4-e6adff33cfdd\") " pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-8s7j6"
Oct 30 01:01:16.807859 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:16.807568    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkb6b\" (UniqueName: \"kubernetes.io/projected/70b897af-fd89-4e91-b0f4-e6adff33cfdd-kube-api-access-dkb6b\") pod \"e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-8s7j6\" (UID: \"70b897af-fd89-4e91-b0f4-e6adff33cfdd\") " pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-8s7j6"
Oct 30 01:01:16.807859 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:16.807680    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70b897af-fd89-4e91-b0f4-e6adff33cfdd-workload-socket\") pod \"e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-8s7j6\" (UID: \"70b897af-fd89-4e91-b0f4-e6adff33cfdd\") " pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-8s7j6"
Oct 30 01:01:16.807859 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:16.807714    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70b897af-fd89-4e91-b0f4-e6adff33cfdd-istio-token\") pod \"e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-8s7j6\" (UID: \"70b897af-fd89-4e91-b0f4-e6adff33cfdd\") " pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-8s7j6"
Oct 30 01:01:16.807859 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:16.807797    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70b897af-fd89-4e91-b0f4-e6adff33cfdd-istiod-ca-cert\") pod \"e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-8s7j6\" (UID: \"70b897af-fd89-4e91-b0f4-e6adff33cfdd\") " pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-8s7j6"
Oct 30 01:01:16.807859 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:16.807847    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70b897af-fd89-4e91-b0f4-e6adff33cfdd-istio-data\") pod \"e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-8s7j6\" (UID: \"70b897af-fd89-4e91-b0f4-e6adff33cfdd\") " pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-8s7j6"
Oct 30 01:01:16.808120 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:16.807877    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70b897af-fd89-4e91-b0f4-e6adff33cfdd-credential-socket\") pod \"e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-8s7j6\" (UID: \"70b897af-fd89-4e91-b0f4-e6adff33cfdd\") " pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-8s7j6"
Oct 30 01:01:16.808120 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:16.807927    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70b897af-fd89-4e91-b0f4-e6adff33cfdd-istio-envoy\") pod \"e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-8s7j6\" (UID: \"70b897af-fd89-4e91-b0f4-e6adff33cfdd\") " pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-8s7j6"
Oct 30 01:01:16.808774 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:16.808742    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70b897af-fd89-4e91-b0f4-e6adff33cfdd-istio-podinfo\") pod \"e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-8s7j6\" (UID: \"70b897af-fd89-4e91-b0f4-e6adff33cfdd\") " pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-8s7j6"
Oct 30 01:01:16.809053 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:16.809025    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70b897af-fd89-4e91-b0f4-e6adff33cfdd-workload-socket\") pod \"e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-8s7j6\" (UID: \"70b897af-fd89-4e91-b0f4-e6adff33cfdd\") " pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-8s7j6"
Oct 30 01:01:16.809424 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:16.809402    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70b897af-fd89-4e91-b0f4-e6adff33cfdd-credential-socket\") pod \"e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-8s7j6\" (UID: \"70b897af-fd89-4e91-b0f4-e6adff33cfdd\") " pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-8s7j6"
Oct 30 01:01:16.809663 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:16.809632    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70b897af-fd89-4e91-b0f4-e6adff33cfdd-workload-certs\") pod \"e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-8s7j6\" (UID: \"70b897af-fd89-4e91-b0f4-e6adff33cfdd\") " pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-8s7j6"
Oct 30 01:01:16.810044 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:16.810018    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70b897af-fd89-4e91-b0f4-e6adff33cfdd-istio-data\") pod \"e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-8s7j6\" (UID: \"70b897af-fd89-4e91-b0f4-e6adff33cfdd\") " pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-8s7j6"
Oct 30 01:01:16.810471 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:16.810449    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70b897af-fd89-4e91-b0f4-e6adff33cfdd-istiod-ca-cert\") pod \"e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-8s7j6\" (UID: \"70b897af-fd89-4e91-b0f4-e6adff33cfdd\") " pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-8s7j6"
Oct 30 01:01:16.812696 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:16.812671    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70b897af-fd89-4e91-b0f4-e6adff33cfdd-istio-podinfo\") pod \"e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-8s7j6\" (UID: \"70b897af-fd89-4e91-b0f4-e6adff33cfdd\") " pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-8s7j6"
Oct 30 01:01:16.814411 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:16.814261    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70b897af-fd89-4e91-b0f4-e6adff33cfdd-istio-envoy\") pod \"e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-8s7j6\" (UID: \"70b897af-fd89-4e91-b0f4-e6adff33cfdd\") " pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-8s7j6"
Oct 30 01:01:16.834655 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:16.834523    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70b897af-fd89-4e91-b0f4-e6adff33cfdd-istio-token\") pod \"e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-8s7j6\" (UID: \"70b897af-fd89-4e91-b0f4-e6adff33cfdd\") " pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-8s7j6"
Oct 30 01:01:16.835130 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:16.835107    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkb6b\" (UniqueName: \"kubernetes.io/projected/70b897af-fd89-4e91-b0f4-e6adff33cfdd-kube-api-access-dkb6b\") pod \"e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-8s7j6\" (UID: \"70b897af-fd89-4e91-b0f4-e6adff33cfdd\") " pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-8s7j6"
Oct 30 01:01:16.938278 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:16.938237    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-8s7j6"
Oct 30 01:01:17.021423 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:17.017732    5002 patch_prober.go:29] interesting pod/e2e-inf-b-03d892-openshift-default-6d4cdfbfd9-qs6xc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body=
Oct 30 01:01:17.021423 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:17.017807    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-03d892-openshift-default-6d4cdfbfd9-qs6xc" podUID="7d24afeb-7f16-4e65-b159-9e29b3f32c24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused"
Oct 30 01:01:17.480888 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:17.479966    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-8dbe73-openshift-default-6c44d95ddf-5hz6q"
Oct 30 01:01:17.483203 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:17.482904    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-8dbe73-openshift-default-6c44d95ddf-5hz6q"
Oct 30 01:01:17.484323 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:17.484274    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-8dbe73-openshift-default-6c44d95ddf-5hz6q"
Oct 30 01:01:17.612760 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:17.612721    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6444d7-hxdhq"]
Oct 30 01:01:17.623295 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:17.623263    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6444d7-hxdhq"]
Oct 30 01:01:17.623622 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:17.623602    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6444d7-hxdhq"
Oct 30 01:01:17.637677 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:17.637516    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02a17ef7-d5cb-496d-80a0-bfb1f6a0ac87-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-6444d7-hxdhq\" (UID: \"02a17ef7-d5cb-496d-80a0-bfb1f6a0ac87\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6444d7-hxdhq"
Oct 30 01:01:17.637862 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:17.637786    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7bx8\" (UniqueName: \"kubernetes.io/projected/02a17ef7-d5cb-496d-80a0-bfb1f6a0ac87-kube-api-access-p7bx8\") pod \"maas-api-revoke-keys-e2e-inf-a-6444d7-hxdhq\" (UID: \"02a17ef7-d5cb-496d-80a0-bfb1f6a0ac87\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6444d7-hxdhq"
Oct 30 01:01:17.638116 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:17.637970    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02a17ef7-d5cb-496d-80a0-bfb1f6a0ac87-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-6444d7-hxdhq\" (UID: \"02a17ef7-d5cb-496d-80a0-bfb1f6a0ac87\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6444d7-hxdhq"
Oct 30 01:01:17.640776 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:17.640756    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:01:17.641308 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:17.641293    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5phsh\""
Oct 30 01:01:17.678412 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:17.674655    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:01:17.678412 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:17.674768    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:01:17.722813 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:17.717354    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-8s7j6"]
Oct 30 01:01:17.738365 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:17.738305    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:01:17.738625 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:17.738601    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:01:17.738806 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:17.738787    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:01:17.740176 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:17.740150    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02a17ef7-d5cb-496d-80a0-bfb1f6a0ac87-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-6444d7-hxdhq\" (UID: \"02a17ef7-d5cb-496d-80a0-bfb1f6a0ac87\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6444d7-hxdhq"
Oct 30 01:01:17.740487 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:17.740462    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02a17ef7-d5cb-496d-80a0-bfb1f6a0ac87-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-6444d7-hxdhq\" (UID: \"02a17ef7-d5cb-496d-80a0-bfb1f6a0ac87\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6444d7-hxdhq"
Oct 30 01:01:17.740929 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:17.740905    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7bx8\" (UniqueName: \"kubernetes.io/projected/02a17ef7-d5cb-496d-80a0-bfb1f6a0ac87-kube-api-access-p7bx8\") pod \"maas-api-revoke-keys-e2e-inf-a-6444d7-hxdhq\" (UID: \"02a17ef7-d5cb-496d-80a0-bfb1f6a0ac87\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6444d7-hxdhq"
Oct 30 01:01:17.765442 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:17.758673    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02a17ef7-d5cb-496d-80a0-bfb1f6a0ac87-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-6444d7-hxdhq\" (UID: \"02a17ef7-d5cb-496d-80a0-bfb1f6a0ac87\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6444d7-hxdhq"
Oct 30 01:01:17.804400 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:17.800568    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02a17ef7-d5cb-496d-80a0-bfb1f6a0ac87-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-6444d7-hxdhq\" (UID: \"02a17ef7-d5cb-496d-80a0-bfb1f6a0ac87\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6444d7-hxdhq"
Oct 30 01:01:17.808397 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:17.806336    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7bx8\" (UniqueName: \"kubernetes.io/projected/02a17ef7-d5cb-496d-80a0-bfb1f6a0ac87-kube-api-access-p7bx8\") pod \"maas-api-revoke-keys-e2e-inf-a-6444d7-hxdhq\" (UID: \"02a17ef7-d5cb-496d-80a0-bfb1f6a0ac87\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6444d7-hxdhq"
Oct 30 01:01:17.954466 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:17.947884    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6444d7-hxdhq"
Oct 30 01:01:18.063455 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:18.061296    5002 patch_prober.go:29] interesting pod/e2e-inf-b-03d892-openshift-default-6d4cdfbfd9-qs6xc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body=
Oct 30 01:01:18.063455 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:18.061350    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-03d892-openshift-default-6d4cdfbfd9-qs6xc" podUID="7d24afeb-7f16-4e65-b159-9e29b3f32c24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused"
Oct 30 01:01:19.039411 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:19.039277    5002 patch_prober.go:29] interesting pod/e2e-inf-b-03d892-openshift-default-6d4cdfbfd9-qs6xc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body=
Oct 30 01:01:19.039411 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:19.039349    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-03d892-openshift-default-6d4cdfbfd9-qs6xc" podUID="7d24afeb-7f16-4e65-b159-9e29b3f32c24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused"
Oct 30 01:01:19.309726 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:19.302452    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1a11cf/distinct-w2-1a11cf-kserve-854cf4cdf8-cddw6" podUID="32a03478-46eb-4363-97b6-51a32c4099c7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:01:19.309726 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:19.303570    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-8s7j6" event={"ID":"70b897af-fd89-4e91-b0f4-e6adff33cfdd","Type":"ContainerStarted","Data":"a60c000933a68b8e5f964cea4522e277d96b81275d5d99ed30aa5453c39c37b0"}
Oct 30 01:01:19.309726 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:19.303623    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-8s7j6" event={"ID":"70b897af-fd89-4e91-b0f4-e6adff33cfdd","Type":"ContainerStarted","Data":"e1563650df956a017d0166dafa3833fc87751dc6a52719e44956d050e9380571"}
Oct 30 01:01:19.381407 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:19.377338    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-8s7j6" podStartSLOduration=3.377316501 podStartE2EDuration="3.377316501s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:01:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:01:19.375935068 +0000 UTC m=+1774.992367973" watchObservedRunningTime="2026-10-30 01:01:19.377316501 +0000 UTC m=+1774.993749381"
Oct 30 01:01:19.437415 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:19.430458    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-29be73/premium-w1-29be73-kserve-7b777cfc8c-7kgp5" podUID="e6fabbdd-86ca-4cbf-85be-ef250575a355" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:01:19.531405 ip-10-0-128-7 kubenswrapper[5002]: W1030 01:01:19.521440    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02a17ef7_d5cb_496d_80a0_bfb1f6a0ac87.slice/crio-323389a6060a7ee10b3700779814ed0b0e4575333dcf4afc2e9cf5974b83770a WatchSource:0}: Error finding container 323389a6060a7ee10b3700779814ed0b0e4575333dcf4afc2e9cf5974b83770a: Status 404 returned error can't find the container with id 323389a6060a7ee10b3700779814ed0b0e4575333dcf4afc2e9cf5974b83770a
Oct 30 01:01:19.585485 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:19.564184    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6444d7-hxdhq"]
Oct 30 01:01:19.664490 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:19.643496    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-03d892/test-model-03d892-kserve-7f8c7cb699-kp2mb" podUID="b3e31ee3-d2eb-42b1-84dc-173ed4434f84" containerName="main" probeResult="failure" output="Get \"https://10.132.0.136:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:01:19.672707 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:19.672589    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-52650f/premium-w0-52650f-kserve-c5bf66f9d-qq5dm" podUID="3455d983-b11b-4cee-bfce-d1f399cf9679" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:01:19.757409 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:19.751527    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca5363/distinct-2-w4-ca5363-kserve-576576f9b5-xkkqc" podUID="5de8bebf-cfad-49da-9a98-08aedde4442c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Oct 30 01:01:19.757409 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:19.752527    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w1-48df96/auth-test-model-48df96-kserve-9bb4b5585-cvvln" podUID="22351ddc-9884-4058-b71e-7c320c2da701" containerName="main" probeResult="failure" output="Get \"https://10.132.0.139:8000/ready\": context deadline exceeded"
Oct 30 01:01:19.810112 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:19.809500    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cefb27/simulator-w3-cefb27-kserve-6b669c9d9f-zx2vm" podUID="7121e81e-f903-42c0-a7a4-bc18d49194e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:01:19.956406 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:19.956324    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-8s7j6"
Oct 30 01:01:19.969928 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:19.968497    5002 patch_prober.go:29] interesting pod/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-8s7j6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body=
Oct 30 01:01:19.969928 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:19.968565    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-8s7j6" podUID="70b897af-fd89-4e91-b0f4-e6adff33cfdd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused"
Oct 30 01:01:20.022405 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:20.018825    5002 patch_prober.go:29] interesting pod/e2e-inf-b-03d892-openshift-default-6d4cdfbfd9-qs6xc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body=
Oct 30 01:01:20.022405 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:20.018895    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-03d892-openshift-default-6d4cdfbfd9-qs6xc" podUID="7d24afeb-7f16-4e65-b159-9e29b3f32c24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused"
Oct 30 01:01:20.079399 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:20.070169    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-52650f/distinct-w0-52650f-kserve-d85899b5-vlhps" podUID="ea1850d9-abba-4977-a540-f6e0eaefb17d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:01:20.221602 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:20.219984    5002 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-1a11cf/distinct-2-w2-1a11cf-kserve-758bb6dd57-j5f2k" podUID="f2d66dea-9f0b-43f3-a774-bb065e88f378" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:01:20.221602 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:20.220431    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-481ab7/test-model-481ab7-kserve-59c6875d8f-wz4bj" podUID="75c0d4c3-f369-4930-839f-25cfad1f3f5e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.120:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:01:20.243687 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:20.242525    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-29be73/simulator-w1-29be73-kserve-9958d76db-fks2t" podUID="fbb3c806-8967-4d39-84c6-9dc022fc5cae" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:01:20.260426 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:20.244802    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-51b95e/test-model-51b95e-kserve-5c4d8899f7-hc4c2" podUID="87ebc816-fbe5-4a3e-b650-39f2a72e9037" containerName="main" probeResult="failure" output="Get \"https://10.132.0.142:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:01:20.315296 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:20.304871    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1a11cf/distinct-w2-1a11cf-kserve-854cf4cdf8-cddw6" podUID="32a03478-46eb-4363-97b6-51a32c4099c7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:01:20.315296 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:20.311694    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-1a11cf/distinct-w2-1a11cf-kserve-854cf4cdf8-cddw6"
Oct 30 01:01:20.316626 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:20.316577    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1a11cf/distinct-2-w2-1a11cf-kserve-758bb6dd57-j5f2k" podUID="f2d66dea-9f0b-43f3-a774-bb065e88f378" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:01:20.373535 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:20.373497    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6444d7-hxdhq" event={"ID":"02a17ef7-d5cb-496d-80a0-bfb1f6a0ac87","Type":"ContainerStarted","Data":"217a2ee7d3bfe7dc6022dbe22bcd61bd9a8e48fa2caeb8e534a2513534442545"}
Oct 30 01:01:20.373829 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:20.373809    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6444d7-hxdhq" event={"ID":"02a17ef7-d5cb-496d-80a0-bfb1f6a0ac87","Type":"ContainerStarted","Data":"323389a6060a7ee10b3700779814ed0b0e4575333dcf4afc2e9cf5974b83770a"}
Oct 30 01:01:20.413414 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:20.413348    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-8dbe73/test-model-8dbe73-kserve-78dbd9d5cc-5zdxp" podUID="f3cd6ac5-b9a0-4096-94ee-851e2042e9fa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.137:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:01:20.414021 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:20.413988    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6f4fba/simulator-w5-6f4fba-kserve-5bf8dfddd5-wcwzh" podUID="3a2c2f8f-649c-4ab6-af98-5d87df0e511f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/ready\": context deadline exceeded"
Oct 30 01:01:20.418807 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:20.418771    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-021d6d/simulator-w6-021d6d-kserve-6cc97fdf86-dqtpw" podUID="50586f76-4f3b-4e7a-83ac-6c573faee415" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": context deadline exceeded"
Oct 30 01:01:20.431544 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:20.431493    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-29be73/premium-w1-29be73-kserve-7b777cfc8c-7kgp5" podUID="e6fabbdd-86ca-4cbf-85be-ef250575a355" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:01:20.435401 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:20.432101    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-29be73/premium-w1-29be73-kserve-7b777cfc8c-7kgp5"
Oct 30 01:01:20.441582 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:20.441539    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-84b89f/test-model-84b89f-kserve-d8c475485-5prlt" podUID="00a3e39a-01f0-4d98-9350-aa0b17a6c434" containerName="main" probeResult="failure" output="Get \"https://10.132.0.138:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:01:20.728426 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:20.724730    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-03d892/test-model-03d892-kserve-7f8c7cb699-kp2mb" podUID="b3e31ee3-d2eb-42b1-84dc-173ed4434f84" containerName="main" probeResult="failure" output="Get \"https://10.132.0.136:8000/ready\": context deadline exceeded"
Oct 30 01:01:20.730978 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:20.730843    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-52650f/premium-w0-52650f-kserve-c5bf66f9d-qq5dm" podUID="3455d983-b11b-4cee-bfce-d1f399cf9679" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": context deadline exceeded"
Oct 30 01:01:20.819432 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:20.784619    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w1-48df96/auth-test-model-48df96-kserve-9bb4b5585-cvvln" podUID="22351ddc-9884-4058-b71e-7c320c2da701" containerName="main" probeResult="failure" output="Get \"https://10.132.0.139:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:01:20.819432 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:20.815196    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6444d7-hxdhq" podStartSLOduration=3.815172741 podStartE2EDuration="3.815172741s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:01:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:01:20.515820567 +0000 UTC m=+1776.132253477" watchObservedRunningTime="2026-10-30 01:01:20.815172741 +0000 UTC m=+1776.431605623"
Oct 30 01:01:20.871428 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:20.841012    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cefb27/simulator-w3-cefb27-kserve-6b669c9d9f-zx2vm" podUID="7121e81e-f903-42c0-a7a4-bc18d49194e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:01:20.871428 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:20.841949    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-03d892/test-model-03d892-kserve-7f8c7cb699-kp2mb"
Oct 30 01:01:20.871428 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:20.842006    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-52650f/premium-w0-52650f-kserve-c5bf66f9d-qq5dm"
Oct 30 01:01:20.871428 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:20.842040    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w1-48df96/auth-test-model-48df96-kserve-9bb4b5585-cvvln"
Oct 30 01:01:20.871428 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:20.842073    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-cefb27/simulator-w3-cefb27-kserve-6b669c9d9f-zx2vm"
Oct 30 01:01:20.871428 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:20.845345    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-29be73/premium-w1-29be73-kserve-7b777cfc8c-7kgp5"
Oct 30 01:01:20.871428 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:20.855461    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w1-48df96/auth-test-model-48df96-kserve-9bb4b5585-cvvln"
Oct 30 01:01:20.941887 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:20.941856    5002 patch_prober.go:29] interesting pod/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-8s7j6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body=
Oct 30 01:01:20.945672 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:20.942155    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-8s7j6" podUID="70b897af-fd89-4e91-b0f4-e6adff33cfdd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused"
Oct 30 01:01:20.945672 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:20.943551    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-52650f/premium-w0-52650f-kserve-c5bf66f9d-qq5dm"
Oct 30 01:01:20.968406 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:20.966605    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-cefb27/simulator-w3-cefb27-kserve-6b669c9d9f-zx2vm"
Oct 30 01:01:21.027460 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:21.024471    5002 patch_prober.go:29] interesting pod/e2e-inf-b-03d892-openshift-default-6d4cdfbfd9-qs6xc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body=
Oct 30 01:01:21.027460 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:21.024532    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-03d892-openshift-default-6d4cdfbfd9-qs6xc" podUID="7d24afeb-7f16-4e65-b159-9e29b3f32c24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused"
Oct 30 01:01:21.035403 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:21.034034    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-03d892/test-model-03d892-kserve-7f8c7cb699-kp2mb"
Oct 30 01:01:21.113780 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:21.101510    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-52650f/distinct-w0-52650f-kserve-d85899b5-vlhps" podUID="ea1850d9-abba-4977-a540-f6e0eaefb17d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:01:21.113780 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:21.101725    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-52650f/distinct-w0-52650f-kserve-d85899b5-vlhps"
Oct 30 01:01:21.130003 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:21.129973    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-1a11cf/distinct-w2-1a11cf-kserve-854cf4cdf8-cddw6"
Oct 30 01:01:21.137426 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:21.137358    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-52650f/distinct-w0-52650f-kserve-d85899b5-vlhps"
Oct 30 01:01:21.659867 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:21.657641    5002 generic.go:356] "Generic (PLEG): container finished" podID="02a17ef7-d5cb-496d-80a0-bfb1f6a0ac87" containerID="217a2ee7d3bfe7dc6022dbe22bcd61bd9a8e48fa2caeb8e534a2513534442545" exitCode=0
Oct 30 01:01:21.659867 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:21.657777    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6444d7-hxdhq" event={"ID":"02a17ef7-d5cb-496d-80a0-bfb1f6a0ac87","Type":"ContainerDied","Data":"217a2ee7d3bfe7dc6022dbe22bcd61bd9a8e48fa2caeb8e534a2513534442545"}
Oct 30 01:01:21.719204 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:21.719076    5002 generic.go:356] "Generic (PLEG): container finished" podID="a498076a-ee51-4409-b7d4-a0fabd0f385e" containerID="6a4bc3dbb2ca630ef750fb9f407640b45b1313836cc71723ba752f8e79dbb471" exitCode=0
Oct 30 01:01:21.792463 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:21.792425    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-fwrpg" event={"ID":"a498076a-ee51-4409-b7d4-a0fabd0f385e","Type":"ContainerDied","Data":"6a4bc3dbb2ca630ef750fb9f407640b45b1313836cc71723ba752f8e79dbb471"}
Oct 30 01:01:21.814185 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:21.814150    5002 generic.go:356] "Generic (PLEG): container finished" podID="a4f0d6bc-42a1-4f97-8713-daad565c8d7f" containerID="603aaf5acff7be23073e0383dd80cc2cf4e78f778ca57de14f8771ba8b16790f" exitCode=0
Oct 30 01:01:21.816638 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:21.816606    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-jwx9q" event={"ID":"a4f0d6bc-42a1-4f97-8713-daad565c8d7f","Type":"ContainerDied","Data":"603aaf5acff7be23073e0383dd80cc2cf4e78f778ca57de14f8771ba8b16790f"}
Oct 30 01:01:21.979483 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:21.959021    5002 patch_prober.go:29] interesting pod/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-8s7j6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body=
Oct 30 01:01:21.979483 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:21.959072    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-8s7j6" podUID="70b897af-fd89-4e91-b0f4-e6adff33cfdd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused"
Oct 30 01:01:22.030460 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:22.030423    5002 patch_prober.go:29] interesting pod/e2e-inf-b-03d892-openshift-default-6d4cdfbfd9-qs6xc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body=
Oct 30 01:01:22.030742 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:22.030710    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-03d892-openshift-default-6d4cdfbfd9-qs6xc" podUID="7d24afeb-7f16-4e65-b159-9e29b3f32c24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused"
Oct 30 01:01:22.320726 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:22.320654    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-jwx9q"
Oct 30 01:01:22.445288 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:22.445257    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4f0d6bc-42a1-4f97-8713-daad565c8d7f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4f0d6bc-42a1-4f97-8713-daad565c8d7f-istio-envoy\") pod \"a4f0d6bc-42a1-4f97-8713-daad565c8d7f\" (UID: \"a4f0d6bc-42a1-4f97-8713-daad565c8d7f\") "
Oct 30 01:01:22.624209 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:22.458415    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4f0d6bc-42a1-4f97-8713-daad565c8d7f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4f0d6bc-42a1-4f97-8713-daad565c8d7f-istiod-ca-cert\") pod \"a4f0d6bc-42a1-4f97-8713-daad565c8d7f\" (UID: \"a4f0d6bc-42a1-4f97-8713-daad565c8d7f\") "
Oct 30 01:01:22.624646 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:22.624607    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4f0d6bc-42a1-4f97-8713-daad565c8d7f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4f0d6bc-42a1-4f97-8713-daad565c8d7f-credential-socket\") pod \"a4f0d6bc-42a1-4f97-8713-daad565c8d7f\" (UID: \"a4f0d6bc-42a1-4f97-8713-daad565c8d7f\") "
Oct 30 01:01:22.624916 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:22.624888    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a4f0d6bc-42a1-4f97-8713-daad565c8d7f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4f0d6bc-42a1-4f97-8713-daad565c8d7f-istio-podinfo\") pod \"a4f0d6bc-42a1-4f97-8713-daad565c8d7f\" (UID: \"a4f0d6bc-42a1-4f97-8713-daad565c8d7f\") "
Oct 30 01:01:22.634948 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:22.634906    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4f0d6bc-42a1-4f97-8713-daad565c8d7f-kube-api-access-f2jnl\" (UniqueName: \"kubernetes.io/projected/a4f0d6bc-42a1-4f97-8713-daad565c8d7f-kube-api-access-f2jnl\") pod \"a4f0d6bc-42a1-4f97-8713-daad565c8d7f\" (UID: \"a4f0d6bc-42a1-4f97-8713-daad565c8d7f\") "
Oct 30 01:01:22.635305 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:22.635284    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4f0d6bc-42a1-4f97-8713-daad565c8d7f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4f0d6bc-42a1-4f97-8713-daad565c8d7f-istio-data\") pod \"a4f0d6bc-42a1-4f97-8713-daad565c8d7f\" (UID: \"a4f0d6bc-42a1-4f97-8713-daad565c8d7f\") "
Oct 30 01:01:22.635623 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:22.635595    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4f0d6bc-42a1-4f97-8713-daad565c8d7f-istio-token\" (UniqueName: \"kubernetes.io/projected/a4f0d6bc-42a1-4f97-8713-daad565c8d7f-istio-token\") pod \"a4f0d6bc-42a1-4f97-8713-daad565c8d7f\" (UID: \"a4f0d6bc-42a1-4f97-8713-daad565c8d7f\") "
Oct 30 01:01:22.635772 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:22.635757    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4f0d6bc-42a1-4f97-8713-daad565c8d7f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4f0d6bc-42a1-4f97-8713-daad565c8d7f-workload-socket\") pod \"a4f0d6bc-42a1-4f97-8713-daad565c8d7f\" (UID: \"a4f0d6bc-42a1-4f97-8713-daad565c8d7f\") "
Oct 30 01:01:22.635959 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:22.635925    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4f0d6bc-42a1-4f97-8713-daad565c8d7f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4f0d6bc-42a1-4f97-8713-daad565c8d7f-workload-certs\") pod \"a4f0d6bc-42a1-4f97-8713-daad565c8d7f\" (UID: \"a4f0d6bc-42a1-4f97-8713-daad565c8d7f\") "
Oct 30 01:01:22.644507 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:22.459136    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4f0d6bc-42a1-4f97-8713-daad565c8d7f-istiod-ca-cert" pod "a4f0d6bc-42a1-4f97-8713-daad565c8d7f" (UID: "a4f0d6bc-42a1-4f97-8713-daad565c8d7f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:01:22.645097 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:22.645063    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4f0d6bc-42a1-4f97-8713-daad565c8d7f-credential-socket" pod "a4f0d6bc-42a1-4f97-8713-daad565c8d7f" (UID: "a4f0d6bc-42a1-4f97-8713-daad565c8d7f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:01:22.648557 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:22.647287    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4f0d6bc-42a1-4f97-8713-daad565c8d7f-istio-data" pod "a4f0d6bc-42a1-4f97-8713-daad565c8d7f" (UID: "a4f0d6bc-42a1-4f97-8713-daad565c8d7f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:01:22.648704 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:22.648600    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-fwrpg"
Oct 30 01:01:22.651431 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:22.649313    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4f0d6bc-42a1-4f97-8713-daad565c8d7f-workload-certs" pod "a4f0d6bc-42a1-4f97-8713-daad565c8d7f" (UID: "a4f0d6bc-42a1-4f97-8713-daad565c8d7f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:01:22.680563 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:22.680512    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4f0d6bc-42a1-4f97-8713-daad565c8d7f-workload-socket" pod "a4f0d6bc-42a1-4f97-8713-daad565c8d7f" (UID: "a4f0d6bc-42a1-4f97-8713-daad565c8d7f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:01:22.716118 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:22.716060    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a4f0d6bc-42a1-4f97-8713-daad565c8d7f-istio-podinfo" pod "a4f0d6bc-42a1-4f97-8713-daad565c8d7f" (UID: "a4f0d6bc-42a1-4f97-8713-daad565c8d7f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:01:22.722283 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:22.722246    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4f0d6bc-42a1-4f97-8713-daad565c8d7f-istio-envoy" pod "a4f0d6bc-42a1-4f97-8713-daad565c8d7f" (UID: "a4f0d6bc-42a1-4f97-8713-daad565c8d7f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:01:22.742766 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:22.742724    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a498076a-ee51-4409-b7d4-a0fabd0f385e-kube-api-access-58b2n\" (UniqueName: \"kubernetes.io/projected/a498076a-ee51-4409-b7d4-a0fabd0f385e-kube-api-access-58b2n\") pod \"a498076a-ee51-4409-b7d4-a0fabd0f385e\" (UID: \"a498076a-ee51-4409-b7d4-a0fabd0f385e\") "
Oct 30 01:01:22.742962 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:22.742815    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a498076a-ee51-4409-b7d4-a0fabd0f385e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a498076a-ee51-4409-b7d4-a0fabd0f385e-istio-data\") pod \"a498076a-ee51-4409-b7d4-a0fabd0f385e\" (UID: \"a498076a-ee51-4409-b7d4-a0fabd0f385e\") "
Oct 30 01:01:22.742962 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:22.742924    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a498076a-ee51-4409-b7d4-a0fabd0f385e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a498076a-ee51-4409-b7d4-a0fabd0f385e-istio-podinfo\") pod \"a498076a-ee51-4409-b7d4-a0fabd0f385e\" (UID: \"a498076a-ee51-4409-b7d4-a0fabd0f385e\") "
Oct 30 01:01:22.743096 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:22.743007    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a498076a-ee51-4409-b7d4-a0fabd0f385e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a498076a-ee51-4409-b7d4-a0fabd0f385e-workload-socket\") pod \"a498076a-ee51-4409-b7d4-a0fabd0f385e\" (UID: \"a498076a-ee51-4409-b7d4-a0fabd0f385e\") "
Oct 30 01:01:22.743096 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:22.743063    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a498076a-ee51-4409-b7d4-a0fabd0f385e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a498076a-ee51-4409-b7d4-a0fabd0f385e-istio-envoy\") pod \"a498076a-ee51-4409-b7d4-a0fabd0f385e\" (UID: \"a498076a-ee51-4409-b7d4-a0fabd0f385e\") "
Oct 30 01:01:22.743207 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:22.743114    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a498076a-ee51-4409-b7d4-a0fabd0f385e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a498076a-ee51-4409-b7d4-a0fabd0f385e-credential-socket\") pod \"a498076a-ee51-4409-b7d4-a0fabd0f385e\" (UID: \"a498076a-ee51-4409-b7d4-a0fabd0f385e\") "
Oct 30 01:01:22.743293 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:22.743274    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a498076a-ee51-4409-b7d4-a0fabd0f385e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a498076a-ee51-4409-b7d4-a0fabd0f385e-istiod-ca-cert\") pod \"a498076a-ee51-4409-b7d4-a0fabd0f385e\" (UID: \"a498076a-ee51-4409-b7d4-a0fabd0f385e\") "
Oct 30 01:01:22.743357 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:22.743315    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a498076a-ee51-4409-b7d4-a0fabd0f385e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a498076a-ee51-4409-b7d4-a0fabd0f385e-workload-certs\") pod \"a498076a-ee51-4409-b7d4-a0fabd0f385e\" (UID: \"a498076a-ee51-4409-b7d4-a0fabd0f385e\") "
Oct 30 01:01:22.743357 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:22.743351    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a498076a-ee51-4409-b7d4-a0fabd0f385e-istio-token\" (UniqueName: \"kubernetes.io/projected/a498076a-ee51-4409-b7d4-a0fabd0f385e-istio-token\") pod \"a498076a-ee51-4409-b7d4-a0fabd0f385e\" (UID: \"a498076a-ee51-4409-b7d4-a0fabd0f385e\") "
Oct 30 01:01:22.744876 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:22.744848    5002 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4f0d6bc-42a1-4f97-8713-daad565c8d7f-workload-socket\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:01:22.745001 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:22.744893    5002 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4f0d6bc-42a1-4f97-8713-daad565c8d7f-workload-certs\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:01:22.745001 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:22.744911    5002 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4f0d6bc-42a1-4f97-8713-daad565c8d7f-istio-envoy\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:01:22.745001 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:22.744925    5002 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4f0d6bc-42a1-4f97-8713-daad565c8d7f-istiod-ca-cert\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:01:22.745001 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:22.744968    5002 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4f0d6bc-42a1-4f97-8713-daad565c8d7f-credential-socket\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:01:22.745001 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:22.744994    5002 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4f0d6bc-42a1-4f97-8713-daad565c8d7f-istio-podinfo\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:01:22.745260 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:22.745009    5002 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4f0d6bc-42a1-4f97-8713-daad565c8d7f-istio-data\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:01:22.746130 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:22.746105    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4f0d6bc-42a1-4f97-8713-daad565c8d7f-istio-token" pod "a4f0d6bc-42a1-4f97-8713-daad565c8d7f" (UID: "a4f0d6bc-42a1-4f97-8713-daad565c8d7f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:01:22.746349 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:22.746326    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4f0d6bc-42a1-4f97-8713-daad565c8d7f-kube-api-access-f2jnl" pod "a4f0d6bc-42a1-4f97-8713-daad565c8d7f" (UID: "a4f0d6bc-42a1-4f97-8713-daad565c8d7f"). InnerVolumeSpecName "kube-api-access-f2jnl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:01:22.751970 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:22.749725    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a498076a-ee51-4409-b7d4-a0fabd0f385e-credential-socket" pod "a498076a-ee51-4409-b7d4-a0fabd0f385e" (UID: "a498076a-ee51-4409-b7d4-a0fabd0f385e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:01:22.751970 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:22.749950    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a498076a-ee51-4409-b7d4-a0fabd0f385e-workload-socket" pod "a498076a-ee51-4409-b7d4-a0fabd0f385e" (UID: "a498076a-ee51-4409-b7d4-a0fabd0f385e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:01:22.773135 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:22.768520    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a498076a-ee51-4409-b7d4-a0fabd0f385e-workload-certs" pod "a498076a-ee51-4409-b7d4-a0fabd0f385e" (UID: "a498076a-ee51-4409-b7d4-a0fabd0f385e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:01:22.773135 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:22.768766    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a498076a-ee51-4409-b7d4-a0fabd0f385e-istio-data" pod "a498076a-ee51-4409-b7d4-a0fabd0f385e" (UID: "a498076a-ee51-4409-b7d4-a0fabd0f385e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:01:22.773135 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:22.769652    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a498076a-ee51-4409-b7d4-a0fabd0f385e-istiod-ca-cert" pod "a498076a-ee51-4409-b7d4-a0fabd0f385e" (UID: "a498076a-ee51-4409-b7d4-a0fabd0f385e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:01:22.775204 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:22.775167    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a498076a-ee51-4409-b7d4-a0fabd0f385e-istio-envoy" pod "a498076a-ee51-4409-b7d4-a0fabd0f385e" (UID: "a498076a-ee51-4409-b7d4-a0fabd0f385e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:01:22.780885 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:22.777552    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a498076a-ee51-4409-b7d4-a0fabd0f385e-kube-api-access-58b2n" pod "a498076a-ee51-4409-b7d4-a0fabd0f385e" (UID: "a498076a-ee51-4409-b7d4-a0fabd0f385e"). InnerVolumeSpecName "kube-api-access-58b2n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:01:22.789403 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:22.788347    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a498076a-ee51-4409-b7d4-a0fabd0f385e-istio-token" pod "a498076a-ee51-4409-b7d4-a0fabd0f385e" (UID: "a498076a-ee51-4409-b7d4-a0fabd0f385e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:01:22.827544 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:22.827481    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a498076a-ee51-4409-b7d4-a0fabd0f385e-istio-podinfo" pod "a498076a-ee51-4409-b7d4-a0fabd0f385e" (UID: "a498076a-ee51-4409-b7d4-a0fabd0f385e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:01:22.848773 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:22.848727    5002 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4f0d6bc-42a1-4f97-8713-daad565c8d7f-istio-token\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:01:22.848773 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:22.848770    5002 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a498076a-ee51-4409-b7d4-a0fabd0f385e-istiod-ca-cert\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:01:22.849045 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:22.848786    5002 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a498076a-ee51-4409-b7d4-a0fabd0f385e-workload-certs\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:01:22.849045 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:22.848798    5002 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a498076a-ee51-4409-b7d4-a0fabd0f385e-istio-token\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:01:22.849045 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:22.848813    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-58b2n\" (UniqueName: \"kubernetes.io/projected/a498076a-ee51-4409-b7d4-a0fabd0f385e-kube-api-access-58b2n\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:01:22.849045 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:22.848828    5002 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a498076a-ee51-4409-b7d4-a0fabd0f385e-istio-data\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:01:22.849045 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:22.848842    5002 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a498076a-ee51-4409-b7d4-a0fabd0f385e-istio-podinfo\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:01:22.849045 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:22.848856    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f2jnl\" (UniqueName: \"kubernetes.io/projected/a4f0d6bc-42a1-4f97-8713-daad565c8d7f-kube-api-access-f2jnl\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:01:22.849045 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:22.848870    5002 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a498076a-ee51-4409-b7d4-a0fabd0f385e-workload-socket\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:01:22.849045 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:22.848883    5002 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a498076a-ee51-4409-b7d4-a0fabd0f385e-istio-envoy\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:01:22.849045 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:22.848895    5002 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a498076a-ee51-4409-b7d4-a0fabd0f385e-credential-socket\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:01:22.961024 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:22.951929    5002 patch_prober.go:29] interesting pod/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-8s7j6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body=
Oct 30 01:01:22.961024 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:22.951989    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-8s7j6" podUID="70b897af-fd89-4e91-b0f4-e6adff33cfdd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused"
Oct 30 01:01:22.961024 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:22.953815    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-fwrpg" event={"ID":"a498076a-ee51-4409-b7d4-a0fabd0f385e","Type":"ContainerDied","Data":"737463eec608712aedd9bff6452b7662e206fc2278be7c75ce297b0ee2f0f9f5"}
Oct 30 01:01:22.961024 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:22.953881    5002 scope.go:122] "RemoveContainer" containerID="6a4bc3dbb2ca630ef750fb9f407640b45b1313836cc71723ba752f8e79dbb471"
Oct 30 01:01:22.961024 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:22.954033    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-fwrpg"
Oct 30 01:01:22.986405 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:22.981990    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-jwx9q"
Oct 30 01:01:23.003180 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:22.986900    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-jwx9q" event={"ID":"a4f0d6bc-42a1-4f97-8713-daad565c8d7f","Type":"ContainerDied","Data":"c6beb31e285dfd4cd6a48b9e82d4a6c997aafdb6cc11333d74fe758bcae99ec2"}
Oct 30 01:01:23.009626 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:23.009601    5002 scope.go:122] "RemoveContainer" containerID="603aaf5acff7be23073e0383dd80cc2cf4e78f778ca57de14f8771ba8b16790f"
Oct 30 01:01:23.021208 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:23.021176    5002 patch_prober.go:29] interesting pod/e2e-inf-b-03d892-openshift-default-6d4cdfbfd9-qs6xc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body=
Oct 30 01:01:23.021330 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:23.021233    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-03d892-openshift-default-6d4cdfbfd9-qs6xc" podUID="7d24afeb-7f16-4e65-b159-9e29b3f32c24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused"
Oct 30 01:01:23.069190 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:23.069155    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-jwx9q"]
Oct 30 01:01:23.126653 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:23.126606    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-jwx9q"]
Oct 30 01:01:23.132447 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:23.132410    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-fwrpg"]
Oct 30 01:01:23.139026 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:23.138985    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-fwrpg"]
Oct 30 01:01:23.418400 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:23.417627    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6444d7-hxdhq"
Oct 30 01:01:23.433259 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:23.433217    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/02a17ef7-d5cb-496d-80a0-bfb1f6a0ac87-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02a17ef7-d5cb-496d-80a0-bfb1f6a0ac87-maas-api-xks-ca\") pod \"02a17ef7-d5cb-496d-80a0-bfb1f6a0ac87\" (UID: \"02a17ef7-d5cb-496d-80a0-bfb1f6a0ac87\") "
Oct 30 01:01:23.433472 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:23.433374    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02a17ef7-d5cb-496d-80a0-bfb1f6a0ac87-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02a17ef7-d5cb-496d-80a0-bfb1f6a0ac87-maas-api-openshift-ca\") pod \"02a17ef7-d5cb-496d-80a0-bfb1f6a0ac87\" (UID: \"02a17ef7-d5cb-496d-80a0-bfb1f6a0ac87\") "
Oct 30 01:01:23.433472 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:23.433467    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02a17ef7-d5cb-496d-80a0-bfb1f6a0ac87-kube-api-access-p7bx8\" (UniqueName: \"kubernetes.io/projected/02a17ef7-d5cb-496d-80a0-bfb1f6a0ac87-kube-api-access-p7bx8\") pod \"02a17ef7-d5cb-496d-80a0-bfb1f6a0ac87\" (UID: \"02a17ef7-d5cb-496d-80a0-bfb1f6a0ac87\") "
Oct 30 01:01:23.435554 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:23.435518    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02a17ef7-d5cb-496d-80a0-bfb1f6a0ac87-maas-api-openshift-ca" pod "02a17ef7-d5cb-496d-80a0-bfb1f6a0ac87" (UID: "02a17ef7-d5cb-496d-80a0-bfb1f6a0ac87"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:01:23.436539 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:23.436511    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02a17ef7-d5cb-496d-80a0-bfb1f6a0ac87-maas-api-xks-ca" pod "02a17ef7-d5cb-496d-80a0-bfb1f6a0ac87" (UID: "02a17ef7-d5cb-496d-80a0-bfb1f6a0ac87"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:01:23.440565 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:23.440536    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02a17ef7-d5cb-496d-80a0-bfb1f6a0ac87-kube-api-access-p7bx8" pod "02a17ef7-d5cb-496d-80a0-bfb1f6a0ac87" (UID: "02a17ef7-d5cb-496d-80a0-bfb1f6a0ac87"). InnerVolumeSpecName "kube-api-access-p7bx8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:01:23.542401 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:23.535939    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02a17ef7-d5cb-496d-80a0-bfb1f6a0ac87-maas-api-xks-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:01:23.542401 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:23.535983    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02a17ef7-d5cb-496d-80a0-bfb1f6a0ac87-maas-api-openshift-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:01:23.542401 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:23.536007    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p7bx8\" (UniqueName: \"kubernetes.io/projected/02a17ef7-d5cb-496d-80a0-bfb1f6a0ac87-kube-api-access-p7bx8\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:01:23.940440 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:23.938607    5002 patch_prober.go:29] interesting pod/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-8s7j6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body=
Oct 30 01:01:23.940440 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:23.938670    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-8s7j6" podUID="70b897af-fd89-4e91-b0f4-e6adff33cfdd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused"
Oct 30 01:01:24.005483 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:24.004981    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6444d7-hxdhq" event={"ID":"02a17ef7-d5cb-496d-80a0-bfb1f6a0ac87","Type":"ContainerDied","Data":"323389a6060a7ee10b3700779814ed0b0e4575333dcf4afc2e9cf5974b83770a"}
Oct 30 01:01:24.005483 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:24.005034    5002 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="323389a6060a7ee10b3700779814ed0b0e4575333dcf4afc2e9cf5974b83770a"
Oct 30 01:01:24.005483 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:24.005085    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6444d7-hxdhq"
Oct 30 01:01:24.126187 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:24.126152    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-03d892-openshift-default-6d4cdfbfd9-qs6xc"
Oct 30 01:01:24.127568 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:24.127541    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-03d892-openshift-default-6d4cdfbfd9-qs6xc"
Oct 30 01:01:24.128428 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:24.128370    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-03d892-openshift-default-6d4cdfbfd9-qs6xc"
Oct 30 01:01:24.696407 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:24.691790    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a498076a-ee51-4409-b7d4-a0fabd0f385e" path="/var/lib/kubelet/pods/a498076a-ee51-4409-b7d4-a0fabd0f385e/volumes"
Oct 30 01:01:24.703416 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:24.701714    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4f0d6bc-42a1-4f97-8713-daad565c8d7f" path="/var/lib/kubelet/pods/a4f0d6bc-42a1-4f97-8713-daad565c8d7f/volumes"
Oct 30 01:01:24.939416 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:24.939357    5002 patch_prober.go:29] interesting pod/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-8s7j6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body=
Oct 30 01:01:24.939569 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:24.939459    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-8s7j6" podUID="70b897af-fd89-4e91-b0f4-e6adff33cfdd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused"
Oct 30 01:01:25.939255 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:25.939211    5002 patch_prober.go:29] interesting pod/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-8s7j6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body=
Oct 30 01:01:25.939721 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:25.939277    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-8s7j6" podUID="70b897af-fd89-4e91-b0f4-e6adff33cfdd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused"
Oct 30 01:01:26.974404 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:26.973198    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-8s7j6"
Oct 30 01:01:26.975848 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:26.974878    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-8s7j6"
Oct 30 01:01:26.975988 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:26.975948    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-8s7j6"
Oct 30 01:01:42.865032 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:01:42.864992    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbjdq" containerName="extractor"
Oct 30 01:01:44.850243 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:44.850214    5002 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-g7c2w_5dc56e7e-f3c1-43eb-a21b-8be6bb4706cf/console-operator/1.log"
Oct 30 01:01:44.858865 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:44.858833    5002 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-g7c2w_5dc56e7e-f3c1-43eb-a21b-8be6bb4706cf/console-operator/1.log"
Oct 30 01:01:47.167756 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:47.167719    5002 scope.go:122] "RemoveContainer" containerID="3e61e37ea7be59c1f590eaf6a33ae6e26cc8aef860019cdf5bce700c0cebeb04"
Oct 30 01:01:47.178272 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:47.178244    5002 scope.go:122] "RemoveContainer" containerID="06cd0bbfb970047f19c28fe86cf05a040fd55a7b4c561fcc3e9e6130741aa439"
Oct 30 01:01:49.213116 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:49.213086    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6444d7-hxdhq"]
Oct 30 01:01:49.228360 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:49.228327    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6444d7-hxdhq"]
Oct 30 01:01:50.497877 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:50.497842    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02a17ef7-d5cb-496d-80a0-bfb1f6a0ac87" path="/var/lib/kubelet/pods/02a17ef7-d5cb-496d-80a0-bfb1f6a0ac87/volumes"
Oct 30 01:01:53.030746 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:53.030700    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4d83951-wxdv6"]
Oct 30 01:01:53.044497 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:53.044452    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4d83951-wxdv6"]
Oct 30 01:01:54.500187 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:54.500144    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25e53f18-a85a-4601-874d-e58d0704f6df" path="/var/lib/kubelet/pods/25e53f18-a85a-4601-874d-e58d0704f6df/volumes"
Oct 30 01:01:56.246289 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:56.246255    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-6444d7-openshift-default-7bf76bdcd9-6nnq2"]
Oct 30 01:01:56.247133 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:56.247097    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-6444d7-openshift-default-7bf76bdcd9-6nnq2" podUID="750a98de-3674-4c5b-835c-48b928841cd7" containerName="istio-proxy" containerID="cri-o://128a0167b68df93e5d9adce649b2359b6064d025992b8ab66c9709b7e511ebff" gracePeriod=30
Oct 30 01:01:56.798608 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:56.798563    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b3bdf9-cx7hq"]
Oct 30 01:01:56.799290 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:56.799267    5002 cpu_manager.go:414] "containerMap: removing container" podUID="02a17ef7-d5cb-496d-80a0-bfb1f6a0ac87" containerName="revoke-keys"
Oct 30 01:01:56.799441 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:56.799291    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02a17ef7-d5cb-496d-80a0-bfb1f6a0ac87" containerName="revoke-keys"
Oct 30 01:01:56.799441 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:56.799345    5002 cpu_manager.go:414] "containerMap: removing container" podUID="a4f0d6bc-42a1-4f97-8713-daad565c8d7f" containerName="istio-proxy"
Oct 30 01:01:56.799441 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:56.799354    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4f0d6bc-42a1-4f97-8713-daad565c8d7f" containerName="istio-proxy"
Oct 30 01:01:56.799441 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:56.799370    5002 cpu_manager.go:414] "containerMap: removing container" podUID="a498076a-ee51-4409-b7d4-a0fabd0f385e" containerName="istio-proxy"
Oct 30 01:01:56.799441 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:56.799399    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a498076a-ee51-4409-b7d4-a0fabd0f385e" containerName="istio-proxy"
Oct 30 01:01:56.799697 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:56.799601    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4f0d6bc-42a1-4f97-8713-daad565c8d7f" containerName="istio-proxy"
Oct 30 01:01:56.799697 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:56.799619    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="a498076a-ee51-4409-b7d4-a0fabd0f385e" containerName="istio-proxy"
Oct 30 01:01:56.799697 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:56.799639    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="02a17ef7-d5cb-496d-80a0-bfb1f6a0ac87" containerName="revoke-keys"
Oct 30 01:01:56.805364 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:56.805342    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b3bdf9-cx7hq"
Oct 30 01:01:56.821507 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:56.821479    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b3bdf9-cx7hq"]
Oct 30 01:01:56.831039 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:56.831015    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:01:56.831190 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:56.831052    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:01:56.911365 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:56.911328    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:01:56.911566 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:56.911363    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/256f58c3-48bd-4f56-8500-ea60e5ec8efc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-b3bdf9-cx7hq\" (UID: \"256f58c3-48bd-4f56-8500-ea60e5ec8efc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b3bdf9-cx7hq"
Oct 30 01:01:56.911566 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:56.911495    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5phsh\""
Oct 30 01:01:56.911566 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:56.911536    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fvnw\" (UniqueName: \"kubernetes.io/projected/256f58c3-48bd-4f56-8500-ea60e5ec8efc-kube-api-access-8fvnw\") pod \"maas-api-revoke-keys-e2e-inf-b-b3bdf9-cx7hq\" (UID: \"256f58c3-48bd-4f56-8500-ea60e5ec8efc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b3bdf9-cx7hq"
Oct 30 01:01:56.911674 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:56.911641    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/256f58c3-48bd-4f56-8500-ea60e5ec8efc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-b3bdf9-cx7hq\" (UID: \"256f58c3-48bd-4f56-8500-ea60e5ec8efc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b3bdf9-cx7hq"
Oct 30 01:01:57.013295 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:57.013255    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/256f58c3-48bd-4f56-8500-ea60e5ec8efc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-b3bdf9-cx7hq\" (UID: \"256f58c3-48bd-4f56-8500-ea60e5ec8efc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b3bdf9-cx7hq"
Oct 30 01:01:57.013579 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:57.013557    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/256f58c3-48bd-4f56-8500-ea60e5ec8efc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-b3bdf9-cx7hq\" (UID: \"256f58c3-48bd-4f56-8500-ea60e5ec8efc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b3bdf9-cx7hq"
Oct 30 01:01:57.013676 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:57.013662    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fvnw\" (UniqueName: \"kubernetes.io/projected/256f58c3-48bd-4f56-8500-ea60e5ec8efc-kube-api-access-8fvnw\") pod \"maas-api-revoke-keys-e2e-inf-b-b3bdf9-cx7hq\" (UID: \"256f58c3-48bd-4f56-8500-ea60e5ec8efc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b3bdf9-cx7hq"
Oct 30 01:01:57.014181 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:57.014153    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/256f58c3-48bd-4f56-8500-ea60e5ec8efc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-b3bdf9-cx7hq\" (UID: \"256f58c3-48bd-4f56-8500-ea60e5ec8efc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b3bdf9-cx7hq"
Oct 30 01:01:57.015915 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:57.015890    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/256f58c3-48bd-4f56-8500-ea60e5ec8efc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-b3bdf9-cx7hq\" (UID: \"256f58c3-48bd-4f56-8500-ea60e5ec8efc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b3bdf9-cx7hq"
Oct 30 01:01:57.041336 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:57.041314    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fvnw\" (UniqueName: \"kubernetes.io/projected/256f58c3-48bd-4f56-8500-ea60e5ec8efc-kube-api-access-8fvnw\") pod \"maas-api-revoke-keys-e2e-inf-b-b3bdf9-cx7hq\" (UID: \"256f58c3-48bd-4f56-8500-ea60e5ec8efc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b3bdf9-cx7hq"
Oct 30 01:01:57.120340 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:57.120268    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b3bdf9-cx7hq"
Oct 30 01:01:57.452638 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:57.452588    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b3bdf9-cx7hq"]
Oct 30 01:01:58.375956 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:58.373982    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b3bdf9-cx7hq" event={"ID":"256f58c3-48bd-4f56-8500-ea60e5ec8efc","Type":"ContainerStarted","Data":"62f03545c7803e2527c8fcd846da98daaf12fec8f9baf68f92257693c12cf9c4"}
Oct 30 01:01:58.375956 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:58.374051    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b3bdf9-cx7hq" event={"ID":"256f58c3-48bd-4f56-8500-ea60e5ec8efc","Type":"ContainerStarted","Data":"f266ac137c0eb8adc33e10a4d71948b8f98e68f0eada6276613110444e71109f"}
Oct 30 01:01:58.426629 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:58.426572    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b3bdf9-cx7hq" podStartSLOduration=2.426555073 podStartE2EDuration="2.426555073s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:01:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:01:58.425686872 +0000 UTC m=+1814.042119771" watchObservedRunningTime="2026-10-30 01:01:58.426555073 +0000 UTC m=+1814.042987956"
Oct 30 01:01:59.384419 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:59.384354    5002 generic.go:356] "Generic (PLEG): container finished" podID="256f58c3-48bd-4f56-8500-ea60e5ec8efc" containerID="62f03545c7803e2527c8fcd846da98daaf12fec8f9baf68f92257693c12cf9c4" exitCode=0
Oct 30 01:01:59.384829 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:01:59.384453    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b3bdf9-cx7hq" event={"ID":"256f58c3-48bd-4f56-8500-ea60e5ec8efc","Type":"ContainerDied","Data":"62f03545c7803e2527c8fcd846da98daaf12fec8f9baf68f92257693c12cf9c4"}
Oct 30 01:02:00.612968 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:00.612942    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b3bdf9-cx7hq"
Oct 30 01:02:00.695308 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:00.695269    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/256f58c3-48bd-4f56-8500-ea60e5ec8efc-kube-api-access-8fvnw\" (UniqueName: \"kubernetes.io/projected/256f58c3-48bd-4f56-8500-ea60e5ec8efc-kube-api-access-8fvnw\") pod \"256f58c3-48bd-4f56-8500-ea60e5ec8efc\" (UID: \"256f58c3-48bd-4f56-8500-ea60e5ec8efc\") "
Oct 30 01:02:00.695510 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:00.695449    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/256f58c3-48bd-4f56-8500-ea60e5ec8efc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/256f58c3-48bd-4f56-8500-ea60e5ec8efc-maas-api-openshift-ca\") pod \"256f58c3-48bd-4f56-8500-ea60e5ec8efc\" (UID: \"256f58c3-48bd-4f56-8500-ea60e5ec8efc\") "
Oct 30 01:02:00.695580 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:00.695522    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/256f58c3-48bd-4f56-8500-ea60e5ec8efc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/256f58c3-48bd-4f56-8500-ea60e5ec8efc-maas-api-xks-ca\") pod \"256f58c3-48bd-4f56-8500-ea60e5ec8efc\" (UID: \"256f58c3-48bd-4f56-8500-ea60e5ec8efc\") "
Oct 30 01:02:00.695872 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:00.695835    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/256f58c3-48bd-4f56-8500-ea60e5ec8efc-maas-api-openshift-ca" pod "256f58c3-48bd-4f56-8500-ea60e5ec8efc" (UID: "256f58c3-48bd-4f56-8500-ea60e5ec8efc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:02:00.696363 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:00.696336    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/256f58c3-48bd-4f56-8500-ea60e5ec8efc-maas-api-openshift-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:02:00.698025 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:00.697995    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/256f58c3-48bd-4f56-8500-ea60e5ec8efc-kube-api-access-8fvnw" pod "256f58c3-48bd-4f56-8500-ea60e5ec8efc" (UID: "256f58c3-48bd-4f56-8500-ea60e5ec8efc"). InnerVolumeSpecName "kube-api-access-8fvnw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:02:00.698290 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:00.698264    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/256f58c3-48bd-4f56-8500-ea60e5ec8efc-maas-api-xks-ca" pod "256f58c3-48bd-4f56-8500-ea60e5ec8efc" (UID: "256f58c3-48bd-4f56-8500-ea60e5ec8efc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:02:00.797453 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:00.797413    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/256f58c3-48bd-4f56-8500-ea60e5ec8efc-maas-api-xks-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:02:00.797453 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:00.797452    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8fvnw\" (UniqueName: \"kubernetes.io/projected/256f58c3-48bd-4f56-8500-ea60e5ec8efc-kube-api-access-8fvnw\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:02:01.406214 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:01.406179    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b3bdf9-cx7hq"
Oct 30 01:02:01.406410 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:01.406177    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b3bdf9-cx7hq" event={"ID":"256f58c3-48bd-4f56-8500-ea60e5ec8efc","Type":"ContainerDied","Data":"f266ac137c0eb8adc33e10a4d71948b8f98e68f0eada6276613110444e71109f"}
Oct 30 01:02:01.406410 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:01.406354    5002 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f266ac137c0eb8adc33e10a4d71948b8f98e68f0eada6276613110444e71109f"
Oct 30 01:02:01.408681 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:01.408651    5002 generic.go:356] "Generic (PLEG): container finished" podID="750a98de-3674-4c5b-835c-48b928841cd7" containerID="128a0167b68df93e5d9adce649b2359b6064d025992b8ab66c9709b7e511ebff" exitCode=0
Oct 30 01:02:01.408801 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:01.408728    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6444d7-openshift-default-7bf76bdcd9-6nnq2" event={"ID":"750a98de-3674-4c5b-835c-48b928841cd7","Type":"ContainerDied","Data":"128a0167b68df93e5d9adce649b2359b6064d025992b8ab66c9709b7e511ebff"}
Oct 30 01:02:01.408845 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:01.408798    5002 scope.go:122] "RemoveContainer" containerID="b9052f3a47331b79001912f9b046332d2fc3bf08b67fde883354b6e30c262916"
Oct 30 01:02:01.567442 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:01.567372    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6444d7-openshift-default-7bf76bdcd9-6nnq2"
Oct 30 01:02:01.609036 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:01.609004    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/750a98de-3674-4c5b-835c-48b928841cd7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/750a98de-3674-4c5b-835c-48b928841cd7-istio-data\") pod \"750a98de-3674-4c5b-835c-48b928841cd7\" (UID: \"750a98de-3674-4c5b-835c-48b928841cd7\") "
Oct 30 01:02:01.609202 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:01.609046    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/750a98de-3674-4c5b-835c-48b928841cd7-istio-token\" (UniqueName: \"kubernetes.io/projected/750a98de-3674-4c5b-835c-48b928841cd7-istio-token\") pod \"750a98de-3674-4c5b-835c-48b928841cd7\" (UID: \"750a98de-3674-4c5b-835c-48b928841cd7\") "
Oct 30 01:02:01.609202 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:01.609097    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/750a98de-3674-4c5b-835c-48b928841cd7-kube-api-access-cz6qq\" (UniqueName: \"kubernetes.io/projected/750a98de-3674-4c5b-835c-48b928841cd7-kube-api-access-cz6qq\") pod \"750a98de-3674-4c5b-835c-48b928841cd7\" (UID: \"750a98de-3674-4c5b-835c-48b928841cd7\") "
Oct 30 01:02:01.609202 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:01.609147    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/750a98de-3674-4c5b-835c-48b928841cd7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/750a98de-3674-4c5b-835c-48b928841cd7-istio-envoy\") pod \"750a98de-3674-4c5b-835c-48b928841cd7\" (UID: \"750a98de-3674-4c5b-835c-48b928841cd7\") "
Oct 30 01:02:01.609368 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:01.609309    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/750a98de-3674-4c5b-835c-48b928841cd7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/750a98de-3674-4c5b-835c-48b928841cd7-workload-certs\") pod \"750a98de-3674-4c5b-835c-48b928841cd7\" (UID: \"750a98de-3674-4c5b-835c-48b928841cd7\") "
Oct 30 01:02:01.609368 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:01.609332    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/750a98de-3674-4c5b-835c-48b928841cd7-istio-data" pod "750a98de-3674-4c5b-835c-48b928841cd7" (UID: "750a98de-3674-4c5b-835c-48b928841cd7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:02:01.609521 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:01.609452    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/750a98de-3674-4c5b-835c-48b928841cd7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/750a98de-3674-4c5b-835c-48b928841cd7-istio-podinfo\") pod \"750a98de-3674-4c5b-835c-48b928841cd7\" (UID: \"750a98de-3674-4c5b-835c-48b928841cd7\") "
Oct 30 01:02:01.609615 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:01.609572    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/750a98de-3674-4c5b-835c-48b928841cd7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/750a98de-3674-4c5b-835c-48b928841cd7-workload-socket\") pod \"750a98de-3674-4c5b-835c-48b928841cd7\" (UID: \"750a98de-3674-4c5b-835c-48b928841cd7\") "
Oct 30 01:02:01.609862 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:01.609828    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/750a98de-3674-4c5b-835c-48b928841cd7-workload-socket" pod "750a98de-3674-4c5b-835c-48b928841cd7" (UID: "750a98de-3674-4c5b-835c-48b928841cd7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:02:01.609862 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:01.609845    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/750a98de-3674-4c5b-835c-48b928841cd7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/750a98de-3674-4c5b-835c-48b928841cd7-credential-socket\") pod \"750a98de-3674-4c5b-835c-48b928841cd7\" (UID: \"750a98de-3674-4c5b-835c-48b928841cd7\") "
Oct 30 01:02:01.609862 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:01.609842    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/750a98de-3674-4c5b-835c-48b928841cd7-workload-certs" pod "750a98de-3674-4c5b-835c-48b928841cd7" (UID: "750a98de-3674-4c5b-835c-48b928841cd7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:02:01.610090 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:01.609951    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/750a98de-3674-4c5b-835c-48b928841cd7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/750a98de-3674-4c5b-835c-48b928841cd7-istiod-ca-cert\") pod \"750a98de-3674-4c5b-835c-48b928841cd7\" (UID: \"750a98de-3674-4c5b-835c-48b928841cd7\") "
Oct 30 01:02:01.610358 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:01.610331    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/750a98de-3674-4c5b-835c-48b928841cd7-istiod-ca-cert" pod "750a98de-3674-4c5b-835c-48b928841cd7" (UID: "750a98de-3674-4c5b-835c-48b928841cd7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:02:01.610964 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:01.610939    5002 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/750a98de-3674-4c5b-835c-48b928841cd7-workload-certs\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:02:01.611069 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:01.610970    5002 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/750a98de-3674-4c5b-835c-48b928841cd7-workload-socket\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:02:01.611069 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:01.610983    5002 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/750a98de-3674-4c5b-835c-48b928841cd7-istiod-ca-cert\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:02:01.611069 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:01.610995    5002 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/750a98de-3674-4c5b-835c-48b928841cd7-istio-data\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:02:01.611292 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:01.611185    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/750a98de-3674-4c5b-835c-48b928841cd7-credential-socket" pod "750a98de-3674-4c5b-835c-48b928841cd7" (UID: "750a98de-3674-4c5b-835c-48b928841cd7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:02:01.612053 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:01.612031    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/750a98de-3674-4c5b-835c-48b928841cd7-istio-podinfo" pod "750a98de-3674-4c5b-835c-48b928841cd7" (UID: "750a98de-3674-4c5b-835c-48b928841cd7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:02:01.612777 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:01.612728    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/750a98de-3674-4c5b-835c-48b928841cd7-istio-envoy" pod "750a98de-3674-4c5b-835c-48b928841cd7" (UID: "750a98de-3674-4c5b-835c-48b928841cd7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:02:01.613585 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:01.613512    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/750a98de-3674-4c5b-835c-48b928841cd7-istio-token" pod "750a98de-3674-4c5b-835c-48b928841cd7" (UID: "750a98de-3674-4c5b-835c-48b928841cd7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:02:01.614453 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:01.614415    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/750a98de-3674-4c5b-835c-48b928841cd7-kube-api-access-cz6qq" pod "750a98de-3674-4c5b-835c-48b928841cd7" (UID: "750a98de-3674-4c5b-835c-48b928841cd7"). InnerVolumeSpecName "kube-api-access-cz6qq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:02:01.712837 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:01.712784    5002 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/750a98de-3674-4c5b-835c-48b928841cd7-credential-socket\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:02:01.712837 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:01.712831    5002 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/750a98de-3674-4c5b-835c-48b928841cd7-istio-token\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:02:01.712837 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:01.712842    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cz6qq\" (UniqueName: \"kubernetes.io/projected/750a98de-3674-4c5b-835c-48b928841cd7-kube-api-access-cz6qq\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:02:01.712837 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:01.712853    5002 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/750a98de-3674-4c5b-835c-48b928841cd7-istio-envoy\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:02:01.713116 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:01.712874    5002 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/750a98de-3674-4c5b-835c-48b928841cd7-istio-podinfo\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:02:02.417425 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:02.417393    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6444d7-openshift-default-7bf76bdcd9-6nnq2"
Oct 30 01:02:02.417425 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:02.417409    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6444d7-openshift-default-7bf76bdcd9-6nnq2" event={"ID":"750a98de-3674-4c5b-835c-48b928841cd7","Type":"ContainerDied","Data":"0f9701dab16f9cb7495f70e43cef32036dfabda278ca66981c1040d7db2df347"}
Oct 30 01:02:02.417682 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:02.417469    5002 scope.go:122] "RemoveContainer" containerID="128a0167b68df93e5d9adce649b2359b6064d025992b8ab66c9709b7e511ebff"
Oct 30 01:02:02.500700 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:02.500669    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-6444d7-openshift-default-7bf76bdcd9-6nnq2"]
Oct 30 01:02:02.504346 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:02.504316    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-6444d7-openshift-default-7bf76bdcd9-6nnq2"]
Oct 30 01:02:04.499491 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:04.499454    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="750a98de-3674-4c5b-835c-48b928841cd7" path="/var/lib/kubelet/pods/750a98de-3674-4c5b-835c-48b928841cd7/volumes"
Oct 30 01:02:06.745412 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:06.745326    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7d8z4"]
Oct 30 01:02:10.005895 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:10.005477    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-52650f/premium-w0-52650f-kserve-c5bf66f9d-qq5dm" podUID="3455d983-b11b-4cee-bfce-d1f399cf9679" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:02:10.037507 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:10.037457    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-03d892/test-model-03d892-kserve-7f8c7cb699-kp2mb" podUID="b3e31ee3-d2eb-42b1-84dc-173ed4434f84" containerName="main" probeResult="failure" output="Get \"https://10.132.0.136:8000/ready\": context deadline exceeded"
Oct 30 01:02:10.137404 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:10.121487    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-481ab7/test-model-481ab7-kserve-59c6875d8f-wz4bj" podUID="75c0d4c3-f369-4930-839f-25cfad1f3f5e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.120:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:02:10.206405 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:10.203487    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-29be73/simulator-w1-29be73-kserve-9958d76db-fks2t" podUID="fbb3c806-8967-4d39-84c6-9dc022fc5cae" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:02:10.230406 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:10.221447    5002 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-ca5363/distinct-2-w4-ca5363-kserve-576576f9b5-xkkqc" podUID="5de8bebf-cfad-49da-9a98-08aedde4442c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:02:10.313461 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:10.309517    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1a11cf/distinct-2-w2-1a11cf-kserve-758bb6dd57-j5f2k" podUID="f2d66dea-9f0b-43f3-a774-bb065e88f378" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:02:10.424406 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:10.422486    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-021d6d/simulator-w6-021d6d-kserve-6cc97fdf86-dqtpw" podUID="50586f76-4f3b-4e7a-83ac-6c573faee415" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:02:12.895681 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:02:12.895635    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbjdq" containerName="extractor"
Oct 30 01:02:19.352948 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:19.352576    5002 patch_prober.go:29] interesting pod/e2e-inf-a-ef1084-openshift-default-68989c6b9f-lrmtq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": read tcp 10.132.0.2:51490->10.132.0.141:15021: read: connection reset by peer" start-of-body=
Oct 30 01:02:19.352948 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:19.352653    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-ef1084-openshift-default-68989c6b9f-lrmtq" podUID="1618b1d3-81ce-4def-9498-43b3fdbc46a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": read tcp 10.132.0.2:51490->10.132.0.141:15021: read: connection reset by peer"
Oct 30 01:02:19.716686 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:19.716642    5002 generic.go:356] "Generic (PLEG): container finished" podID="1618b1d3-81ce-4def-9498-43b3fdbc46a4" containerID="b9bb212d7256f99518451eacffb2043a7ef5781b2b01367473460ad8117c5575" exitCode=0
Oct 30 01:02:19.717046 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:19.717015    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ef1084-openshift-default-68989c6b9f-lrmtq" event={"ID":"1618b1d3-81ce-4def-9498-43b3fdbc46a4","Type":"ContainerDied","Data":"b9bb212d7256f99518451eacffb2043a7ef5781b2b01367473460ad8117c5575"}
Oct 30 01:02:19.717728 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:19.717703    5002 scope.go:122] "RemoveContainer" containerID="b9bb212d7256f99518451eacffb2043a7ef5781b2b01367473460ad8117c5575"
Oct 30 01:02:19.722152 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:19.722112    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:02:19.738647 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:19.738600    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:02:19.738992 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:19.738955    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:02:20.727482 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:20.727445    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ef1084-openshift-default-68989c6b9f-lrmtq" event={"ID":"1618b1d3-81ce-4def-9498-43b3fdbc46a4","Type":"ContainerStarted","Data":"9f9d018de891196758eb912c2a779c5b7d19ea6841ae85b657848faa91f75975"}
Oct 30 01:02:20.881853 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:20.881815    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-ef1084-openshift-default-68989c6b9f-lrmtq"
Oct 30 01:02:20.883576 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:20.883541    5002 patch_prober.go:29] interesting pod/e2e-inf-a-ef1084-openshift-default-68989c6b9f-lrmtq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body=
Oct 30 01:02:20.883742 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:20.883611    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ef1084-openshift-default-68989c6b9f-lrmtq" podUID="1618b1d3-81ce-4def-9498-43b3fdbc46a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused"
Oct 30 01:02:21.870803 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:21.870763    5002 patch_prober.go:29] interesting pod/e2e-inf-a-ef1084-openshift-default-68989c6b9f-lrmtq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body=
Oct 30 01:02:21.871442 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:21.871399    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ef1084-openshift-default-68989c6b9f-lrmtq" podUID="1618b1d3-81ce-4def-9498-43b3fdbc46a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused"
Oct 30 01:02:22.880449 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:22.880000    5002 patch_prober.go:29] interesting pod/e2e-inf-a-ef1084-openshift-default-68989c6b9f-lrmtq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body=
Oct 30 01:02:22.880449 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:22.880065    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ef1084-openshift-default-68989c6b9f-lrmtq" podUID="1618b1d3-81ce-4def-9498-43b3fdbc46a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused"
Oct 30 01:02:22.892440 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:22.890327    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-gvb85"]
Oct 30 01:02:22.892440 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:22.890614    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-gvb85" podUID="70f303a4-967c-4a2f-8378-f30e81101a66" containerName="istio-proxy" containerID="cri-o://fb9406b986c59d5abf58064fd61d7063bb19cf99e799e33f6964c3c28eea99a2" gracePeriod=30
Oct 30 01:02:23.872806 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:23.872770    5002 patch_prober.go:29] interesting pod/e2e-inf-a-ef1084-openshift-default-68989c6b9f-lrmtq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body=
Oct 30 01:02:23.873212 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:23.873184    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ef1084-openshift-default-68989c6b9f-lrmtq" podUID="1618b1d3-81ce-4def-9498-43b3fdbc46a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused"
Oct 30 01:02:24.869157 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:24.869118    5002 patch_prober.go:29] interesting pod/e2e-inf-a-ef1084-openshift-default-68989c6b9f-lrmtq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body=
Oct 30 01:02:24.869769 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:24.869730    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ef1084-openshift-default-68989c6b9f-lrmtq" podUID="1618b1d3-81ce-4def-9498-43b3fdbc46a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused"
Oct 30 01:02:25.872458 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:25.871741    5002 patch_prober.go:29] interesting pod/e2e-inf-a-ef1084-openshift-default-68989c6b9f-lrmtq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body=
Oct 30 01:02:25.872458 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:25.871819    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ef1084-openshift-default-68989c6b9f-lrmtq" podUID="1618b1d3-81ce-4def-9498-43b3fdbc46a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused"
Oct 30 01:02:26.874790 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:26.874757    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-ef1084-openshift-default-68989c6b9f-lrmtq"
Oct 30 01:02:26.875255 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:26.875231    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-ef1084-openshift-default-68989c6b9f-lrmtq"
Oct 30 01:02:26.876088 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:26.876064    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-ef1084-openshift-default-68989c6b9f-lrmtq"
Oct 30 01:02:28.192698 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:28.192667    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-gvb85"
Oct 30 01:02:28.260335 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:28.260295    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70f303a4-967c-4a2f-8378-f30e81101a66-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70f303a4-967c-4a2f-8378-f30e81101a66-credential-socket\") pod \"70f303a4-967c-4a2f-8378-f30e81101a66\" (UID: \"70f303a4-967c-4a2f-8378-f30e81101a66\") "
Oct 30 01:02:28.260569 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:28.260504    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70f303a4-967c-4a2f-8378-f30e81101a66-kube-api-access-rvv8q\" (UniqueName: \"kubernetes.io/projected/70f303a4-967c-4a2f-8378-f30e81101a66-kube-api-access-rvv8q\") pod \"70f303a4-967c-4a2f-8378-f30e81101a66\" (UID: \"70f303a4-967c-4a2f-8378-f30e81101a66\") "
Oct 30 01:02:28.260569 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:28.260551    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70f303a4-967c-4a2f-8378-f30e81101a66-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70f303a4-967c-4a2f-8378-f30e81101a66-istio-data\") pod \"70f303a4-967c-4a2f-8378-f30e81101a66\" (UID: \"70f303a4-967c-4a2f-8378-f30e81101a66\") "
Oct 30 01:02:28.260707 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:28.260602    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/70f303a4-967c-4a2f-8378-f30e81101a66-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70f303a4-967c-4a2f-8378-f30e81101a66-istio-podinfo\") pod \"70f303a4-967c-4a2f-8378-f30e81101a66\" (UID: \"70f303a4-967c-4a2f-8378-f30e81101a66\") "
Oct 30 01:02:28.260763 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:28.260724    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70f303a4-967c-4a2f-8378-f30e81101a66-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70f303a4-967c-4a2f-8378-f30e81101a66-workload-certs\") pod \"70f303a4-967c-4a2f-8378-f30e81101a66\" (UID: \"70f303a4-967c-4a2f-8378-f30e81101a66\") "
Oct 30 01:02:28.260845 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:28.260827    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70f303a4-967c-4a2f-8378-f30e81101a66-istio-token\" (UniqueName: \"kubernetes.io/projected/70f303a4-967c-4a2f-8378-f30e81101a66-istio-token\") pod \"70f303a4-967c-4a2f-8378-f30e81101a66\" (UID: \"70f303a4-967c-4a2f-8378-f30e81101a66\") "
Oct 30 01:02:28.260891 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:28.260867    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70f303a4-967c-4a2f-8378-f30e81101a66-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70f303a4-967c-4a2f-8378-f30e81101a66-istio-envoy\") pod \"70f303a4-967c-4a2f-8378-f30e81101a66\" (UID: \"70f303a4-967c-4a2f-8378-f30e81101a66\") "
Oct 30 01:02:28.260937 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:28.260900    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70f303a4-967c-4a2f-8378-f30e81101a66-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70f303a4-967c-4a2f-8378-f30e81101a66-istiod-ca-cert\") pod \"70f303a4-967c-4a2f-8378-f30e81101a66\" (UID: \"70f303a4-967c-4a2f-8378-f30e81101a66\") "
Oct 30 01:02:28.260973 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:28.260949    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70f303a4-967c-4a2f-8378-f30e81101a66-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70f303a4-967c-4a2f-8378-f30e81101a66-workload-socket\") pod \"70f303a4-967c-4a2f-8378-f30e81101a66\" (UID: \"70f303a4-967c-4a2f-8378-f30e81101a66\") "
Oct 30 01:02:28.263124 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:28.262725    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70f303a4-967c-4a2f-8378-f30e81101a66-istio-data" pod "70f303a4-967c-4a2f-8378-f30e81101a66" (UID: "70f303a4-967c-4a2f-8378-f30e81101a66"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:02:28.263124 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:28.263047    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70f303a4-967c-4a2f-8378-f30e81101a66-credential-socket" pod "70f303a4-967c-4a2f-8378-f30e81101a66" (UID: "70f303a4-967c-4a2f-8378-f30e81101a66"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:02:28.263124 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:28.263101    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70f303a4-967c-4a2f-8378-f30e81101a66-istiod-ca-cert" pod "70f303a4-967c-4a2f-8378-f30e81101a66" (UID: "70f303a4-967c-4a2f-8378-f30e81101a66"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:02:28.265747 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:28.265721    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70f303a4-967c-4a2f-8378-f30e81101a66-workload-certs" pod "70f303a4-967c-4a2f-8378-f30e81101a66" (UID: "70f303a4-967c-4a2f-8378-f30e81101a66"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:02:28.265998 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:28.265979    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70f303a4-967c-4a2f-8378-f30e81101a66-workload-socket" pod "70f303a4-967c-4a2f-8378-f30e81101a66" (UID: "70f303a4-967c-4a2f-8378-f30e81101a66"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:02:28.274889 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:28.274788    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70f303a4-967c-4a2f-8378-f30e81101a66-istio-envoy" pod "70f303a4-967c-4a2f-8378-f30e81101a66" (UID: "70f303a4-967c-4a2f-8378-f30e81101a66"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:02:28.274889 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:28.274783    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/70f303a4-967c-4a2f-8378-f30e81101a66-istio-podinfo" pod "70f303a4-967c-4a2f-8378-f30e81101a66" (UID: "70f303a4-967c-4a2f-8378-f30e81101a66"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:02:28.275070 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:28.274904    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70f303a4-967c-4a2f-8378-f30e81101a66-istio-token" pod "70f303a4-967c-4a2f-8378-f30e81101a66" (UID: "70f303a4-967c-4a2f-8378-f30e81101a66"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:02:28.275070 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:28.274938    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70f303a4-967c-4a2f-8378-f30e81101a66-kube-api-access-rvv8q" pod "70f303a4-967c-4a2f-8378-f30e81101a66" (UID: "70f303a4-967c-4a2f-8378-f30e81101a66"). InnerVolumeSpecName "kube-api-access-rvv8q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:02:28.362895 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:28.362856    5002 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70f303a4-967c-4a2f-8378-f30e81101a66-istiod-ca-cert\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:02:28.362895 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:28.362893    5002 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70f303a4-967c-4a2f-8378-f30e81101a66-workload-socket\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:02:28.363148 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:28.362909    5002 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70f303a4-967c-4a2f-8378-f30e81101a66-credential-socket\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:02:28.363148 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:28.362922    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvv8q\" (UniqueName: \"kubernetes.io/projected/70f303a4-967c-4a2f-8378-f30e81101a66-kube-api-access-rvv8q\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:02:28.363148 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:28.362937    5002 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70f303a4-967c-4a2f-8378-f30e81101a66-istio-data\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:02:28.363148 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:28.362949    5002 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70f303a4-967c-4a2f-8378-f30e81101a66-istio-podinfo\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:02:28.363148 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:28.362960    5002 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70f303a4-967c-4a2f-8378-f30e81101a66-workload-certs\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:02:28.363148 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:28.362971    5002 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70f303a4-967c-4a2f-8378-f30e81101a66-istio-token\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:02:28.363148 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:28.362982    5002 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70f303a4-967c-4a2f-8378-f30e81101a66-istio-envoy\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:02:28.397024 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:28.396983    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-k66r8"]
Oct 30 01:02:28.398482 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:28.398440    5002 cpu_manager.go:414] "containerMap: removing container" podUID="750a98de-3674-4c5b-835c-48b928841cd7" containerName="istio-proxy"
Oct 30 01:02:28.398482 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:28.398469    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="750a98de-3674-4c5b-835c-48b928841cd7" containerName="istio-proxy"
Oct 30 01:02:28.398678 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:28.398501    5002 cpu_manager.go:414] "containerMap: removing container" podUID="70f303a4-967c-4a2f-8378-f30e81101a66" containerName="istio-proxy"
Oct 30 01:02:28.398678 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:28.398509    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70f303a4-967c-4a2f-8378-f30e81101a66" containerName="istio-proxy"
Oct 30 01:02:28.398678 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:28.398546    5002 cpu_manager.go:414] "containerMap: removing container" podUID="750a98de-3674-4c5b-835c-48b928841cd7" containerName="istio-proxy"
Oct 30 01:02:28.398678 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:28.398553    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="750a98de-3674-4c5b-835c-48b928841cd7" containerName="istio-proxy"
Oct 30 01:02:28.398678 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:28.398578    5002 cpu_manager.go:414] "containerMap: removing container" podUID="256f58c3-48bd-4f56-8500-ea60e5ec8efc" containerName="revoke-keys"
Oct 30 01:02:28.398678 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:28.398584    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="256f58c3-48bd-4f56-8500-ea60e5ec8efc" containerName="revoke-keys"
Oct 30 01:02:28.399738 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:28.398958    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="750a98de-3674-4c5b-835c-48b928841cd7" containerName="istio-proxy"
Oct 30 01:02:28.399738 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:28.399003    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="70f303a4-967c-4a2f-8378-f30e81101a66" containerName="istio-proxy"
Oct 30 01:02:28.399738 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:28.399021    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="750a98de-3674-4c5b-835c-48b928841cd7" containerName="istio-proxy"
Oct 30 01:02:28.399738 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:28.399038    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="256f58c3-48bd-4f56-8500-ea60e5ec8efc" containerName="revoke-keys"
Oct 30 01:02:28.409782 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:28.409754    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-k66r8"
Oct 30 01:02:28.430686 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:28.430653    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-k66r8"]
Oct 30 01:02:28.440653 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:28.440623    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-cce513-openshift-default-dockercfg-7qp6x\""
Oct 30 01:02:28.469546 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:28.469509    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05c750c5-39da-44a2-8414-3d7ffe09812d-istio-data\") pod \"e2e-shared-b-w1-cce513-openshift-default-84cc9879-k66r8\" (UID: \"05c750c5-39da-44a2-8414-3d7ffe09812d\") " pod="openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-k66r8"
Oct 30 01:02:28.469899 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:28.469866    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05c750c5-39da-44a2-8414-3d7ffe09812d-workload-certs\") pod \"e2e-shared-b-w1-cce513-openshift-default-84cc9879-k66r8\" (UID: \"05c750c5-39da-44a2-8414-3d7ffe09812d\") " pod="openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-k66r8"
Oct 30 01:02:28.470029 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:28.469934    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05c750c5-39da-44a2-8414-3d7ffe09812d-istio-token\") pod \"e2e-shared-b-w1-cce513-openshift-default-84cc9879-k66r8\" (UID: \"05c750c5-39da-44a2-8414-3d7ffe09812d\") " pod="openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-k66r8"
Oct 30 01:02:28.470029 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:28.469968    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05c750c5-39da-44a2-8414-3d7ffe09812d-istio-podinfo\") pod \"e2e-shared-b-w1-cce513-openshift-default-84cc9879-k66r8\" (UID: \"05c750c5-39da-44a2-8414-3d7ffe09812d\") " pod="openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-k66r8"
Oct 30 01:02:28.470029 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:28.470013    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05c750c5-39da-44a2-8414-3d7ffe09812d-workload-socket\") pod \"e2e-shared-b-w1-cce513-openshift-default-84cc9879-k66r8\" (UID: \"05c750c5-39da-44a2-8414-3d7ffe09812d\") " pod="openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-k66r8"
Oct 30 01:02:28.470213 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:28.470131    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05c750c5-39da-44a2-8414-3d7ffe09812d-istio-envoy\") pod \"e2e-shared-b-w1-cce513-openshift-default-84cc9879-k66r8\" (UID: \"05c750c5-39da-44a2-8414-3d7ffe09812d\") " pod="openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-k66r8"
Oct 30 01:02:28.470480 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:28.470448    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05c750c5-39da-44a2-8414-3d7ffe09812d-istiod-ca-cert\") pod \"e2e-shared-b-w1-cce513-openshift-default-84cc9879-k66r8\" (UID: \"05c750c5-39da-44a2-8414-3d7ffe09812d\") " pod="openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-k66r8"
Oct 30 01:02:28.470597 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:28.470508    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzwvj\" (UniqueName: \"kubernetes.io/projected/05c750c5-39da-44a2-8414-3d7ffe09812d-kube-api-access-zzwvj\") pod \"e2e-shared-b-w1-cce513-openshift-default-84cc9879-k66r8\" (UID: \"05c750c5-39da-44a2-8414-3d7ffe09812d\") " pod="openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-k66r8"
Oct 30 01:02:28.470597 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:28.470581    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05c750c5-39da-44a2-8414-3d7ffe09812d-credential-socket\") pod \"e2e-shared-b-w1-cce513-openshift-default-84cc9879-k66r8\" (UID: \"05c750c5-39da-44a2-8414-3d7ffe09812d\") " pod="openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-k66r8"
Oct 30 01:02:28.572472 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:28.572435    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05c750c5-39da-44a2-8414-3d7ffe09812d-istio-token\") pod \"e2e-shared-b-w1-cce513-openshift-default-84cc9879-k66r8\" (UID: \"05c750c5-39da-44a2-8414-3d7ffe09812d\") " pod="openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-k66r8"
Oct 30 01:02:28.572679 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:28.572480    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05c750c5-39da-44a2-8414-3d7ffe09812d-istio-podinfo\") pod \"e2e-shared-b-w1-cce513-openshift-default-84cc9879-k66r8\" (UID: \"05c750c5-39da-44a2-8414-3d7ffe09812d\") " pod="openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-k66r8"
Oct 30 01:02:28.572679 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:28.572512    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05c750c5-39da-44a2-8414-3d7ffe09812d-workload-socket\") pod \"e2e-shared-b-w1-cce513-openshift-default-84cc9879-k66r8\" (UID: \"05c750c5-39da-44a2-8414-3d7ffe09812d\") " pod="openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-k66r8"
Oct 30 01:02:28.572679 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:28.572594    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05c750c5-39da-44a2-8414-3d7ffe09812d-istio-envoy\") pod \"e2e-shared-b-w1-cce513-openshift-default-84cc9879-k66r8\" (UID: \"05c750c5-39da-44a2-8414-3d7ffe09812d\") " pod="openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-k66r8"
Oct 30 01:02:28.572885 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:28.572734    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05c750c5-39da-44a2-8414-3d7ffe09812d-istiod-ca-cert\") pod \"e2e-shared-b-w1-cce513-openshift-default-84cc9879-k66r8\" (UID: \"05c750c5-39da-44a2-8414-3d7ffe09812d\") " pod="openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-k66r8"
Oct 30 01:02:28.572885 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:28.572772    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzwvj\" (UniqueName: \"kubernetes.io/projected/05c750c5-39da-44a2-8414-3d7ffe09812d-kube-api-access-zzwvj\") pod \"e2e-shared-b-w1-cce513-openshift-default-84cc9879-k66r8\" (UID: \"05c750c5-39da-44a2-8414-3d7ffe09812d\") " pod="openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-k66r8"
Oct 30 01:02:28.572885 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:28.572802    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05c750c5-39da-44a2-8414-3d7ffe09812d-credential-socket\") pod \"e2e-shared-b-w1-cce513-openshift-default-84cc9879-k66r8\" (UID: \"05c750c5-39da-44a2-8414-3d7ffe09812d\") " pod="openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-k66r8"
Oct 30 01:02:28.573048 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:28.572901    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05c750c5-39da-44a2-8414-3d7ffe09812d-istio-data\") pod \"e2e-shared-b-w1-cce513-openshift-default-84cc9879-k66r8\" (UID: \"05c750c5-39da-44a2-8414-3d7ffe09812d\") " pod="openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-k66r8"
Oct 30 01:02:28.573048 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:28.572961    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05c750c5-39da-44a2-8414-3d7ffe09812d-workload-certs\") pod \"e2e-shared-b-w1-cce513-openshift-default-84cc9879-k66r8\" (UID: \"05c750c5-39da-44a2-8414-3d7ffe09812d\") " pod="openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-k66r8"
Oct 30 01:02:28.573173 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:28.573079    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05c750c5-39da-44a2-8414-3d7ffe09812d-workload-socket\") pod \"e2e-shared-b-w1-cce513-openshift-default-84cc9879-k66r8\" (UID: \"05c750c5-39da-44a2-8414-3d7ffe09812d\") " pod="openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-k66r8"
Oct 30 01:02:28.573607 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:28.573575    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05c750c5-39da-44a2-8414-3d7ffe09812d-istiod-ca-cert\") pod \"e2e-shared-b-w1-cce513-openshift-default-84cc9879-k66r8\" (UID: \"05c750c5-39da-44a2-8414-3d7ffe09812d\") " pod="openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-k66r8"
Oct 30 01:02:28.573857 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:28.573833    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05c750c5-39da-44a2-8414-3d7ffe09812d-credential-socket\") pod \"e2e-shared-b-w1-cce513-openshift-default-84cc9879-k66r8\" (UID: \"05c750c5-39da-44a2-8414-3d7ffe09812d\") " pod="openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-k66r8"
Oct 30 01:02:28.573962 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:28.573934    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05c750c5-39da-44a2-8414-3d7ffe09812d-istio-data\") pod \"e2e-shared-b-w1-cce513-openshift-default-84cc9879-k66r8\" (UID: \"05c750c5-39da-44a2-8414-3d7ffe09812d\") " pod="openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-k66r8"
Oct 30 01:02:28.574075 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:28.574054    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05c750c5-39da-44a2-8414-3d7ffe09812d-workload-certs\") pod \"e2e-shared-b-w1-cce513-openshift-default-84cc9879-k66r8\" (UID: \"05c750c5-39da-44a2-8414-3d7ffe09812d\") " pod="openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-k66r8"
Oct 30 01:02:28.575296 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:28.575274    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05c750c5-39da-44a2-8414-3d7ffe09812d-istio-envoy\") pod \"e2e-shared-b-w1-cce513-openshift-default-84cc9879-k66r8\" (UID: \"05c750c5-39da-44a2-8414-3d7ffe09812d\") " pod="openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-k66r8"
Oct 30 01:02:28.575440 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:28.575363    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05c750c5-39da-44a2-8414-3d7ffe09812d-istio-podinfo\") pod \"e2e-shared-b-w1-cce513-openshift-default-84cc9879-k66r8\" (UID: \"05c750c5-39da-44a2-8414-3d7ffe09812d\") " pod="openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-k66r8"
Oct 30 01:02:28.600145 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:28.600114    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05c750c5-39da-44a2-8414-3d7ffe09812d-istio-token\") pod \"e2e-shared-b-w1-cce513-openshift-default-84cc9879-k66r8\" (UID: \"05c750c5-39da-44a2-8414-3d7ffe09812d\") " pod="openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-k66r8"
Oct 30 01:02:28.600606 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:28.600580    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzwvj\" (UniqueName: \"kubernetes.io/projected/05c750c5-39da-44a2-8414-3d7ffe09812d-kube-api-access-zzwvj\") pod \"e2e-shared-b-w1-cce513-openshift-default-84cc9879-k66r8\" (UID: \"05c750c5-39da-44a2-8414-3d7ffe09812d\") " pod="openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-k66r8"
Oct 30 01:02:28.724337 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:28.724299    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-k66r8"
Oct 30 01:02:28.852397 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:28.852334    5002 generic.go:356] "Generic (PLEG): container finished" podID="70f303a4-967c-4a2f-8378-f30e81101a66" containerID="fb9406b986c59d5abf58064fd61d7063bb19cf99e799e33f6964c3c28eea99a2" exitCode=0
Oct 30 01:02:28.853589 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:28.853554    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-gvb85"
Oct 30 01:02:28.854217 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:28.854193    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-gvb85" event={"ID":"70f303a4-967c-4a2f-8378-f30e81101a66","Type":"ContainerDied","Data":"fb9406b986c59d5abf58064fd61d7063bb19cf99e799e33f6964c3c28eea99a2"}
Oct 30 01:02:28.854310 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:28.854243    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-gvb85" event={"ID":"70f303a4-967c-4a2f-8378-f30e81101a66","Type":"ContainerDied","Data":"93af0cfc599394d0ac6abed5cf670450a5206a86289f3a3d545c2d713158d339"}
Oct 30 01:02:28.854310 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:28.854271    5002 scope.go:122] "RemoveContainer" containerID="fb9406b986c59d5abf58064fd61d7063bb19cf99e799e33f6964c3c28eea99a2"
Oct 30 01:02:28.875593 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:28.875566    5002 scope.go:122] "RemoveContainer" containerID="fb9406b986c59d5abf58064fd61d7063bb19cf99e799e33f6964c3c28eea99a2"
Oct 30 01:02:28.875923 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:02:28.875900    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb9406b986c59d5abf58064fd61d7063bb19cf99e799e33f6964c3c28eea99a2\": container with ID starting with fb9406b986c59d5abf58064fd61d7063bb19cf99e799e33f6964c3c28eea99a2 not found: ID does not exist" containerID="fb9406b986c59d5abf58064fd61d7063bb19cf99e799e33f6964c3c28eea99a2"
Oct 30 01:02:28.875994 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:28.875936    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb9406b986c59d5abf58064fd61d7063bb19cf99e799e33f6964c3c28eea99a2"} err="failed to get container status \"fb9406b986c59d5abf58064fd61d7063bb19cf99e799e33f6964c3c28eea99a2\": rpc error: code = NotFound desc = could not find container \"fb9406b986c59d5abf58064fd61d7063bb19cf99e799e33f6964c3c28eea99a2\": container with ID starting with fb9406b986c59d5abf58064fd61d7063bb19cf99e799e33f6964c3c28eea99a2 not found: ID does not exist"
Oct 30 01:02:28.916998 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:28.916964    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-gvb85"]
Oct 30 01:02:28.931450 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:28.931416    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-gvb85"]
Oct 30 01:02:29.044585 ip-10-0-128-7 kubenswrapper[5002]: W1030 01:02:29.044552    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05c750c5_39da_44a2_8414_3d7ffe09812d.slice/crio-acecf27eec63f68d538ec430c91d8f9a860785efc088ed6a54b8fa03ecdb920c WatchSource:0}: Error finding container acecf27eec63f68d538ec430c91d8f9a860785efc088ed6a54b8fa03ecdb920c: Status 404 returned error can't find the container with id acecf27eec63f68d538ec430c91d8f9a860785efc088ed6a54b8fa03ecdb920c
Oct 30 01:02:29.045639 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:29.045613    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-k66r8"]
Oct 30 01:02:29.047525 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:29.047491    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:02:29.047611 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:29.047568    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:02:29.047687 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:29.047614    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:02:29.919197 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:29.919156    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-k66r8" event={"ID":"05c750c5-39da-44a2-8414-3d7ffe09812d","Type":"ContainerStarted","Data":"472ce76d6c1beb7fc03522ef5e89b6db866188096422af5fcf1ed1e16a23aef1"}
Oct 30 01:02:29.919819 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:29.919790    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-k66r8" event={"ID":"05c750c5-39da-44a2-8414-3d7ffe09812d","Type":"ContainerStarted","Data":"acecf27eec63f68d538ec430c91d8f9a860785efc088ed6a54b8fa03ecdb920c"}
Oct 30 01:02:29.999417 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:29.996814    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-19bc68-54864556d8-n2648"]
Oct 30 01:02:30.004186 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:30.004124    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-k66r8" podStartSLOduration=2.004104098 podStartE2EDuration="2.004104098s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:02:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:02:30.003047564 +0000 UTC m=+1845.619480456" watchObservedRunningTime="2026-10-30 01:02:30.004104098 +0000 UTC m=+1845.620536979"
Oct 30 01:02:30.004888 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:30.004810    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-19bc68-54864556d8-n2648"
Oct 30 01:02:30.013784 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:30.013750    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-19bc68-54864556d8-n2648"]
Oct 30 01:02:30.024704 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:30.024680    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-19bc68\""
Oct 30 01:02:30.046554 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:30.046516    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-19bc68-59cc54d647-zvgzf"]
Oct 30 01:02:30.102909 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:30.100867    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-19bc68-59cc54d647-zvgzf"
Oct 30 01:02:30.234402 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:30.234167    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-19bc68-dockercfg-vmlxb\""
Oct 30 01:02:30.283668 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:30.266549    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vp655\" (UniqueName: \"kubernetes.io/projected/64d5b052-11b9-46e0-b159-d43050f9778b-kube-api-access-vp655\") pod \"payload-processing-e2e-inf-b-19bc68-54864556d8-n2648\" (UID: \"64d5b052-11b9-46e0-b159-d43050f9778b\") " pod="openshift-ingress/payload-processing-e2e-inf-b-19bc68-54864556d8-n2648"
Oct 30 01:02:30.283668 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:30.266680    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64d5b052-11b9-46e0-b159-d43050f9778b-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-19bc68-54864556d8-n2648\" (UID: \"64d5b052-11b9-46e0-b159-d43050f9778b\") " pod="openshift-ingress/payload-processing-e2e-inf-b-19bc68-54864556d8-n2648"
Oct 30 01:02:30.283668 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:30.266869    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6xqk\" (UniqueName: \"kubernetes.io/projected/9edfd30b-6f37-4849-9dd8-b5b294c003d7-kube-api-access-p6xqk\") pod \"payload-pre-processing-e2e-inf-b-19bc68-59cc54d647-zvgzf\" (UID: \"9edfd30b-6f37-4849-9dd8-b5b294c003d7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-19bc68-59cc54d647-zvgzf"
Oct 30 01:02:30.283668 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:30.267013    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9edfd30b-6f37-4849-9dd8-b5b294c003d7-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-19bc68-59cc54d647-zvgzf\" (UID: \"9edfd30b-6f37-4849-9dd8-b5b294c003d7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-19bc68-59cc54d647-zvgzf"
Oct 30 01:02:30.417404 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:30.383472    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6xqk\" (UniqueName: \"kubernetes.io/projected/9edfd30b-6f37-4849-9dd8-b5b294c003d7-kube-api-access-p6xqk\") pod \"payload-pre-processing-e2e-inf-b-19bc68-59cc54d647-zvgzf\" (UID: \"9edfd30b-6f37-4849-9dd8-b5b294c003d7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-19bc68-59cc54d647-zvgzf"
Oct 30 01:02:30.443847 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:30.430602    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9edfd30b-6f37-4849-9dd8-b5b294c003d7-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-19bc68-59cc54d647-zvgzf\" (UID: \"9edfd30b-6f37-4849-9dd8-b5b294c003d7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-19bc68-59cc54d647-zvgzf"
Oct 30 01:02:30.449946 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:30.444455    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vp655\" (UniqueName: \"kubernetes.io/projected/64d5b052-11b9-46e0-b159-d43050f9778b-kube-api-access-vp655\") pod \"payload-processing-e2e-inf-b-19bc68-54864556d8-n2648\" (UID: \"64d5b052-11b9-46e0-b159-d43050f9778b\") " pod="openshift-ingress/payload-processing-e2e-inf-b-19bc68-54864556d8-n2648"
Oct 30 01:02:30.449946 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:30.444648    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64d5b052-11b9-46e0-b159-d43050f9778b-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-19bc68-54864556d8-n2648\" (UID: \"64d5b052-11b9-46e0-b159-d43050f9778b\") " pod="openshift-ingress/payload-processing-e2e-inf-b-19bc68-54864556d8-n2648"
Oct 30 01:02:30.449946 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:30.445990    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64d5b052-11b9-46e0-b159-d43050f9778b-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-19bc68-54864556d8-n2648\" (UID: \"64d5b052-11b9-46e0-b159-d43050f9778b\") " pod="openshift-ingress/payload-processing-e2e-inf-b-19bc68-54864556d8-n2648"
Oct 30 01:02:30.474400 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:30.471276    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9edfd30b-6f37-4849-9dd8-b5b294c003d7-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-19bc68-59cc54d647-zvgzf\" (UID: \"9edfd30b-6f37-4849-9dd8-b5b294c003d7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-19bc68-59cc54d647-zvgzf"
Oct 30 01:02:30.548104 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:30.546478    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vp655\" (UniqueName: \"kubernetes.io/projected/64d5b052-11b9-46e0-b159-d43050f9778b-kube-api-access-vp655\") pod \"payload-processing-e2e-inf-b-19bc68-54864556d8-n2648\" (UID: \"64d5b052-11b9-46e0-b159-d43050f9778b\") " pod="openshift-ingress/payload-processing-e2e-inf-b-19bc68-54864556d8-n2648"
Oct 30 01:02:30.579413 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:30.551074    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6xqk\" (UniqueName: \"kubernetes.io/projected/9edfd30b-6f37-4849-9dd8-b5b294c003d7-kube-api-access-p6xqk\") pod \"payload-pre-processing-e2e-inf-b-19bc68-59cc54d647-zvgzf\" (UID: \"9edfd30b-6f37-4849-9dd8-b5b294c003d7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-19bc68-59cc54d647-zvgzf"
Oct 30 01:02:30.605426 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:30.597459    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70f303a4-967c-4a2f-8378-f30e81101a66" path="/var/lib/kubelet/pods/70f303a4-967c-4a2f-8378-f30e81101a66/volumes"
Oct 30 01:02:30.605426 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:30.598004    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-19bc68-59cc54d647-zvgzf"]
Oct 30 01:02:30.624511 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:30.624476    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-19bc68-54864556d8-n2648"
Oct 30 01:02:30.724617 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:30.724577    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-k66r8"
Oct 30 01:02:30.726352 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:30.726320    5002 patch_prober.go:29] interesting pod/e2e-shared-b-w1-cce513-openshift-default-84cc9879-k66r8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body=
Oct 30 01:02:30.726881 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:30.726412    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-k66r8" podUID="05c750c5-39da-44a2-8414-3d7ffe09812d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused"
Oct 30 01:02:30.787778 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:30.787077    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-19bc68-59cc54d647-zvgzf"
Oct 30 01:02:30.995782 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:30.995498    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-19bc68-54864556d8-n2648"]
Oct 30 01:02:31.148994 ip-10-0-128-7 kubenswrapper[5002]: W1030 01:02:31.148961    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9edfd30b_6f37_4849_9dd8_b5b294c003d7.slice/crio-fccbea97d001348be473676d73ca0a358c620a51280a97aeac71b330a8ada7b0 WatchSource:0}: Error finding container fccbea97d001348be473676d73ca0a358c620a51280a97aeac71b330a8ada7b0: Status 404 returned error can't find the container with id fccbea97d001348be473676d73ca0a358c620a51280a97aeac71b330a8ada7b0
Oct 30 01:02:31.186358 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:31.186329    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-19bc68-59cc54d647-zvgzf"]
Oct 30 01:02:31.725056 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:31.725022    5002 patch_prober.go:29] interesting pod/e2e-shared-b-w1-cce513-openshift-default-84cc9879-k66r8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body=
Oct 30 01:02:31.725239 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:31.725082    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-k66r8" podUID="05c750c5-39da-44a2-8414-3d7ffe09812d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused"
Oct 30 01:02:31.986461 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:31.986063    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-19bc68-54864556d8-n2648" event={"ID":"64d5b052-11b9-46e0-b159-d43050f9778b","Type":"ContainerStarted","Data":"d533ea1be077eb77213d543d0426907e005d9d03ebf2c45f128ab3a0d37e37af"}
Oct 30 01:02:31.986461 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:31.986157    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-19bc68-54864556d8-n2648" event={"ID":"64d5b052-11b9-46e0-b159-d43050f9778b","Type":"ContainerStarted","Data":"943813a0faf51e81dc3e8874d3c0a48039fe14a7d65f1aecbba577e3d4a20105"}
Oct 30 01:02:31.992989 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:31.987726    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-19bc68-54864556d8-n2648"
Oct 30 01:02:31.994142 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:31.994108    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-19bc68-59cc54d647-zvgzf" event={"ID":"9edfd30b-6f37-4849-9dd8-b5b294c003d7","Type":"ContainerStarted","Data":"d9403cef7ffdc8103ed3d3008cb8846b1ea63e616d3edd548cf1323ce66b7c88"}
Oct 30 01:02:31.994290 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:31.994175    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-19bc68-59cc54d647-zvgzf" event={"ID":"9edfd30b-6f37-4849-9dd8-b5b294c003d7","Type":"ContainerStarted","Data":"fccbea97d001348be473676d73ca0a358c620a51280a97aeac71b330a8ada7b0"}
Oct 30 01:02:31.994516 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:31.994500    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-19bc68-59cc54d647-zvgzf"
Oct 30 01:02:32.033692 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:32.033326    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-19bc68-54864556d8-n2648" podStartSLOduration=3.033310078 podStartE2EDuration="3.033310078s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:02:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:02:32.030531317 +0000 UTC m=+1847.646964195" watchObservedRunningTime="2026-10-30 01:02:32.033310078 +0000 UTC m=+1847.649742958"
Oct 30 01:02:32.091408 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:32.078068    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-19bc68-59cc54d647-zvgzf" podStartSLOduration=2.078045354 podStartE2EDuration="2.078045354s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:02:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:02:32.076796261 +0000 UTC m=+1847.693229142" watchObservedRunningTime="2026-10-30 01:02:32.078045354 +0000 UTC m=+1847.694478234"
Oct 30 01:02:32.724759 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:32.724721    5002 patch_prober.go:29] interesting pod/e2e-shared-b-w1-cce513-openshift-default-84cc9879-k66r8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body=
Oct 30 01:02:32.724945 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:32.724777    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-k66r8" podUID="05c750c5-39da-44a2-8414-3d7ffe09812d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused"
Oct 30 01:02:33.725297 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:33.725266    5002 patch_prober.go:29] interesting pod/e2e-shared-b-w1-cce513-openshift-default-84cc9879-k66r8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body=
Oct 30 01:02:33.725712 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:33.725316    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-k66r8" podUID="05c750c5-39da-44a2-8414-3d7ffe09812d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused"
Oct 30 01:02:33.776406 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:33.775517    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-5007e1-openshift-default-64c644d8cc-wrs68"]
Oct 30 01:02:33.781406 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:33.780226    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-5007e1-openshift-default-64c644d8cc-wrs68"
Oct 30 01:02:33.794498 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:33.794470    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-5007e1-openshift-default-64c644d8cc-wrs68"]
Oct 30 01:02:33.796967 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:33.796935    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-5007e1-openshift-default-dockercfg-h59nd\""
Oct 30 01:02:33.883510 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:33.883460    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlrs9\" (UniqueName: \"kubernetes.io/projected/f864da04-4d22-42be-8334-d559fc8cdbe6-kube-api-access-dlrs9\") pod \"e2e-inf-b-5007e1-openshift-default-64c644d8cc-wrs68\" (UID: \"f864da04-4d22-42be-8334-d559fc8cdbe6\") " pod="openshift-ingress/e2e-inf-b-5007e1-openshift-default-64c644d8cc-wrs68"
Oct 30 01:02:33.883718 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:33.883613    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f864da04-4d22-42be-8334-d559fc8cdbe6-istio-data\") pod \"e2e-inf-b-5007e1-openshift-default-64c644d8cc-wrs68\" (UID: \"f864da04-4d22-42be-8334-d559fc8cdbe6\") " pod="openshift-ingress/e2e-inf-b-5007e1-openshift-default-64c644d8cc-wrs68"
Oct 30 01:02:33.883799 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:33.883730    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f864da04-4d22-42be-8334-d559fc8cdbe6-credential-socket\") pod \"e2e-inf-b-5007e1-openshift-default-64c644d8cc-wrs68\" (UID: \"f864da04-4d22-42be-8334-d559fc8cdbe6\") " pod="openshift-ingress/e2e-inf-b-5007e1-openshift-default-64c644d8cc-wrs68"
Oct 30 01:02:33.883799 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:33.883762    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f864da04-4d22-42be-8334-d559fc8cdbe6-istio-podinfo\") pod \"e2e-inf-b-5007e1-openshift-default-64c644d8cc-wrs68\" (UID: \"f864da04-4d22-42be-8334-d559fc8cdbe6\") " pod="openshift-ingress/e2e-inf-b-5007e1-openshift-default-64c644d8cc-wrs68"
Oct 30 01:02:33.883911 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:33.883885    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f864da04-4d22-42be-8334-d559fc8cdbe6-workload-socket\") pod \"e2e-inf-b-5007e1-openshift-default-64c644d8cc-wrs68\" (UID: \"f864da04-4d22-42be-8334-d559fc8cdbe6\") " pod="openshift-ingress/e2e-inf-b-5007e1-openshift-default-64c644d8cc-wrs68"
Oct 30 01:02:33.883970 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:33.883939    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f864da04-4d22-42be-8334-d559fc8cdbe6-istiod-ca-cert\") pod \"e2e-inf-b-5007e1-openshift-default-64c644d8cc-wrs68\" (UID: \"f864da04-4d22-42be-8334-d559fc8cdbe6\") " pod="openshift-ingress/e2e-inf-b-5007e1-openshift-default-64c644d8cc-wrs68"
Oct 30 01:02:33.884035 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:33.883966    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f864da04-4d22-42be-8334-d559fc8cdbe6-istio-envoy\") pod \"e2e-inf-b-5007e1-openshift-default-64c644d8cc-wrs68\" (UID: \"f864da04-4d22-42be-8334-d559fc8cdbe6\") " pod="openshift-ingress/e2e-inf-b-5007e1-openshift-default-64c644d8cc-wrs68"
Oct 30 01:02:33.884035 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:33.883988    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f864da04-4d22-42be-8334-d559fc8cdbe6-istio-token\") pod \"e2e-inf-b-5007e1-openshift-default-64c644d8cc-wrs68\" (UID: \"f864da04-4d22-42be-8334-d559fc8cdbe6\") " pod="openshift-ingress/e2e-inf-b-5007e1-openshift-default-64c644d8cc-wrs68"
Oct 30 01:02:33.884139 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:33.884066    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f864da04-4d22-42be-8334-d559fc8cdbe6-workload-certs\") pod \"e2e-inf-b-5007e1-openshift-default-64c644d8cc-wrs68\" (UID: \"f864da04-4d22-42be-8334-d559fc8cdbe6\") " pod="openshift-ingress/e2e-inf-b-5007e1-openshift-default-64c644d8cc-wrs68"
Oct 30 01:02:33.989182 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:33.989083    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlrs9\" (UniqueName: \"kubernetes.io/projected/f864da04-4d22-42be-8334-d559fc8cdbe6-kube-api-access-dlrs9\") pod \"e2e-inf-b-5007e1-openshift-default-64c644d8cc-wrs68\" (UID: \"f864da04-4d22-42be-8334-d559fc8cdbe6\") " pod="openshift-ingress/e2e-inf-b-5007e1-openshift-default-64c644d8cc-wrs68"
Oct 30 01:02:33.989847 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:33.989818    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f864da04-4d22-42be-8334-d559fc8cdbe6-istio-data\") pod \"e2e-inf-b-5007e1-openshift-default-64c644d8cc-wrs68\" (UID: \"f864da04-4d22-42be-8334-d559fc8cdbe6\") " pod="openshift-ingress/e2e-inf-b-5007e1-openshift-default-64c644d8cc-wrs68"
Oct 30 01:02:33.990170 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:33.990153    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f864da04-4d22-42be-8334-d559fc8cdbe6-credential-socket\") pod \"e2e-inf-b-5007e1-openshift-default-64c644d8cc-wrs68\" (UID: \"f864da04-4d22-42be-8334-d559fc8cdbe6\") " pod="openshift-ingress/e2e-inf-b-5007e1-openshift-default-64c644d8cc-wrs68"
Oct 30 01:02:33.990321 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:33.990306    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f864da04-4d22-42be-8334-d559fc8cdbe6-istio-podinfo\") pod \"e2e-inf-b-5007e1-openshift-default-64c644d8cc-wrs68\" (UID: \"f864da04-4d22-42be-8334-d559fc8cdbe6\") " pod="openshift-ingress/e2e-inf-b-5007e1-openshift-default-64c644d8cc-wrs68"
Oct 30 01:02:33.990577 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:33.990563    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f864da04-4d22-42be-8334-d559fc8cdbe6-workload-socket\") pod \"e2e-inf-b-5007e1-openshift-default-64c644d8cc-wrs68\" (UID: \"f864da04-4d22-42be-8334-d559fc8cdbe6\") " pod="openshift-ingress/e2e-inf-b-5007e1-openshift-default-64c644d8cc-wrs68"
Oct 30 01:02:33.990750 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:33.990736    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f864da04-4d22-42be-8334-d559fc8cdbe6-istiod-ca-cert\") pod \"e2e-inf-b-5007e1-openshift-default-64c644d8cc-wrs68\" (UID: \"f864da04-4d22-42be-8334-d559fc8cdbe6\") " pod="openshift-ingress/e2e-inf-b-5007e1-openshift-default-64c644d8cc-wrs68"
Oct 30 01:02:33.990875 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:33.990862    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f864da04-4d22-42be-8334-d559fc8cdbe6-istio-envoy\") pod \"e2e-inf-b-5007e1-openshift-default-64c644d8cc-wrs68\" (UID: \"f864da04-4d22-42be-8334-d559fc8cdbe6\") " pod="openshift-ingress/e2e-inf-b-5007e1-openshift-default-64c644d8cc-wrs68"
Oct 30 01:02:33.990964 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:33.990953    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f864da04-4d22-42be-8334-d559fc8cdbe6-istio-token\") pod \"e2e-inf-b-5007e1-openshift-default-64c644d8cc-wrs68\" (UID: \"f864da04-4d22-42be-8334-d559fc8cdbe6\") " pod="openshift-ingress/e2e-inf-b-5007e1-openshift-default-64c644d8cc-wrs68"
Oct 30 01:02:33.991110 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:33.991095    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f864da04-4d22-42be-8334-d559fc8cdbe6-workload-certs\") pod \"e2e-inf-b-5007e1-openshift-default-64c644d8cc-wrs68\" (UID: \"f864da04-4d22-42be-8334-d559fc8cdbe6\") " pod="openshift-ingress/e2e-inf-b-5007e1-openshift-default-64c644d8cc-wrs68"
Oct 30 01:02:33.991750 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:33.991726    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f864da04-4d22-42be-8334-d559fc8cdbe6-workload-socket\") pod \"e2e-inf-b-5007e1-openshift-default-64c644d8cc-wrs68\" (UID: \"f864da04-4d22-42be-8334-d559fc8cdbe6\") " pod="openshift-ingress/e2e-inf-b-5007e1-openshift-default-64c644d8cc-wrs68"
Oct 30 01:02:33.992711 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:33.992688    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f864da04-4d22-42be-8334-d559fc8cdbe6-credential-socket\") pod \"e2e-inf-b-5007e1-openshift-default-64c644d8cc-wrs68\" (UID: \"f864da04-4d22-42be-8334-d559fc8cdbe6\") " pod="openshift-ingress/e2e-inf-b-5007e1-openshift-default-64c644d8cc-wrs68"
Oct 30 01:02:33.993065 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:33.993017    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f864da04-4d22-42be-8334-d559fc8cdbe6-istio-data\") pod \"e2e-inf-b-5007e1-openshift-default-64c644d8cc-wrs68\" (UID: \"f864da04-4d22-42be-8334-d559fc8cdbe6\") " pod="openshift-ingress/e2e-inf-b-5007e1-openshift-default-64c644d8cc-wrs68"
Oct 30 01:02:33.993198 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:33.993167    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f864da04-4d22-42be-8334-d559fc8cdbe6-istiod-ca-cert\") pod \"e2e-inf-b-5007e1-openshift-default-64c644d8cc-wrs68\" (UID: \"f864da04-4d22-42be-8334-d559fc8cdbe6\") " pod="openshift-ingress/e2e-inf-b-5007e1-openshift-default-64c644d8cc-wrs68"
Oct 30 01:02:33.993436 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:33.993418    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f864da04-4d22-42be-8334-d559fc8cdbe6-workload-certs\") pod \"e2e-inf-b-5007e1-openshift-default-64c644d8cc-wrs68\" (UID: \"f864da04-4d22-42be-8334-d559fc8cdbe6\") " pod="openshift-ingress/e2e-inf-b-5007e1-openshift-default-64c644d8cc-wrs68"
Oct 30 01:02:33.997420 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:33.996025    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f864da04-4d22-42be-8334-d559fc8cdbe6-istio-podinfo\") pod \"e2e-inf-b-5007e1-openshift-default-64c644d8cc-wrs68\" (UID: \"f864da04-4d22-42be-8334-d559fc8cdbe6\") " pod="openshift-ingress/e2e-inf-b-5007e1-openshift-default-64c644d8cc-wrs68"
Oct 30 01:02:33.999467 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:33.998413    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f864da04-4d22-42be-8334-d559fc8cdbe6-istio-envoy\") pod \"e2e-inf-b-5007e1-openshift-default-64c644d8cc-wrs68\" (UID: \"f864da04-4d22-42be-8334-d559fc8cdbe6\") " pod="openshift-ingress/e2e-inf-b-5007e1-openshift-default-64c644d8cc-wrs68"
Oct 30 01:02:34.014992 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:34.014958    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlrs9\" (UniqueName: \"kubernetes.io/projected/f864da04-4d22-42be-8334-d559fc8cdbe6-kube-api-access-dlrs9\") pod \"e2e-inf-b-5007e1-openshift-default-64c644d8cc-wrs68\" (UID: \"f864da04-4d22-42be-8334-d559fc8cdbe6\") " pod="openshift-ingress/e2e-inf-b-5007e1-openshift-default-64c644d8cc-wrs68"
Oct 30 01:02:34.016861 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:34.016834    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f864da04-4d22-42be-8334-d559fc8cdbe6-istio-token\") pod \"e2e-inf-b-5007e1-openshift-default-64c644d8cc-wrs68\" (UID: \"f864da04-4d22-42be-8334-d559fc8cdbe6\") " pod="openshift-ingress/e2e-inf-b-5007e1-openshift-default-64c644d8cc-wrs68"
Oct 30 01:02:34.102256 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:34.102220    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-5007e1-openshift-default-64c644d8cc-wrs68"
Oct 30 01:02:34.310455 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:34.310275    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-99424893-422qg"]
Oct 30 01:02:34.316830 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:34.315360    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99424893-422qg"
Oct 30 01:02:34.332407 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:34.332291    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-99424893-422qg"]
Oct 30 01:02:34.332407 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:34.332376    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:02:34.332582 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:34.332555    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:02:34.332781 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:34.332766    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:02:34.332872 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:34.332817    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5phsh\""
Oct 30 01:02:34.406360 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:34.406314    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9866a8d4-b855-461b-bcb6-2dbd0bcdef0b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-99424893-422qg\" (UID: \"9866a8d4-b855-461b-bcb6-2dbd0bcdef0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99424893-422qg"
Oct 30 01:02:34.406598 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:34.406374    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9866a8d4-b855-461b-bcb6-2dbd0bcdef0b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-99424893-422qg\" (UID: \"9866a8d4-b855-461b-bcb6-2dbd0bcdef0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99424893-422qg"
Oct 30 01:02:34.406598 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:34.406549    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9v4dp\" (UniqueName: \"kubernetes.io/projected/9866a8d4-b855-461b-bcb6-2dbd0bcdef0b-kube-api-access-9v4dp\") pod \"maas-api-revoke-sub-99424893-422qg\" (UID: \"9866a8d4-b855-461b-bcb6-2dbd0bcdef0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99424893-422qg"
Oct 30 01:02:34.491228 ip-10-0-128-7 kubenswrapper[5002]: W1030 01:02:34.491192    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf864da04_4d22_42be_8334_d559fc8cdbe6.slice/crio-b35087288c809634a370301688bf5606d59e524e2999d46b267ef706037a1fdd WatchSource:0}: Error finding container b35087288c809634a370301688bf5606d59e524e2999d46b267ef706037a1fdd: Status 404 returned error can't find the container with id b35087288c809634a370301688bf5606d59e524e2999d46b267ef706037a1fdd
Oct 30 01:02:34.495769 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:34.495739    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:02:34.495922 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:34.495908    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:02:34.496000 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:34.495990    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:02:34.500169 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:34.500138    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-5007e1-openshift-default-64c644d8cc-wrs68"]
Oct 30 01:02:34.507776 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:34.507745    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9866a8d4-b855-461b-bcb6-2dbd0bcdef0b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-99424893-422qg\" (UID: \"9866a8d4-b855-461b-bcb6-2dbd0bcdef0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99424893-422qg"
Oct 30 01:02:34.507915 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:34.507786    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9866a8d4-b855-461b-bcb6-2dbd0bcdef0b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-99424893-422qg\" (UID: \"9866a8d4-b855-461b-bcb6-2dbd0bcdef0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99424893-422qg"
Oct 30 01:02:34.507915 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:34.507873    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9v4dp\" (UniqueName: \"kubernetes.io/projected/9866a8d4-b855-461b-bcb6-2dbd0bcdef0b-kube-api-access-9v4dp\") pod \"maas-api-revoke-sub-99424893-422qg\" (UID: \"9866a8d4-b855-461b-bcb6-2dbd0bcdef0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99424893-422qg"
Oct 30 01:02:34.508812 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:34.508792    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9866a8d4-b855-461b-bcb6-2dbd0bcdef0b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-99424893-422qg\" (UID: \"9866a8d4-b855-461b-bcb6-2dbd0bcdef0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99424893-422qg"
Oct 30 01:02:34.510536 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:34.510518    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9866a8d4-b855-461b-bcb6-2dbd0bcdef0b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-99424893-422qg\" (UID: \"9866a8d4-b855-461b-bcb6-2dbd0bcdef0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99424893-422qg"
Oct 30 01:02:34.532783 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:34.532759    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9v4dp\" (UniqueName: \"kubernetes.io/projected/9866a8d4-b855-461b-bcb6-2dbd0bcdef0b-kube-api-access-9v4dp\") pod \"maas-api-revoke-sub-99424893-422qg\" (UID: \"9866a8d4-b855-461b-bcb6-2dbd0bcdef0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99424893-422qg"
Oct 30 01:02:34.633791 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:34.633759    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99424893-422qg"
Oct 30 01:02:34.724862 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:34.724829    5002 patch_prober.go:29] interesting pod/e2e-shared-b-w1-cce513-openshift-default-84cc9879-k66r8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body=
Oct 30 01:02:34.725102 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:34.725073    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-k66r8" podUID="05c750c5-39da-44a2-8414-3d7ffe09812d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused"
Oct 30 01:02:35.031927 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:35.031885    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5007e1-openshift-default-64c644d8cc-wrs68" event={"ID":"f864da04-4d22-42be-8334-d559fc8cdbe6","Type":"ContainerStarted","Data":"0cfd0e885b2e6e16bdecc6f9cb22da07156f27ae8a8ab8684e4257e298ea736b"}
Oct 30 01:02:35.032572 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:35.032543    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5007e1-openshift-default-64c644d8cc-wrs68" event={"ID":"f864da04-4d22-42be-8334-d559fc8cdbe6","Type":"ContainerStarted","Data":"b35087288c809634a370301688bf5606d59e524e2999d46b267ef706037a1fdd"}
Oct 30 01:02:35.103174 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:35.103082    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-5007e1-openshift-default-64c644d8cc-wrs68"
Oct 30 01:02:35.104288 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:35.104256    5002 patch_prober.go:29] interesting pod/e2e-inf-b-5007e1-openshift-default-64c644d8cc-wrs68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body=
Oct 30 01:02:35.104462 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:35.104311    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5007e1-openshift-default-64c644d8cc-wrs68" podUID="f864da04-4d22-42be-8334-d559fc8cdbe6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused"
Oct 30 01:02:35.110030 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:35.109984    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-5007e1-openshift-default-64c644d8cc-wrs68" podStartSLOduration=2.109971728 podStartE2EDuration="2.109971728s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:02:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:02:35.10960464 +0000 UTC m=+1850.726037519" watchObservedRunningTime="2026-10-30 01:02:35.109971728 +0000 UTC m=+1850.726404608"
Oct 30 01:02:35.204479 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:35.204439    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29a0c0c1-8jlgp"]
Oct 30 01:02:35.214144 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:35.214113    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29a0c0c1-8jlgp"]
Oct 30 01:02:35.214308 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:35.214229    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29a0c0c1-8jlgp"
Oct 30 01:02:35.216611 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:35.216586    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-99424893-422qg"]
Oct 30 01:02:35.373460 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:35.350338    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d6db577-4b91-4d50-8ac5-7b73b24aeb80-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-29a0c0c1-8jlgp\" (UID: \"6d6db577-4b91-4d50-8ac5-7b73b24aeb80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29a0c0c1-8jlgp"
Oct 30 01:02:35.373460 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:35.350514    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d6db577-4b91-4d50-8ac5-7b73b24aeb80-maas-api-xks-ca\") pod \"maas-api-revoke-sub-29a0c0c1-8jlgp\" (UID: \"6d6db577-4b91-4d50-8ac5-7b73b24aeb80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29a0c0c1-8jlgp"
Oct 30 01:02:35.373460 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:35.350694    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kt4s8\" (UniqueName: \"kubernetes.io/projected/6d6db577-4b91-4d50-8ac5-7b73b24aeb80-kube-api-access-kt4s8\") pod \"maas-api-revoke-sub-29a0c0c1-8jlgp\" (UID: \"6d6db577-4b91-4d50-8ac5-7b73b24aeb80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29a0c0c1-8jlgp"
Oct 30 01:02:35.451840 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:35.451807    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d6db577-4b91-4d50-8ac5-7b73b24aeb80-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-29a0c0c1-8jlgp\" (UID: \"6d6db577-4b91-4d50-8ac5-7b73b24aeb80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29a0c0c1-8jlgp"
Oct 30 01:02:35.452017 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:35.451937    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d6db577-4b91-4d50-8ac5-7b73b24aeb80-maas-api-xks-ca\") pod \"maas-api-revoke-sub-29a0c0c1-8jlgp\" (UID: \"6d6db577-4b91-4d50-8ac5-7b73b24aeb80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29a0c0c1-8jlgp"
Oct 30 01:02:35.452127 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:35.452108    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kt4s8\" (UniqueName: \"kubernetes.io/projected/6d6db577-4b91-4d50-8ac5-7b73b24aeb80-kube-api-access-kt4s8\") pod \"maas-api-revoke-sub-29a0c0c1-8jlgp\" (UID: \"6d6db577-4b91-4d50-8ac5-7b73b24aeb80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29a0c0c1-8jlgp"
Oct 30 01:02:35.452723 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:35.452696    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d6db577-4b91-4d50-8ac5-7b73b24aeb80-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-29a0c0c1-8jlgp\" (UID: \"6d6db577-4b91-4d50-8ac5-7b73b24aeb80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29a0c0c1-8jlgp"
Oct 30 01:02:35.454813 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:35.454780    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d6db577-4b91-4d50-8ac5-7b73b24aeb80-maas-api-xks-ca\") pod \"maas-api-revoke-sub-29a0c0c1-8jlgp\" (UID: \"6d6db577-4b91-4d50-8ac5-7b73b24aeb80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29a0c0c1-8jlgp"
Oct 30 01:02:35.478527 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:35.478490    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kt4s8\" (UniqueName: \"kubernetes.io/projected/6d6db577-4b91-4d50-8ac5-7b73b24aeb80-kube-api-access-kt4s8\") pod \"maas-api-revoke-sub-29a0c0c1-8jlgp\" (UID: \"6d6db577-4b91-4d50-8ac5-7b73b24aeb80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29a0c0c1-8jlgp"
Oct 30 01:02:35.613832 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:35.613797    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29a0c0c1-8jlgp"
Oct 30 01:02:35.725036 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:35.725000    5002 patch_prober.go:29] interesting pod/e2e-shared-b-w1-cce513-openshift-default-84cc9879-k66r8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body=
Oct 30 01:02:35.725172 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:35.725060    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-k66r8" podUID="05c750c5-39da-44a2-8414-3d7ffe09812d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused"
Oct 30 01:02:36.013290 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:36.013258    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29a0c0c1-8jlgp"]
Oct 30 01:02:36.043247 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:36.043213    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29a0c0c1-8jlgp" event={"ID":"6d6db577-4b91-4d50-8ac5-7b73b24aeb80","Type":"ContainerStarted","Data":"e1719cdf651ac9832a45c3d3f9369f938d6b643ed823696d78e76bdaa6b3dad6"}
Oct 30 01:02:36.045424 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:36.045306    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99424893-422qg" event={"ID":"9866a8d4-b855-461b-bcb6-2dbd0bcdef0b","Type":"ContainerStarted","Data":"b20841f2f72abc8027f221c1090145bd17a4676d10931ca810b6877230ccf502"}
Oct 30 01:02:36.045424 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:36.045358    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99424893-422qg" event={"ID":"9866a8d4-b855-461b-bcb6-2dbd0bcdef0b","Type":"ContainerStarted","Data":"37b580983fca1264dbe28939892e3c3cc291c5180678991414632ca6f4de2013"}
Oct 30 01:02:36.107647 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:36.103604    5002 patch_prober.go:29] interesting pod/e2e-inf-b-5007e1-openshift-default-64c644d8cc-wrs68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body=
Oct 30 01:02:36.107647 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:36.103676    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5007e1-openshift-default-64c644d8cc-wrs68" podUID="f864da04-4d22-42be-8334-d559fc8cdbe6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused"
Oct 30 01:02:36.115408 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:36.114204    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99424893-422qg" podStartSLOduration=2.11418253 podStartE2EDuration="2.11418253s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:02:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:02:36.102920736 +0000 UTC m=+1851.719353640" watchObservedRunningTime="2026-10-30 01:02:36.11418253 +0000 UTC m=+1851.730615410"
Oct 30 01:02:36.730555 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:36.730522    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-k66r8"
Oct 30 01:02:36.731021 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:36.731001    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-k66r8"
Oct 30 01:02:36.731992 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:36.731977    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-k66r8"
Oct 30 01:02:37.092815 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:37.092717    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29a0c0c1-8jlgp" event={"ID":"6d6db577-4b91-4d50-8ac5-7b73b24aeb80","Type":"ContainerStarted","Data":"6b47c750fe98ef9ef3ee221b1171fbc2b633dccad9a283684553dd72673b1bda"}
Oct 30 01:02:37.095588 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:37.095554    5002 generic.go:356] "Generic (PLEG): container finished" podID="9866a8d4-b855-461b-bcb6-2dbd0bcdef0b" containerID="b20841f2f72abc8027f221c1090145bd17a4676d10931ca810b6877230ccf502" exitCode=0
Oct 30 01:02:37.095737 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:37.095635    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99424893-422qg" event={"ID":"9866a8d4-b855-461b-bcb6-2dbd0bcdef0b","Type":"ContainerDied","Data":"b20841f2f72abc8027f221c1090145bd17a4676d10931ca810b6877230ccf502"}
Oct 30 01:02:37.103142 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:37.103103    5002 patch_prober.go:29] interesting pod/e2e-inf-b-5007e1-openshift-default-64c644d8cc-wrs68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body=
Oct 30 01:02:37.103366 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:37.103342    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5007e1-openshift-default-64c644d8cc-wrs68" podUID="f864da04-4d22-42be-8334-d559fc8cdbe6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused"
Oct 30 01:02:37.141305 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:37.141250    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29a0c0c1-8jlgp" podStartSLOduration=2.141237598 podStartE2EDuration="2.141237598s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:02:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:02:37.137031553 +0000 UTC m=+1852.753464431" watchObservedRunningTime="2026-10-30 01:02:37.141237598 +0000 UTC m=+1852.757670508"
Oct 30 01:02:38.090591 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:38.090493    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7d8z4"]
Oct 30 01:02:38.103200 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:38.103163    5002 patch_prober.go:29] interesting pod/e2e-inf-b-5007e1-openshift-default-64c644d8cc-wrs68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body=
Oct 30 01:02:38.103742 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:38.103237    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5007e1-openshift-default-64c644d8cc-wrs68" podUID="f864da04-4d22-42be-8334-d559fc8cdbe6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused"
Oct 30 01:02:38.109705 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:38.109668    5002 generic.go:356] "Generic (PLEG): container finished" podID="6d6db577-4b91-4d50-8ac5-7b73b24aeb80" containerID="6b47c750fe98ef9ef3ee221b1171fbc2b633dccad9a283684553dd72673b1bda" exitCode=0
Oct 30 01:02:38.109848 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:38.109756    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29a0c0c1-8jlgp" event={"ID":"6d6db577-4b91-4d50-8ac5-7b73b24aeb80","Type":"ContainerDied","Data":"6b47c750fe98ef9ef3ee221b1171fbc2b633dccad9a283684553dd72673b1bda"}
Oct 30 01:02:38.510749 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:38.510717    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99424893-422qg"
Oct 30 01:02:38.575171 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:38.575041    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9866a8d4-b855-461b-bcb6-2dbd0bcdef0b-kube-api-access-9v4dp\" (UniqueName: \"kubernetes.io/projected/9866a8d4-b855-461b-bcb6-2dbd0bcdef0b-kube-api-access-9v4dp\") pod \"9866a8d4-b855-461b-bcb6-2dbd0bcdef0b\" (UID: \"9866a8d4-b855-461b-bcb6-2dbd0bcdef0b\") "
Oct 30 01:02:38.592813 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:38.591702    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9866a8d4-b855-461b-bcb6-2dbd0bcdef0b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9866a8d4-b855-461b-bcb6-2dbd0bcdef0b-maas-api-openshift-ca\") pod \"9866a8d4-b855-461b-bcb6-2dbd0bcdef0b\" (UID: \"9866a8d4-b855-461b-bcb6-2dbd0bcdef0b\") "
Oct 30 01:02:38.592813 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:38.591838    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9866a8d4-b855-461b-bcb6-2dbd0bcdef0b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9866a8d4-b855-461b-bcb6-2dbd0bcdef0b-maas-api-xks-ca\") pod \"9866a8d4-b855-461b-bcb6-2dbd0bcdef0b\" (UID: \"9866a8d4-b855-461b-bcb6-2dbd0bcdef0b\") "
Oct 30 01:02:38.593866 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:38.593839    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9866a8d4-b855-461b-bcb6-2dbd0bcdef0b-maas-api-openshift-ca" pod "9866a8d4-b855-461b-bcb6-2dbd0bcdef0b" (UID: "9866a8d4-b855-461b-bcb6-2dbd0bcdef0b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:02:38.602275 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:38.601929    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9866a8d4-b855-461b-bcb6-2dbd0bcdef0b-kube-api-access-9v4dp" pod "9866a8d4-b855-461b-bcb6-2dbd0bcdef0b" (UID: "9866a8d4-b855-461b-bcb6-2dbd0bcdef0b"). InnerVolumeSpecName "kube-api-access-9v4dp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:02:38.602539 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:38.602506    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9866a8d4-b855-461b-bcb6-2dbd0bcdef0b-maas-api-xks-ca" pod "9866a8d4-b855-461b-bcb6-2dbd0bcdef0b" (UID: "9866a8d4-b855-461b-bcb6-2dbd0bcdef0b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:02:38.695460 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:38.694544    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9v4dp\" (UniqueName: \"kubernetes.io/projected/9866a8d4-b855-461b-bcb6-2dbd0bcdef0b-kube-api-access-9v4dp\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:02:38.695460 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:38.694578    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9866a8d4-b855-461b-bcb6-2dbd0bcdef0b-maas-api-openshift-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:02:38.695460 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:38.694588    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9866a8d4-b855-461b-bcb6-2dbd0bcdef0b-maas-api-xks-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:02:39.103073 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:39.103032    5002 patch_prober.go:29] interesting pod/e2e-inf-b-5007e1-openshift-default-64c644d8cc-wrs68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body=
Oct 30 01:02:39.103284 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:39.103092    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5007e1-openshift-default-64c644d8cc-wrs68" podUID="f864da04-4d22-42be-8334-d559fc8cdbe6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused"
Oct 30 01:02:39.118511 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:39.118482    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99424893-422qg"
Oct 30 01:02:39.118651 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:39.118508    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99424893-422qg" event={"ID":"9866a8d4-b855-461b-bcb6-2dbd0bcdef0b","Type":"ContainerDied","Data":"37b580983fca1264dbe28939892e3c3cc291c5180678991414632ca6f4de2013"}
Oct 30 01:02:39.118651 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:39.118565    5002 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="37b580983fca1264dbe28939892e3c3cc291c5180678991414632ca6f4de2013"
Oct 30 01:02:39.513409 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:39.488421    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29a0c0c1-8jlgp"
Oct 30 01:02:39.530400 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:39.527145    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d6db577-4b91-4d50-8ac5-7b73b24aeb80-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d6db577-4b91-4d50-8ac5-7b73b24aeb80-maas-api-openshift-ca\") pod \"6d6db577-4b91-4d50-8ac5-7b73b24aeb80\" (UID: \"6d6db577-4b91-4d50-8ac5-7b73b24aeb80\") "
Oct 30 01:02:39.530400 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:39.527274    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d6db577-4b91-4d50-8ac5-7b73b24aeb80-kube-api-access-kt4s8\" (UniqueName: \"kubernetes.io/projected/6d6db577-4b91-4d50-8ac5-7b73b24aeb80-kube-api-access-kt4s8\") pod \"6d6db577-4b91-4d50-8ac5-7b73b24aeb80\" (UID: \"6d6db577-4b91-4d50-8ac5-7b73b24aeb80\") "
Oct 30 01:02:39.530400 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:39.527531    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6d6db577-4b91-4d50-8ac5-7b73b24aeb80-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d6db577-4b91-4d50-8ac5-7b73b24aeb80-maas-api-xks-ca\") pod \"6d6db577-4b91-4d50-8ac5-7b73b24aeb80\" (UID: \"6d6db577-4b91-4d50-8ac5-7b73b24aeb80\") "
Oct 30 01:02:39.530400 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:39.529346    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d6db577-4b91-4d50-8ac5-7b73b24aeb80-maas-api-openshift-ca" pod "6d6db577-4b91-4d50-8ac5-7b73b24aeb80" (UID: "6d6db577-4b91-4d50-8ac5-7b73b24aeb80"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:02:39.605399 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:39.604582    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d6db577-4b91-4d50-8ac5-7b73b24aeb80-maas-api-xks-ca" pod "6d6db577-4b91-4d50-8ac5-7b73b24aeb80" (UID: "6d6db577-4b91-4d50-8ac5-7b73b24aeb80"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:02:39.723413 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:39.697519    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d6db577-4b91-4d50-8ac5-7b73b24aeb80-maas-api-openshift-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:02:39.723413 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:39.697571    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d6db577-4b91-4d50-8ac5-7b73b24aeb80-maas-api-xks-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:02:39.740765 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:39.726094    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d6db577-4b91-4d50-8ac5-7b73b24aeb80-kube-api-access-kt4s8" pod "6d6db577-4b91-4d50-8ac5-7b73b24aeb80" (UID: "6d6db577-4b91-4d50-8ac5-7b73b24aeb80"). InnerVolumeSpecName "kube-api-access-kt4s8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:02:39.985458 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:39.985341    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kt4s8\" (UniqueName: \"kubernetes.io/projected/6d6db577-4b91-4d50-8ac5-7b73b24aeb80-kube-api-access-kt4s8\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:02:40.106211 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:40.106169    5002 patch_prober.go:29] interesting pod/e2e-inf-b-5007e1-openshift-default-64c644d8cc-wrs68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body=
Oct 30 01:02:40.106825 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:40.106787    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5007e1-openshift-default-64c644d8cc-wrs68" podUID="f864da04-4d22-42be-8334-d559fc8cdbe6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused"
Oct 30 01:02:40.417083 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:40.413987    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6f4fba/simulator-w5-6f4fba-kserve-5bf8dfddd5-wcwzh" podUID="3a2c2f8f-649c-4ab6-af98-5d87df0e511f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:02:40.459049 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:40.459009    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29a0c0c1-8jlgp" event={"ID":"6d6db577-4b91-4d50-8ac5-7b73b24aeb80","Type":"ContainerDied","Data":"e1719cdf651ac9832a45c3d3f9369f938d6b643ed823696d78e76bdaa6b3dad6"}
Oct 30 01:02:40.459321 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:40.459304    5002 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e1719cdf651ac9832a45c3d3f9369f938d6b643ed823696d78e76bdaa6b3dad6"
Oct 30 01:02:40.459559 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:40.459544    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29a0c0c1-8jlgp"
Oct 30 01:02:40.799556 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:40.799437    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b3bdf9-cx7hq"]
Oct 30 01:02:40.806433 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:40.806392    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b3bdf9-cx7hq"]
Oct 30 01:02:41.103253 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:41.103168    5002 patch_prober.go:29] interesting pod/e2e-inf-b-5007e1-openshift-default-64c644d8cc-wrs68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body=
Oct 30 01:02:41.103253 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:41.103234    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5007e1-openshift-default-64c644d8cc-wrs68" podUID="f864da04-4d22-42be-8334-d559fc8cdbe6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused"
Oct 30 01:02:42.102642 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:42.102606    5002 patch_prober.go:29] interesting pod/e2e-inf-b-5007e1-openshift-default-64c644d8cc-wrs68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body=
Oct 30 01:02:42.103061 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:42.102660    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5007e1-openshift-default-64c644d8cc-wrs68" podUID="f864da04-4d22-42be-8334-d559fc8cdbe6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused"
Oct 30 01:02:42.498518 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:42.498468    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="256f58c3-48bd-4f56-8500-ea60e5ec8efc" path="/var/lib/kubelet/pods/256f58c3-48bd-4f56-8500-ea60e5ec8efc/volumes"
Oct 30 01:02:42.866937 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:02:42.866848    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbjdq" containerName="extractor"
Oct 30 01:02:43.008799 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:43.008767    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-19bc68-59cc54d647-zvgzf"
Oct 30 01:02:43.113001 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:43.112970    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-5007e1-openshift-default-64c644d8cc-wrs68"
Oct 30 01:02:43.113999 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:43.113754    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-5007e1-openshift-default-64c644d8cc-wrs68"
Oct 30 01:02:43.114997 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:43.114977    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-5007e1-openshift-default-64c644d8cc-wrs68"
Oct 30 01:02:44.017778 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:44.017748    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-19bc68-54864556d8-n2648"
Oct 30 01:02:45.209659 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:45.209619    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-481ab7/test-model-481ab7-kserve-59c6875d8f-wz4bj"]
Oct 30 01:02:45.904680 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:45.904641    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-efcbc3bd-5q876"]
Oct 30 01:02:45.905842 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:45.905817    5002 cpu_manager.go:414] "containerMap: removing container" podUID="9866a8d4-b855-461b-bcb6-2dbd0bcdef0b" containerName="revoke-keys"
Oct 30 01:02:45.905967 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:45.905955    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9866a8d4-b855-461b-bcb6-2dbd0bcdef0b" containerName="revoke-keys"
Oct 30 01:02:45.906111 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:45.906099    5002 cpu_manager.go:414] "containerMap: removing container" podUID="6d6db577-4b91-4d50-8ac5-7b73b24aeb80" containerName="revoke-keys"
Oct 30 01:02:45.906191 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:45.906181    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d6db577-4b91-4d50-8ac5-7b73b24aeb80" containerName="revoke-keys"
Oct 30 01:02:45.906435 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:45.906422    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="9866a8d4-b855-461b-bcb6-2dbd0bcdef0b" containerName="revoke-keys"
Oct 30 01:02:45.906543 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:45.906533    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d6db577-4b91-4d50-8ac5-7b73b24aeb80" containerName="revoke-keys"
Oct 30 01:02:45.910404 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:45.910372    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efcbc3bd-5q876"
Oct 30 01:02:45.930693 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:45.930657    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-efcbc3bd-5q876"]
Oct 30 01:02:45.940371 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:45.940340    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5phsh\""
Oct 30 01:02:45.940548 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:45.940449    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:02:45.971462 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:45.971423    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:02:45.975535 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:45.975501    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:02:45.990769 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:45.990716    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7b6t2\" (UniqueName: \"kubernetes.io/projected/ce5d1af7-5443-41bb-9c6d-dbd173f6b2fd-kube-api-access-7b6t2\") pod \"maas-api-revoke-sub-efcbc3bd-5q876\" (UID: \"ce5d1af7-5443-41bb-9c6d-dbd173f6b2fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efcbc3bd-5q876"
Oct 30 01:02:45.990922 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:45.990779    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce5d1af7-5443-41bb-9c6d-dbd173f6b2fd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-efcbc3bd-5q876\" (UID: \"ce5d1af7-5443-41bb-9c6d-dbd173f6b2fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efcbc3bd-5q876"
Oct 30 01:02:45.990973 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:45.990924    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce5d1af7-5443-41bb-9c6d-dbd173f6b2fd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-efcbc3bd-5q876\" (UID: \"ce5d1af7-5443-41bb-9c6d-dbd173f6b2fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efcbc3bd-5q876"
Oct 30 01:02:46.092942 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:46.092900    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce5d1af7-5443-41bb-9c6d-dbd173f6b2fd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-efcbc3bd-5q876\" (UID: \"ce5d1af7-5443-41bb-9c6d-dbd173f6b2fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efcbc3bd-5q876"
Oct 30 01:02:46.093134 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:46.093070    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce5d1af7-5443-41bb-9c6d-dbd173f6b2fd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-efcbc3bd-5q876\" (UID: \"ce5d1af7-5443-41bb-9c6d-dbd173f6b2fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efcbc3bd-5q876"
Oct 30 01:02:46.093231 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:46.093218    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7b6t2\" (UniqueName: \"kubernetes.io/projected/ce5d1af7-5443-41bb-9c6d-dbd173f6b2fd-kube-api-access-7b6t2\") pod \"maas-api-revoke-sub-efcbc3bd-5q876\" (UID: \"ce5d1af7-5443-41bb-9c6d-dbd173f6b2fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efcbc3bd-5q876"
Oct 30 01:02:46.093990 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:46.093960    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce5d1af7-5443-41bb-9c6d-dbd173f6b2fd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-efcbc3bd-5q876\" (UID: \"ce5d1af7-5443-41bb-9c6d-dbd173f6b2fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efcbc3bd-5q876"
Oct 30 01:02:46.095622 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:46.095593    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce5d1af7-5443-41bb-9c6d-dbd173f6b2fd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-efcbc3bd-5q876\" (UID: \"ce5d1af7-5443-41bb-9c6d-dbd173f6b2fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efcbc3bd-5q876"
Oct 30 01:02:46.118623 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:46.118599    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7b6t2\" (UniqueName: \"kubernetes.io/projected/ce5d1af7-5443-41bb-9c6d-dbd173f6b2fd-kube-api-access-7b6t2\") pod \"maas-api-revoke-sub-efcbc3bd-5q876\" (UID: \"ce5d1af7-5443-41bb-9c6d-dbd173f6b2fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efcbc3bd-5q876"
Oct 30 01:02:46.235730 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:46.235696    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efcbc3bd-5q876"
Oct 30 01:02:46.607018 ip-10-0-128-7 kubenswrapper[5002]: W1030 01:02:46.606988    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce5d1af7_5443_41bb_9c6d_dbd173f6b2fd.slice/crio-17d825611ec974dbc55229935c9cc05c4fb9ef8b378f0730bcf2be767d62c5bb WatchSource:0}: Error finding container 17d825611ec974dbc55229935c9cc05c4fb9ef8b378f0730bcf2be767d62c5bb: Status 404 returned error can't find the container with id 17d825611ec974dbc55229935c9cc05c4fb9ef8b378f0730bcf2be767d62c5bb
Oct 30 01:02:46.609911 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:46.609840    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-efcbc3bd-5q876"]
Oct 30 01:02:47.225896 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:47.225862    5002 scope.go:122] "RemoveContainer" containerID="12d3c7094425048a7b6c8bb7e91e4cc973c74ace818b2f54f358df9e5236b73b"
Oct 30 01:02:47.238941 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:47.238785    5002 scope.go:122] "RemoveContainer" containerID="b5c7f311d5708ffb1d37bd7acf0d68522dca6d386b116400a59b37e10188ff1e"
Oct 30 01:02:47.560837 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:47.560720    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efcbc3bd-5q876" event={"ID":"ce5d1af7-5443-41bb-9c6d-dbd173f6b2fd","Type":"ContainerStarted","Data":"8c2cf7d8d68bb93abf721f230173bd6209acfca141428bfd55415834c4e1eacb"}
Oct 30 01:02:47.560837 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:47.560811    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efcbc3bd-5q876" event={"ID":"ce5d1af7-5443-41bb-9c6d-dbd173f6b2fd","Type":"ContainerStarted","Data":"17d825611ec974dbc55229935c9cc05c4fb9ef8b378f0730bcf2be767d62c5bb"}
Oct 30 01:02:47.604969 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:47.604911    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efcbc3bd-5q876" podStartSLOduration=2.604898888 podStartE2EDuration="2.604898888s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:02:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:02:47.60346015 +0000 UTC m=+1863.219893061" watchObservedRunningTime="2026-10-30 01:02:47.604898888 +0000 UTC m=+1863.221331785"
Oct 30 01:02:48.174403 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:48.171552    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-b3bdf9-openshift-default-755bccddf-cb9sz"]
Oct 30 01:02:48.174403 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:48.171786    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-b3bdf9-openshift-default-755bccddf-cb9sz" podUID="6dcd8f60-f8b4-476e-951c-21736a8d5255" containerName="istio-proxy" containerID="cri-o://d5313e0432da2b710ceb81de77388b9eb5f08959c5cc92670745085758d46302" gracePeriod=30
Oct 30 01:02:48.591625 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:48.591593    5002 generic.go:356] "Generic (PLEG): container finished" podID="ce5d1af7-5443-41bb-9c6d-dbd173f6b2fd" containerID="8c2cf7d8d68bb93abf721f230173bd6209acfca141428bfd55415834c4e1eacb" exitCode=0
Oct 30 01:02:48.592032 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:48.591658    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efcbc3bd-5q876" event={"ID":"ce5d1af7-5443-41bb-9c6d-dbd173f6b2fd","Type":"ContainerDied","Data":"8c2cf7d8d68bb93abf721f230173bd6209acfca141428bfd55415834c4e1eacb"}
Oct 30 01:02:50.249108 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:50.249074    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efcbc3bd-5q876"
Oct 30 01:02:50.347592 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:50.347540    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce5d1af7-5443-41bb-9c6d-dbd173f6b2fd-kube-api-access-7b6t2\" (UniqueName: \"kubernetes.io/projected/ce5d1af7-5443-41bb-9c6d-dbd173f6b2fd-kube-api-access-7b6t2\") pod \"ce5d1af7-5443-41bb-9c6d-dbd173f6b2fd\" (UID: \"ce5d1af7-5443-41bb-9c6d-dbd173f6b2fd\") "
Oct 30 01:02:50.347776 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:50.347666    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ce5d1af7-5443-41bb-9c6d-dbd173f6b2fd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce5d1af7-5443-41bb-9c6d-dbd173f6b2fd-maas-api-xks-ca\") pod \"ce5d1af7-5443-41bb-9c6d-dbd173f6b2fd\" (UID: \"ce5d1af7-5443-41bb-9c6d-dbd173f6b2fd\") "
Oct 30 01:02:50.347776 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:50.347719    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce5d1af7-5443-41bb-9c6d-dbd173f6b2fd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce5d1af7-5443-41bb-9c6d-dbd173f6b2fd-maas-api-openshift-ca\") pod \"ce5d1af7-5443-41bb-9c6d-dbd173f6b2fd\" (UID: \"ce5d1af7-5443-41bb-9c6d-dbd173f6b2fd\") "
Oct 30 01:02:50.348621 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:50.348585    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce5d1af7-5443-41bb-9c6d-dbd173f6b2fd-maas-api-openshift-ca" pod "ce5d1af7-5443-41bb-9c6d-dbd173f6b2fd" (UID: "ce5d1af7-5443-41bb-9c6d-dbd173f6b2fd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:02:50.350619 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:50.350595    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce5d1af7-5443-41bb-9c6d-dbd173f6b2fd-kube-api-access-7b6t2" pod "ce5d1af7-5443-41bb-9c6d-dbd173f6b2fd" (UID: "ce5d1af7-5443-41bb-9c6d-dbd173f6b2fd"). InnerVolumeSpecName "kube-api-access-7b6t2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:02:50.378906 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:50.378858    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce5d1af7-5443-41bb-9c6d-dbd173f6b2fd-maas-api-xks-ca" pod "ce5d1af7-5443-41bb-9c6d-dbd173f6b2fd" (UID: "ce5d1af7-5443-41bb-9c6d-dbd173f6b2fd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:02:50.450464 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:50.450427    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7b6t2\" (UniqueName: \"kubernetes.io/projected/ce5d1af7-5443-41bb-9c6d-dbd173f6b2fd-kube-api-access-7b6t2\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:02:50.450464 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:50.450463    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce5d1af7-5443-41bb-9c6d-dbd173f6b2fd-maas-api-xks-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:02:50.450464 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:50.450476    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce5d1af7-5443-41bb-9c6d-dbd173f6b2fd-maas-api-openshift-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:02:50.610313 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:50.610285    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efcbc3bd-5q876"
Oct 30 01:02:50.610539 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:50.610324    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efcbc3bd-5q876" event={"ID":"ce5d1af7-5443-41bb-9c6d-dbd173f6b2fd","Type":"ContainerDied","Data":"17d825611ec974dbc55229935c9cc05c4fb9ef8b378f0730bcf2be767d62c5bb"}
Oct 30 01:02:50.610539 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:50.610408    5002 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="17d825611ec974dbc55229935c9cc05c4fb9ef8b378f0730bcf2be767d62c5bb"
Oct 30 01:02:53.530323 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:53.530294    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b3bdf9-openshift-default-755bccddf-cb9sz"
Oct 30 01:02:53.617618 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:53.617583    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6dcd8f60-f8b4-476e-951c-21736a8d5255-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6dcd8f60-f8b4-476e-951c-21736a8d5255-workload-certs\") pod \"6dcd8f60-f8b4-476e-951c-21736a8d5255\" (UID: \"6dcd8f60-f8b4-476e-951c-21736a8d5255\") "
Oct 30 01:02:53.617777 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:53.617749    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6dcd8f60-f8b4-476e-951c-21736a8d5255-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6dcd8f60-f8b4-476e-951c-21736a8d5255-istiod-ca-cert\") pod \"6dcd8f60-f8b4-476e-951c-21736a8d5255\" (UID: \"6dcd8f60-f8b4-476e-951c-21736a8d5255\") "
Oct 30 01:02:53.617829 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:53.617782    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6dcd8f60-f8b4-476e-951c-21736a8d5255-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6dcd8f60-f8b4-476e-951c-21736a8d5255-istio-podinfo\") pod \"6dcd8f60-f8b4-476e-951c-21736a8d5255\" (UID: \"6dcd8f60-f8b4-476e-951c-21736a8d5255\") "
Oct 30 01:02:53.617904 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:53.617886    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6dcd8f60-f8b4-476e-951c-21736a8d5255-workload-certs" pod "6dcd8f60-f8b4-476e-951c-21736a8d5255" (UID: "6dcd8f60-f8b4-476e-951c-21736a8d5255"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:02:53.617945 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:53.617902    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6dcd8f60-f8b4-476e-951c-21736a8d5255-kube-api-access-75g24\" (UniqueName: \"kubernetes.io/projected/6dcd8f60-f8b4-476e-951c-21736a8d5255-kube-api-access-75g24\") pod \"6dcd8f60-f8b4-476e-951c-21736a8d5255\" (UID: \"6dcd8f60-f8b4-476e-951c-21736a8d5255\") "
Oct 30 01:02:53.617945 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:53.617932    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6dcd8f60-f8b4-476e-951c-21736a8d5255-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dcd8f60-f8b4-476e-951c-21736a8d5255-credential-socket\") pod \"6dcd8f60-f8b4-476e-951c-21736a8d5255\" (UID: \"6dcd8f60-f8b4-476e-951c-21736a8d5255\") "
Oct 30 01:02:53.618053 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:53.618036    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6dcd8f60-f8b4-476e-951c-21736a8d5255-istio-token\" (UniqueName: \"kubernetes.io/projected/6dcd8f60-f8b4-476e-951c-21736a8d5255-istio-token\") pod \"6dcd8f60-f8b4-476e-951c-21736a8d5255\" (UID: \"6dcd8f60-f8b4-476e-951c-21736a8d5255\") "
Oct 30 01:02:53.618174 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:53.618156    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6dcd8f60-f8b4-476e-951c-21736a8d5255-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dcd8f60-f8b4-476e-951c-21736a8d5255-workload-socket\") pod \"6dcd8f60-f8b4-476e-951c-21736a8d5255\" (UID: \"6dcd8f60-f8b4-476e-951c-21736a8d5255\") "
Oct 30 01:02:53.618241 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:53.618198    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6dcd8f60-f8b4-476e-951c-21736a8d5255-istiod-ca-cert" pod "6dcd8f60-f8b4-476e-951c-21736a8d5255" (UID: "6dcd8f60-f8b4-476e-951c-21736a8d5255"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:02:53.618241 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:53.618210    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6dcd8f60-f8b4-476e-951c-21736a8d5255-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6dcd8f60-f8b4-476e-951c-21736a8d5255-istio-data\") pod \"6dcd8f60-f8b4-476e-951c-21736a8d5255\" (UID: \"6dcd8f60-f8b4-476e-951c-21736a8d5255\") "
Oct 30 01:02:53.618350 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:53.618275    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6dcd8f60-f8b4-476e-951c-21736a8d5255-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6dcd8f60-f8b4-476e-951c-21736a8d5255-istio-envoy\") pod \"6dcd8f60-f8b4-476e-951c-21736a8d5255\" (UID: \"6dcd8f60-f8b4-476e-951c-21736a8d5255\") "
Oct 30 01:02:53.618460 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:53.618429    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6dcd8f60-f8b4-476e-951c-21736a8d5255-istio-data" pod "6dcd8f60-f8b4-476e-951c-21736a8d5255" (UID: "6dcd8f60-f8b4-476e-951c-21736a8d5255"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:02:53.618827 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:53.618798    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6dcd8f60-f8b4-476e-951c-21736a8d5255-credential-socket" pod "6dcd8f60-f8b4-476e-951c-21736a8d5255" (UID: "6dcd8f60-f8b4-476e-951c-21736a8d5255"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:02:53.619301 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:53.619273    5002 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6dcd8f60-f8b4-476e-951c-21736a8d5255-workload-certs\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:02:53.619473 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:53.619305    5002 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6dcd8f60-f8b4-476e-951c-21736a8d5255-istiod-ca-cert\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:02:53.619473 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:53.619319    5002 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dcd8f60-f8b4-476e-951c-21736a8d5255-credential-socket\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:02:53.619473 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:53.619335    5002 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6dcd8f60-f8b4-476e-951c-21736a8d5255-istio-data\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:02:53.620037 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:53.620010    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6dcd8f60-f8b4-476e-951c-21736a8d5255-workload-socket" pod "6dcd8f60-f8b4-476e-951c-21736a8d5255" (UID: "6dcd8f60-f8b4-476e-951c-21736a8d5255"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:02:53.620862 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:53.620838    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6dcd8f60-f8b4-476e-951c-21736a8d5255-istio-token" pod "6dcd8f60-f8b4-476e-951c-21736a8d5255" (UID: "6dcd8f60-f8b4-476e-951c-21736a8d5255"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:02:53.620862 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:53.620850    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6dcd8f60-f8b4-476e-951c-21736a8d5255-istio-podinfo" pod "6dcd8f60-f8b4-476e-951c-21736a8d5255" (UID: "6dcd8f60-f8b4-476e-951c-21736a8d5255"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:02:53.621619 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:53.621597    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6dcd8f60-f8b4-476e-951c-21736a8d5255-istio-envoy" pod "6dcd8f60-f8b4-476e-951c-21736a8d5255" (UID: "6dcd8f60-f8b4-476e-951c-21736a8d5255"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:02:53.621698 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:53.621664    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6dcd8f60-f8b4-476e-951c-21736a8d5255-kube-api-access-75g24" pod "6dcd8f60-f8b4-476e-951c-21736a8d5255" (UID: "6dcd8f60-f8b4-476e-951c-21736a8d5255"). InnerVolumeSpecName "kube-api-access-75g24". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:02:53.635529 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:53.635500    5002 generic.go:356] "Generic (PLEG): container finished" podID="6dcd8f60-f8b4-476e-951c-21736a8d5255" containerID="d5313e0432da2b710ceb81de77388b9eb5f08959c5cc92670745085758d46302" exitCode=0
Oct 30 01:02:53.635627 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:53.635554    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b3bdf9-openshift-default-755bccddf-cb9sz"
Oct 30 01:02:53.635627 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:53.635582    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b3bdf9-openshift-default-755bccddf-cb9sz" event={"ID":"6dcd8f60-f8b4-476e-951c-21736a8d5255","Type":"ContainerDied","Data":"d5313e0432da2b710ceb81de77388b9eb5f08959c5cc92670745085758d46302"}
Oct 30 01:02:53.635751 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:53.635644    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b3bdf9-openshift-default-755bccddf-cb9sz" event={"ID":"6dcd8f60-f8b4-476e-951c-21736a8d5255","Type":"ContainerDied","Data":"3101ab809de26e58178acad2959242a0ed2826d42552d27a40d6b0c1f18d0715"}
Oct 30 01:02:53.635751 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:53.635677    5002 scope.go:122] "RemoveContainer" containerID="d5313e0432da2b710ceb81de77388b9eb5f08959c5cc92670745085758d46302"
Oct 30 01:02:53.684567 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:53.668583    5002 scope.go:122] "RemoveContainer" containerID="d5313e0432da2b710ceb81de77388b9eb5f08959c5cc92670745085758d46302"
Oct 30 01:02:53.684567 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:02:53.683862    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5313e0432da2b710ceb81de77388b9eb5f08959c5cc92670745085758d46302\": container with ID starting with d5313e0432da2b710ceb81de77388b9eb5f08959c5cc92670745085758d46302 not found: ID does not exist" containerID="d5313e0432da2b710ceb81de77388b9eb5f08959c5cc92670745085758d46302"
Oct 30 01:02:53.698405 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:53.683906    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5313e0432da2b710ceb81de77388b9eb5f08959c5cc92670745085758d46302"} err="failed to get container status \"d5313e0432da2b710ceb81de77388b9eb5f08959c5cc92670745085758d46302\": rpc error: code = NotFound desc = could not find container \"d5313e0432da2b710ceb81de77388b9eb5f08959c5cc92670745085758d46302\": container with ID starting with d5313e0432da2b710ceb81de77388b9eb5f08959c5cc92670745085758d46302 not found: ID does not exist"
Oct 30 01:02:53.728805 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:53.726614    5002 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6dcd8f60-f8b4-476e-951c-21736a8d5255-istio-podinfo\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:02:53.728805 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:53.726645    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-75g24\" (UniqueName: \"kubernetes.io/projected/6dcd8f60-f8b4-476e-951c-21736a8d5255-kube-api-access-75g24\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:02:53.728805 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:53.726660    5002 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6dcd8f60-f8b4-476e-951c-21736a8d5255-istio-token\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:02:53.728805 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:53.726675    5002 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dcd8f60-f8b4-476e-951c-21736a8d5255-workload-socket\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:02:53.728805 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:53.726688    5002 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6dcd8f60-f8b4-476e-951c-21736a8d5255-istio-envoy\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:02:53.728805 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:53.727078    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-b3bdf9-openshift-default-755bccddf-cb9sz"]
Oct 30 01:02:53.741251 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:53.735197    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-b3bdf9-openshift-default-755bccddf-cb9sz"]
Oct 30 01:02:54.500455 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:02:54.500414    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6dcd8f60-f8b4-476e-951c-21736a8d5255" path="/var/lib/kubelet/pods/6dcd8f60-f8b4-476e-951c-21736a8d5255/volumes"
Oct 30 01:03:00.239597 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:00.239550    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-481ab7/test-model-481ab7-kserve-59c6875d8f-wz4bj" podUID="75c0d4c3-f369-4930-839f-25cfad1f3f5e" containerName="main" containerID="cri-o://da9ce5bebe1b498d835085d541d8dce734cead76e6329fcdd00e4d5170411498" gracePeriod=45
Oct 30 01:03:00.557500 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:00.557471    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-481ab7/test-model-481ab7-kserve-59c6875d8f-wz4bj"
Oct 30 01:03:00.688357 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:00.688316    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75c0d4c3-f369-4930-839f-25cfad1f3f5e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75c0d4c3-f369-4930-839f-25cfad1f3f5e-tmp-dir\") pod \"75c0d4c3-f369-4930-839f-25cfad1f3f5e\" (UID: \"75c0d4c3-f369-4930-839f-25cfad1f3f5e\") "
Oct 30 01:03:00.688562 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:00.688438    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75c0d4c3-f369-4930-839f-25cfad1f3f5e-home\" (UniqueName: \"kubernetes.io/empty-dir/75c0d4c3-f369-4930-839f-25cfad1f3f5e-home\") pod \"75c0d4c3-f369-4930-839f-25cfad1f3f5e\" (UID: \"75c0d4c3-f369-4930-839f-25cfad1f3f5e\") "
Oct 30 01:03:00.688562 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:00.688504    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75c0d4c3-f369-4930-839f-25cfad1f3f5e-kube-api-access-cxt27\" (UniqueName: \"kubernetes.io/projected/75c0d4c3-f369-4930-839f-25cfad1f3f5e-kube-api-access-cxt27\") pod \"75c0d4c3-f369-4930-839f-25cfad1f3f5e\" (UID: \"75c0d4c3-f369-4930-839f-25cfad1f3f5e\") "
Oct 30 01:03:00.688562 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:00.688552    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75c0d4c3-f369-4930-839f-25cfad1f3f5e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/75c0d4c3-f369-4930-839f-25cfad1f3f5e-dshm\") pod \"75c0d4c3-f369-4930-839f-25cfad1f3f5e\" (UID: \"75c0d4c3-f369-4930-839f-25cfad1f3f5e\") "
Oct 30 01:03:00.688754 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:00.688626    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/75c0d4c3-f369-4930-839f-25cfad1f3f5e-tls-certs\" (UniqueName: \"kubernetes.io/secret/75c0d4c3-f369-4930-839f-25cfad1f3f5e-tls-certs\") pod \"75c0d4c3-f369-4930-839f-25cfad1f3f5e\" (UID: \"75c0d4c3-f369-4930-839f-25cfad1f3f5e\") "
Oct 30 01:03:00.688754 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:00.688687    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75c0d4c3-f369-4930-839f-25cfad1f3f5e-tmp-dir" pod "75c0d4c3-f369-4930-839f-25cfad1f3f5e" (UID: "75c0d4c3-f369-4930-839f-25cfad1f3f5e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:03:00.688903 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:00.688876    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75c0d4c3-f369-4930-839f-25cfad1f3f5e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75c0d4c3-f369-4930-839f-25cfad1f3f5e-model-cache\") pod \"75c0d4c3-f369-4930-839f-25cfad1f3f5e\" (UID: \"75c0d4c3-f369-4930-839f-25cfad1f3f5e\") "
Oct 30 01:03:00.689062 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:00.689032    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75c0d4c3-f369-4930-839f-25cfad1f3f5e-home" pod "75c0d4c3-f369-4930-839f-25cfad1f3f5e" (UID: "75c0d4c3-f369-4930-839f-25cfad1f3f5e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:03:00.689810 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:00.689150    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75c0d4c3-f369-4930-839f-25cfad1f3f5e-model-cache" pod "75c0d4c3-f369-4930-839f-25cfad1f3f5e" (UID: "75c0d4c3-f369-4930-839f-25cfad1f3f5e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:03:00.689810 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:00.689642    5002 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75c0d4c3-f369-4930-839f-25cfad1f3f5e-model-cache\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:03:00.689810 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:00.689688    5002 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75c0d4c3-f369-4930-839f-25cfad1f3f5e-tmp-dir\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:03:00.689810 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:00.689701    5002 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75c0d4c3-f369-4930-839f-25cfad1f3f5e-home\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:03:00.691562 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:00.691531    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75c0d4c3-f369-4930-839f-25cfad1f3f5e-dshm" pod "75c0d4c3-f369-4930-839f-25cfad1f3f5e" (UID: "75c0d4c3-f369-4930-839f-25cfad1f3f5e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:03:00.691675 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:00.691621    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75c0d4c3-f369-4930-839f-25cfad1f3f5e-tls-certs" pod "75c0d4c3-f369-4930-839f-25cfad1f3f5e" (UID: "75c0d4c3-f369-4930-839f-25cfad1f3f5e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:03:00.691771 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:00.691748    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75c0d4c3-f369-4930-839f-25cfad1f3f5e-kube-api-access-cxt27" pod "75c0d4c3-f369-4930-839f-25cfad1f3f5e" (UID: "75c0d4c3-f369-4930-839f-25cfad1f3f5e"). InnerVolumeSpecName "kube-api-access-cxt27". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:03:00.706071 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:00.706037    5002 generic.go:356] "Generic (PLEG): container finished" podID="75c0d4c3-f369-4930-839f-25cfad1f3f5e" containerID="da9ce5bebe1b498d835085d541d8dce734cead76e6329fcdd00e4d5170411498" exitCode=0
Oct 30 01:03:00.706224 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:00.706102    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-481ab7/test-model-481ab7-kserve-59c6875d8f-wz4bj"
Oct 30 01:03:00.706224 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:00.706124    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-481ab7/test-model-481ab7-kserve-59c6875d8f-wz4bj" event={"ID":"75c0d4c3-f369-4930-839f-25cfad1f3f5e","Type":"ContainerDied","Data":"da9ce5bebe1b498d835085d541d8dce734cead76e6329fcdd00e4d5170411498"}
Oct 30 01:03:00.706224 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:00.706187    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-481ab7/test-model-481ab7-kserve-59c6875d8f-wz4bj" event={"ID":"75c0d4c3-f369-4930-839f-25cfad1f3f5e","Type":"ContainerDied","Data":"d411ac0b01763d0cd7ce377724fa68592a340c0c6cc00348ae1a97d07eb9d55a"}
Oct 30 01:03:00.706224 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:00.706213    5002 scope.go:122] "RemoveContainer" containerID="da9ce5bebe1b498d835085d541d8dce734cead76e6329fcdd00e4d5170411498"
Oct 30 01:03:00.728509 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:00.728485    5002 scope.go:122] "RemoveContainer" containerID="da9ce5bebe1b498d835085d541d8dce734cead76e6329fcdd00e4d5170411498"
Oct 30 01:03:00.728843 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:03:00.728818    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da9ce5bebe1b498d835085d541d8dce734cead76e6329fcdd00e4d5170411498\": container with ID starting with da9ce5bebe1b498d835085d541d8dce734cead76e6329fcdd00e4d5170411498 not found: ID does not exist" containerID="da9ce5bebe1b498d835085d541d8dce734cead76e6329fcdd00e4d5170411498"
Oct 30 01:03:00.728894 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:00.728854    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da9ce5bebe1b498d835085d541d8dce734cead76e6329fcdd00e4d5170411498"} err="failed to get container status \"da9ce5bebe1b498d835085d541d8dce734cead76e6329fcdd00e4d5170411498\": rpc error: code = NotFound desc = could not find container \"da9ce5bebe1b498d835085d541d8dce734cead76e6329fcdd00e4d5170411498\": container with ID starting with da9ce5bebe1b498d835085d541d8dce734cead76e6329fcdd00e4d5170411498 not found: ID does not exist"
Oct 30 01:03:00.765214 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:00.765175    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-481ab7/test-model-481ab7-kserve-59c6875d8f-wz4bj"]
Oct 30 01:03:00.768838 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:00.768807    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-481ab7/test-model-481ab7-kserve-59c6875d8f-wz4bj"]
Oct 30 01:03:00.791849 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:00.791652    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cxt27\" (UniqueName: \"kubernetes.io/projected/75c0d4c3-f369-4930-839f-25cfad1f3f5e-kube-api-access-cxt27\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:03:00.791849 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:00.791688    5002 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75c0d4c3-f369-4930-839f-25cfad1f3f5e-dshm\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:03:00.791849 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:00.791705    5002 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75c0d4c3-f369-4930-839f-25cfad1f3f5e-tls-certs\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:03:02.498484 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:02.498438    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75c0d4c3-f369-4930-839f-25cfad1f3f5e" path="/var/lib/kubelet/pods/75c0d4c3-f369-4930-839f-25cfad1f3f5e/volumes"
Oct 30 01:03:12.866298 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:03:12.866250    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbjdq" containerName="extractor"
Oct 30 01:03:15.093879 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:15.093839    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-8dbe73/test-model-8dbe73-kserve-78dbd9d5cc-5zdxp"]
Oct 30 01:03:15.539147 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:15.539088    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-03d892/test-model-03d892-kserve-7f8c7cb699-kp2mb"]
Oct 30 01:03:15.705272 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:15.705192    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bafb00a1-hw4v7"]
Oct 30 01:03:15.706589 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:15.706556    5002 cpu_manager.go:414] "containerMap: removing container" podUID="75c0d4c3-f369-4930-839f-25cfad1f3f5e" containerName="main"
Oct 30 01:03:15.706772 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:15.706758    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75c0d4c3-f369-4930-839f-25cfad1f3f5e" containerName="main"
Oct 30 01:03:15.706998 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:15.706984    5002 cpu_manager.go:414] "containerMap: removing container" podUID="6dcd8f60-f8b4-476e-951c-21736a8d5255" containerName="istio-proxy"
Oct 30 01:03:15.707091 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:15.707081    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6dcd8f60-f8b4-476e-951c-21736a8d5255" containerName="istio-proxy"
Oct 30 01:03:15.707184 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:15.707175    5002 cpu_manager.go:414] "containerMap: removing container" podUID="ce5d1af7-5443-41bb-9c6d-dbd173f6b2fd" containerName="revoke-keys"
Oct 30 01:03:15.707255 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:15.707246    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce5d1af7-5443-41bb-9c6d-dbd173f6b2fd" containerName="revoke-keys"
Oct 30 01:03:15.707514 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:15.707497    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="75c0d4c3-f369-4930-839f-25cfad1f3f5e" containerName="main"
Oct 30 01:03:15.707696 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:15.707684    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="6dcd8f60-f8b4-476e-951c-21736a8d5255" containerName="istio-proxy"
Oct 30 01:03:15.707889 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:15.707876    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce5d1af7-5443-41bb-9c6d-dbd173f6b2fd" containerName="revoke-keys"
Oct 30 01:03:15.740407 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:15.734899    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bafb00a1-hw4v7"]
Oct 30 01:03:15.740407 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:15.735055    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bafb00a1-hw4v7"
Oct 30 01:03:15.795501 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:15.761051    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5phsh\""
Oct 30 01:03:15.795501 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:15.765487    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:03:15.795501 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:15.765576    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:03:15.795501 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:15.765630    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:03:15.838473 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:15.838439    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-481ab7-m4gqt"]
Oct 30 01:03:15.883937 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:15.883895    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-481ab7-m4gqt"]
Oct 30 01:03:15.885089 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:15.885060    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-481ab7-m4gqt"
Oct 30 01:03:15.916622 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:15.916438    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8cef991-175d-4b30-8d0b-33f05e92a992-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bafb00a1-hw4v7\" (UID: \"a8cef991-175d-4b30-8d0b-33f05e92a992\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bafb00a1-hw4v7"
Oct 30 01:03:15.916622 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:15.916575    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8cef991-175d-4b30-8d0b-33f05e92a992-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bafb00a1-hw4v7\" (UID: \"a8cef991-175d-4b30-8d0b-33f05e92a992\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bafb00a1-hw4v7"
Oct 30 01:03:15.916937 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:15.916760    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kg96w\" (UniqueName: \"kubernetes.io/projected/a8cef991-175d-4b30-8d0b-33f05e92a992-kube-api-access-kg96w\") pod \"maas-api-revoke-sub-bafb00a1-hw4v7\" (UID: \"a8cef991-175d-4b30-8d0b-33f05e92a992\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bafb00a1-hw4v7"
Oct 30 01:03:16.018122 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:16.018084    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfxbk\" (UniqueName: \"kubernetes.io/projected/0f9d2df7-6b86-4f0a-8f75-9a3d54f09aed-kube-api-access-rfxbk\") pod \"maas-api-revoke-keys-e2e-inf-a-481ab7-m4gqt\" (UID: \"0f9d2df7-6b86-4f0a-8f75-9a3d54f09aed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-481ab7-m4gqt"
Oct 30 01:03:16.018331 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:16.018152    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8cef991-175d-4b30-8d0b-33f05e92a992-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bafb00a1-hw4v7\" (UID: \"a8cef991-175d-4b30-8d0b-33f05e92a992\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bafb00a1-hw4v7"
Oct 30 01:03:16.018331 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:16.018309    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8cef991-175d-4b30-8d0b-33f05e92a992-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bafb00a1-hw4v7\" (UID: \"a8cef991-175d-4b30-8d0b-33f05e92a992\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bafb00a1-hw4v7"
Oct 30 01:03:16.018530 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:16.018356    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f9d2df7-6b86-4f0a-8f75-9a3d54f09aed-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-481ab7-m4gqt\" (UID: \"0f9d2df7-6b86-4f0a-8f75-9a3d54f09aed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-481ab7-m4gqt"
Oct 30 01:03:16.018604 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:16.018575    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f9d2df7-6b86-4f0a-8f75-9a3d54f09aed-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-481ab7-m4gqt\" (UID: \"0f9d2df7-6b86-4f0a-8f75-9a3d54f09aed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-481ab7-m4gqt"
Oct 30 01:03:16.018744 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:16.018725    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kg96w\" (UniqueName: \"kubernetes.io/projected/a8cef991-175d-4b30-8d0b-33f05e92a992-kube-api-access-kg96w\") pod \"maas-api-revoke-sub-bafb00a1-hw4v7\" (UID: \"a8cef991-175d-4b30-8d0b-33f05e92a992\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bafb00a1-hw4v7"
Oct 30 01:03:16.019008 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:16.018979    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8cef991-175d-4b30-8d0b-33f05e92a992-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bafb00a1-hw4v7\" (UID: \"a8cef991-175d-4b30-8d0b-33f05e92a992\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bafb00a1-hw4v7"
Oct 30 01:03:16.021138 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:16.021107    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8cef991-175d-4b30-8d0b-33f05e92a992-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bafb00a1-hw4v7\" (UID: \"a8cef991-175d-4b30-8d0b-33f05e92a992\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bafb00a1-hw4v7"
Oct 30 01:03:16.043701 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:16.043670    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kg96w\" (UniqueName: \"kubernetes.io/projected/a8cef991-175d-4b30-8d0b-33f05e92a992-kube-api-access-kg96w\") pod \"maas-api-revoke-sub-bafb00a1-hw4v7\" (UID: \"a8cef991-175d-4b30-8d0b-33f05e92a992\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bafb00a1-hw4v7"
Oct 30 01:03:16.119942 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:16.119855    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfxbk\" (UniqueName: \"kubernetes.io/projected/0f9d2df7-6b86-4f0a-8f75-9a3d54f09aed-kube-api-access-rfxbk\") pod \"maas-api-revoke-keys-e2e-inf-a-481ab7-m4gqt\" (UID: \"0f9d2df7-6b86-4f0a-8f75-9a3d54f09aed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-481ab7-m4gqt"
Oct 30 01:03:16.120414 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:16.120012    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f9d2df7-6b86-4f0a-8f75-9a3d54f09aed-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-481ab7-m4gqt\" (UID: \"0f9d2df7-6b86-4f0a-8f75-9a3d54f09aed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-481ab7-m4gqt"
Oct 30 01:03:16.120414 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:16.120125    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f9d2df7-6b86-4f0a-8f75-9a3d54f09aed-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-481ab7-m4gqt\" (UID: \"0f9d2df7-6b86-4f0a-8f75-9a3d54f09aed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-481ab7-m4gqt"
Oct 30 01:03:16.120980 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:16.120955    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f9d2df7-6b86-4f0a-8f75-9a3d54f09aed-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-481ab7-m4gqt\" (UID: \"0f9d2df7-6b86-4f0a-8f75-9a3d54f09aed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-481ab7-m4gqt"
Oct 30 01:03:16.122612 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:16.122585    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f9d2df7-6b86-4f0a-8f75-9a3d54f09aed-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-481ab7-m4gqt\" (UID: \"0f9d2df7-6b86-4f0a-8f75-9a3d54f09aed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-481ab7-m4gqt"
Oct 30 01:03:16.143514 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:16.143486    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bafb00a1-hw4v7"
Oct 30 01:03:16.145277 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:16.145245    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfxbk\" (UniqueName: \"kubernetes.io/projected/0f9d2df7-6b86-4f0a-8f75-9a3d54f09aed-kube-api-access-rfxbk\") pod \"maas-api-revoke-keys-e2e-inf-a-481ab7-m4gqt\" (UID: \"0f9d2df7-6b86-4f0a-8f75-9a3d54f09aed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-481ab7-m4gqt"
Oct 30 01:03:16.230310 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:16.230272    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-481ab7-m4gqt"
Oct 30 01:03:16.460413 ip-10-0-128-7 kubenswrapper[5002]: W1030 01:03:16.460346    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8cef991_175d_4b30_8d0b_33f05e92a992.slice/crio-1e4744bc6a5c98181c8127ca72d3c27e2e7ae8bab9577f3dc09edf253c4ad0c5 WatchSource:0}: Error finding container 1e4744bc6a5c98181c8127ca72d3c27e2e7ae8bab9577f3dc09edf253c4ad0c5: Status 404 returned error can't find the container with id 1e4744bc6a5c98181c8127ca72d3c27e2e7ae8bab9577f3dc09edf253c4ad0c5
Oct 30 01:03:16.461090 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:16.461059    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bafb00a1-hw4v7"]
Oct 30 01:03:16.567920 ip-10-0-128-7 kubenswrapper[5002]: W1030 01:03:16.567887    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f9d2df7_6b86_4f0a_8f75_9a3d54f09aed.slice/crio-697eb6e1ee266bf36641ed38cd63a3a627e130fccca91c82fbbed495422ba6a3 WatchSource:0}: Error finding container 697eb6e1ee266bf36641ed38cd63a3a627e130fccca91c82fbbed495422ba6a3: Status 404 returned error can't find the container with id 697eb6e1ee266bf36641ed38cd63a3a627e130fccca91c82fbbed495422ba6a3
Oct 30 01:03:16.568732 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:16.568704    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-481ab7-m4gqt"]
Oct 30 01:03:16.880477 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:16.880437    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-481ab7-m4gqt" event={"ID":"0f9d2df7-6b86-4f0a-8f75-9a3d54f09aed","Type":"ContainerStarted","Data":"697eb6e1ee266bf36641ed38cd63a3a627e130fccca91c82fbbed495422ba6a3"}
Oct 30 01:03:16.882504 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:16.882469    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bafb00a1-hw4v7" event={"ID":"a8cef991-175d-4b30-8d0b-33f05e92a992","Type":"ContainerStarted","Data":"42100265a44327bd748e2466a4c867c3d416211077d70d4bc0b7291d3e3cc1d2"}
Oct 30 01:03:16.882659 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:16.882520    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bafb00a1-hw4v7" event={"ID":"a8cef991-175d-4b30-8d0b-33f05e92a992","Type":"ContainerStarted","Data":"1e4744bc6a5c98181c8127ca72d3c27e2e7ae8bab9577f3dc09edf253c4ad0c5"}
Oct 30 01:03:16.930470 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:16.930369    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bafb00a1-hw4v7" podStartSLOduration=1.930348886 podStartE2EDuration="1.930348886s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:03:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:03:16.927048613 +0000 UTC m=+1892.543481493" watchObservedRunningTime="2026-10-30 01:03:16.930348886 +0000 UTC m=+1892.546781766"
Oct 30 01:03:17.892547 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:17.892509    5002 generic.go:356] "Generic (PLEG): container finished" podID="0f9d2df7-6b86-4f0a-8f75-9a3d54f09aed" containerID="4013367e10d219e9261f7d3d6d66784279bf130cb51cc7286e141144d06b9854" exitCode=0
Oct 30 01:03:17.893008 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:17.892585    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-481ab7-m4gqt" event={"ID":"0f9d2df7-6b86-4f0a-8f75-9a3d54f09aed","Type":"ContainerDied","Data":"4013367e10d219e9261f7d3d6d66784279bf130cb51cc7286e141144d06b9854"}
Oct 30 01:03:17.894423 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:17.894399    5002 generic.go:356] "Generic (PLEG): container finished" podID="a8cef991-175d-4b30-8d0b-33f05e92a992" containerID="42100265a44327bd748e2466a4c867c3d416211077d70d4bc0b7291d3e3cc1d2" exitCode=0
Oct 30 01:03:17.894561 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:17.894459    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bafb00a1-hw4v7" event={"ID":"a8cef991-175d-4b30-8d0b-33f05e92a992","Type":"ContainerDied","Data":"42100265a44327bd748e2466a4c867c3d416211077d70d4bc0b7291d3e3cc1d2"}
Oct 30 01:03:19.128187 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:19.128154    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bafb00a1-hw4v7"
Oct 30 01:03:19.131936 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:19.131910    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-481ab7-m4gqt"
Oct 30 01:03:19.184472 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:19.184442    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8cef991-175d-4b30-8d0b-33f05e92a992-kube-api-access-kg96w\" (UniqueName: \"kubernetes.io/projected/a8cef991-175d-4b30-8d0b-33f05e92a992-kube-api-access-kg96w\") pod \"a8cef991-175d-4b30-8d0b-33f05e92a992\" (UID: \"a8cef991-175d-4b30-8d0b-33f05e92a992\") "
Oct 30 01:03:19.184666 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:19.184501    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f9d2df7-6b86-4f0a-8f75-9a3d54f09aed-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f9d2df7-6b86-4f0a-8f75-9a3d54f09aed-maas-api-xks-ca\") pod \"0f9d2df7-6b86-4f0a-8f75-9a3d54f09aed\" (UID: \"0f9d2df7-6b86-4f0a-8f75-9a3d54f09aed\") "
Oct 30 01:03:19.184666 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:19.184607    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8cef991-175d-4b30-8d0b-33f05e92a992-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8cef991-175d-4b30-8d0b-33f05e92a992-maas-api-openshift-ca\") pod \"a8cef991-175d-4b30-8d0b-33f05e92a992\" (UID: \"a8cef991-175d-4b30-8d0b-33f05e92a992\") "
Oct 30 01:03:19.184786 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:19.184704    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f9d2df7-6b86-4f0a-8f75-9a3d54f09aed-kube-api-access-rfxbk\" (UniqueName: \"kubernetes.io/projected/0f9d2df7-6b86-4f0a-8f75-9a3d54f09aed-kube-api-access-rfxbk\") pod \"0f9d2df7-6b86-4f0a-8f75-9a3d54f09aed\" (UID: \"0f9d2df7-6b86-4f0a-8f75-9a3d54f09aed\") "
Oct 30 01:03:19.184919 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:19.184898    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a8cef991-175d-4b30-8d0b-33f05e92a992-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8cef991-175d-4b30-8d0b-33f05e92a992-maas-api-xks-ca\") pod \"a8cef991-175d-4b30-8d0b-33f05e92a992\" (UID: \"a8cef991-175d-4b30-8d0b-33f05e92a992\") "
Oct 30 01:03:19.185017 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:19.184938    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f9d2df7-6b86-4f0a-8f75-9a3d54f09aed-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f9d2df7-6b86-4f0a-8f75-9a3d54f09aed-maas-api-openshift-ca\") pod \"0f9d2df7-6b86-4f0a-8f75-9a3d54f09aed\" (UID: \"0f9d2df7-6b86-4f0a-8f75-9a3d54f09aed\") "
Oct 30 01:03:19.185978 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:19.185820    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8cef991-175d-4b30-8d0b-33f05e92a992-maas-api-openshift-ca" pod "a8cef991-175d-4b30-8d0b-33f05e92a992" (UID: "a8cef991-175d-4b30-8d0b-33f05e92a992"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:03:19.186182 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:19.186001    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f9d2df7-6b86-4f0a-8f75-9a3d54f09aed-maas-api-openshift-ca" pod "0f9d2df7-6b86-4f0a-8f75-9a3d54f09aed" (UID: "0f9d2df7-6b86-4f0a-8f75-9a3d54f09aed"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:03:19.192475 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:19.192362    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8cef991-175d-4b30-8d0b-33f05e92a992-maas-api-xks-ca" pod "a8cef991-175d-4b30-8d0b-33f05e92a992" (UID: "a8cef991-175d-4b30-8d0b-33f05e92a992"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:03:19.192953 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:19.192919    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f9d2df7-6b86-4f0a-8f75-9a3d54f09aed-maas-api-xks-ca" pod "0f9d2df7-6b86-4f0a-8f75-9a3d54f09aed" (UID: "0f9d2df7-6b86-4f0a-8f75-9a3d54f09aed"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:03:19.195564 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:19.195542    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f9d2df7-6b86-4f0a-8f75-9a3d54f09aed-kube-api-access-rfxbk" pod "0f9d2df7-6b86-4f0a-8f75-9a3d54f09aed" (UID: "0f9d2df7-6b86-4f0a-8f75-9a3d54f09aed"). InnerVolumeSpecName "kube-api-access-rfxbk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:03:19.198011 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:19.197978    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8cef991-175d-4b30-8d0b-33f05e92a992-kube-api-access-kg96w" pod "a8cef991-175d-4b30-8d0b-33f05e92a992" (UID: "a8cef991-175d-4b30-8d0b-33f05e92a992"). InnerVolumeSpecName "kube-api-access-kg96w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:03:19.287532 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:19.287492    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8cef991-175d-4b30-8d0b-33f05e92a992-maas-api-xks-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:03:19.287532 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:19.287528    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f9d2df7-6b86-4f0a-8f75-9a3d54f09aed-maas-api-openshift-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:03:19.287747 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:19.287543    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kg96w\" (UniqueName: \"kubernetes.io/projected/a8cef991-175d-4b30-8d0b-33f05e92a992-kube-api-access-kg96w\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:03:19.287747 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:19.287557    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f9d2df7-6b86-4f0a-8f75-9a3d54f09aed-maas-api-xks-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:03:19.287747 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:19.287568    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8cef991-175d-4b30-8d0b-33f05e92a992-maas-api-openshift-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:03:19.287747 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:19.287581    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rfxbk\" (UniqueName: \"kubernetes.io/projected/0f9d2df7-6b86-4f0a-8f75-9a3d54f09aed-kube-api-access-rfxbk\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:03:19.910356 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:19.910324    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-481ab7-m4gqt"
Oct 30 01:03:19.910576 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:19.910358    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-481ab7-m4gqt" event={"ID":"0f9d2df7-6b86-4f0a-8f75-9a3d54f09aed","Type":"ContainerDied","Data":"697eb6e1ee266bf36641ed38cd63a3a627e130fccca91c82fbbed495422ba6a3"}
Oct 30 01:03:19.910576 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:19.910492    5002 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="697eb6e1ee266bf36641ed38cd63a3a627e130fccca91c82fbbed495422ba6a3"
Oct 30 01:03:19.911851 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:19.911828    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bafb00a1-hw4v7"
Oct 30 01:03:19.911994 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:19.911871    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bafb00a1-hw4v7" event={"ID":"a8cef991-175d-4b30-8d0b-33f05e92a992","Type":"ContainerDied","Data":"1e4744bc6a5c98181c8127ca72d3c27e2e7ae8bab9577f3dc09edf253c4ad0c5"}
Oct 30 01:03:19.911994 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:19.911918    5002 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1e4744bc6a5c98181c8127ca72d3c27e2e7ae8bab9577f3dc09edf253c4ad0c5"
Oct 30 01:03:23.202841 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:23.202801    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-lhfxq"]
Oct 30 01:03:23.203347 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:23.203054    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-lhfxq" podUID="543ddf6f-c221-48c0-b5d5-881c23397832" containerName="istio-proxy" containerID="cri-o://c2e515361511c699a057ce13a30e41a1bba07e19a8d1ea26054712ab170ff601" gracePeriod=30
Oct 30 01:03:23.410509 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:23.410473    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-vqjxg"]
Oct 30 01:03:23.410770 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:23.410726    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-vqjxg" podUID="00e1797d-0629-4b79-8073-eaa093f48f2a" containerName="istio-proxy" containerID="cri-o://6a9e84f080cf9b9a34947b12801f2a10a8ada55922ca91adf63d0e553378486a" gracePeriod=30
Oct 30 01:03:23.865164 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:23.865125    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-fjlsm"]
Oct 30 01:03:23.865414 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:23.865372    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-fjlsm" podUID="9c20eb99-f057-44bd-94e2-1972b67aa7f0" containerName="istio-proxy" containerID="cri-o://77e997a5afac45add3fe0b7b8a7e5224e426820f35fd950ee3e772038f1b2c5e" gracePeriod=30
Oct 30 01:03:28.124217 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:28.124178    5002 patch_prober.go:29] interesting pod/e2e-worker-w6-021d6d-openshift-default-65c58b6947-vqjxg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 01:03:28.124639 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:28.124248    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-vqjxg" podUID="00e1797d-0629-4b79-8073-eaa093f48f2a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 01:03:28.541951 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:28.541922    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-lhfxq"
Oct 30 01:03:28.691003 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:28.690965    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/543ddf6f-c221-48c0-b5d5-881c23397832-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/543ddf6f-c221-48c0-b5d5-881c23397832-istio-podinfo\") pod \"543ddf6f-c221-48c0-b5d5-881c23397832\" (UID: \"543ddf6f-c221-48c0-b5d5-881c23397832\") "
Oct 30 01:03:28.691232 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:28.691215    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/543ddf6f-c221-48c0-b5d5-881c23397832-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/543ddf6f-c221-48c0-b5d5-881c23397832-credential-socket\") pod \"543ddf6f-c221-48c0-b5d5-881c23397832\" (UID: \"543ddf6f-c221-48c0-b5d5-881c23397832\") "
Oct 30 01:03:28.691449 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:28.691423    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/543ddf6f-c221-48c0-b5d5-881c23397832-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/543ddf6f-c221-48c0-b5d5-881c23397832-istio-envoy\") pod \"543ddf6f-c221-48c0-b5d5-881c23397832\" (UID: \"543ddf6f-c221-48c0-b5d5-881c23397832\") "
Oct 30 01:03:28.691730 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:28.691714    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/543ddf6f-c221-48c0-b5d5-881c23397832-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/543ddf6f-c221-48c0-b5d5-881c23397832-istio-data\") pod \"543ddf6f-c221-48c0-b5d5-881c23397832\" (UID: \"543ddf6f-c221-48c0-b5d5-881c23397832\") "
Oct 30 01:03:28.692854 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:28.692034    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/543ddf6f-c221-48c0-b5d5-881c23397832-istio-data" pod "543ddf6f-c221-48c0-b5d5-881c23397832" (UID: "543ddf6f-c221-48c0-b5d5-881c23397832"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:03:28.692854 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:28.692161    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/543ddf6f-c221-48c0-b5d5-881c23397832-credential-socket" pod "543ddf6f-c221-48c0-b5d5-881c23397832" (UID: "543ddf6f-c221-48c0-b5d5-881c23397832"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:03:28.693137 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:28.693119    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/543ddf6f-c221-48c0-b5d5-881c23397832-istio-token\" (UniqueName: \"kubernetes.io/projected/543ddf6f-c221-48c0-b5d5-881c23397832-istio-token\") pod \"543ddf6f-c221-48c0-b5d5-881c23397832\" (UID: \"543ddf6f-c221-48c0-b5d5-881c23397832\") "
Oct 30 01:03:28.694407 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:28.693623    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/543ddf6f-c221-48c0-b5d5-881c23397832-kube-api-access-8n4rl\" (UniqueName: \"kubernetes.io/projected/543ddf6f-c221-48c0-b5d5-881c23397832-kube-api-access-8n4rl\") pod \"543ddf6f-c221-48c0-b5d5-881c23397832\" (UID: \"543ddf6f-c221-48c0-b5d5-881c23397832\") "
Oct 30 01:03:28.694407 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:28.693665    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/543ddf6f-c221-48c0-b5d5-881c23397832-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/543ddf6f-c221-48c0-b5d5-881c23397832-istiod-ca-cert\") pod \"543ddf6f-c221-48c0-b5d5-881c23397832\" (UID: \"543ddf6f-c221-48c0-b5d5-881c23397832\") "
Oct 30 01:03:28.694407 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:28.693749    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/543ddf6f-c221-48c0-b5d5-881c23397832-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/543ddf6f-c221-48c0-b5d5-881c23397832-workload-certs\") pod \"543ddf6f-c221-48c0-b5d5-881c23397832\" (UID: \"543ddf6f-c221-48c0-b5d5-881c23397832\") "
Oct 30 01:03:28.694407 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:28.693830    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/543ddf6f-c221-48c0-b5d5-881c23397832-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/543ddf6f-c221-48c0-b5d5-881c23397832-workload-socket\") pod \"543ddf6f-c221-48c0-b5d5-881c23397832\" (UID: \"543ddf6f-c221-48c0-b5d5-881c23397832\") "
Oct 30 01:03:28.694407 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:28.694321    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/543ddf6f-c221-48c0-b5d5-881c23397832-workload-certs" pod "543ddf6f-c221-48c0-b5d5-881c23397832" (UID: "543ddf6f-c221-48c0-b5d5-881c23397832"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:03:28.694772 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:28.694654    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/543ddf6f-c221-48c0-b5d5-881c23397832-istiod-ca-cert" pod "543ddf6f-c221-48c0-b5d5-881c23397832" (UID: "543ddf6f-c221-48c0-b5d5-881c23397832"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:03:28.694772 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:28.694715    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/543ddf6f-c221-48c0-b5d5-881c23397832-workload-socket" pod "543ddf6f-c221-48c0-b5d5-881c23397832" (UID: "543ddf6f-c221-48c0-b5d5-881c23397832"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:03:28.695204 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:28.695178    5002 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/543ddf6f-c221-48c0-b5d5-881c23397832-credential-socket\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:03:28.695291 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:28.695209    5002 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/543ddf6f-c221-48c0-b5d5-881c23397832-istio-data\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:03:28.695291 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:28.695224    5002 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/543ddf6f-c221-48c0-b5d5-881c23397832-istiod-ca-cert\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:03:28.695291 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:28.695236    5002 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/543ddf6f-c221-48c0-b5d5-881c23397832-workload-certs\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:03:28.695291 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:28.695249    5002 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/543ddf6f-c221-48c0-b5d5-881c23397832-workload-socket\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:03:28.706867 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:28.706759    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/543ddf6f-c221-48c0-b5d5-881c23397832-istio-podinfo" pod "543ddf6f-c221-48c0-b5d5-881c23397832" (UID: "543ddf6f-c221-48c0-b5d5-881c23397832"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:03:28.706867 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:28.706793    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/543ddf6f-c221-48c0-b5d5-881c23397832-istio-envoy" pod "543ddf6f-c221-48c0-b5d5-881c23397832" (UID: "543ddf6f-c221-48c0-b5d5-881c23397832"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:03:28.706867 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:28.706817    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/543ddf6f-c221-48c0-b5d5-881c23397832-istio-token" pod "543ddf6f-c221-48c0-b5d5-881c23397832" (UID: "543ddf6f-c221-48c0-b5d5-881c23397832"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:03:28.708528 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:28.708500    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/543ddf6f-c221-48c0-b5d5-881c23397832-kube-api-access-8n4rl" pod "543ddf6f-c221-48c0-b5d5-881c23397832" (UID: "543ddf6f-c221-48c0-b5d5-881c23397832"). InnerVolumeSpecName "kube-api-access-8n4rl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:03:28.719559 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:28.718978    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-vqjxg"
Oct 30 01:03:28.796732 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:28.796691    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00e1797d-0629-4b79-8073-eaa093f48f2a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00e1797d-0629-4b79-8073-eaa093f48f2a-workload-socket\") pod \"00e1797d-0629-4b79-8073-eaa093f48f2a\" (UID: \"00e1797d-0629-4b79-8073-eaa093f48f2a\") "
Oct 30 01:03:28.796943 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:28.796755    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00e1797d-0629-4b79-8073-eaa093f48f2a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00e1797d-0629-4b79-8073-eaa093f48f2a-istio-data\") pod \"00e1797d-0629-4b79-8073-eaa093f48f2a\" (UID: \"00e1797d-0629-4b79-8073-eaa093f48f2a\") "
Oct 30 01:03:28.796943 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:28.796873    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00e1797d-0629-4b79-8073-eaa093f48f2a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00e1797d-0629-4b79-8073-eaa093f48f2a-istio-envoy\") pod \"00e1797d-0629-4b79-8073-eaa093f48f2a\" (UID: \"00e1797d-0629-4b79-8073-eaa093f48f2a\") "
Oct 30 01:03:28.797092 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:28.796968    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00e1797d-0629-4b79-8073-eaa093f48f2a-istio-token\" (UniqueName: \"kubernetes.io/projected/00e1797d-0629-4b79-8073-eaa093f48f2a-istio-token\") pod \"00e1797d-0629-4b79-8073-eaa093f48f2a\" (UID: \"00e1797d-0629-4b79-8073-eaa093f48f2a\") "
Oct 30 01:03:28.797154 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:28.797090    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00e1797d-0629-4b79-8073-eaa093f48f2a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00e1797d-0629-4b79-8073-eaa093f48f2a-credential-socket\") pod \"00e1797d-0629-4b79-8073-eaa093f48f2a\" (UID: \"00e1797d-0629-4b79-8073-eaa093f48f2a\") "
Oct 30 01:03:28.797243 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:28.797188    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00e1797d-0629-4b79-8073-eaa093f48f2a-istio-data" pod "00e1797d-0629-4b79-8073-eaa093f48f2a" (UID: "00e1797d-0629-4b79-8073-eaa093f48f2a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:03:28.797243 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:28.797200    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00e1797d-0629-4b79-8073-eaa093f48f2a-kube-api-access-4mqb2\" (UniqueName: \"kubernetes.io/projected/00e1797d-0629-4b79-8073-eaa093f48f2a-kube-api-access-4mqb2\") pod \"00e1797d-0629-4b79-8073-eaa093f48f2a\" (UID: \"00e1797d-0629-4b79-8073-eaa093f48f2a\") "
Oct 30 01:03:28.797366 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:28.797268    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00e1797d-0629-4b79-8073-eaa093f48f2a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00e1797d-0629-4b79-8073-eaa093f48f2a-istiod-ca-cert\") pod \"00e1797d-0629-4b79-8073-eaa093f48f2a\" (UID: \"00e1797d-0629-4b79-8073-eaa093f48f2a\") "
Oct 30 01:03:28.797366 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:28.797331    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00e1797d-0629-4b79-8073-eaa093f48f2a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00e1797d-0629-4b79-8073-eaa093f48f2a-workload-certs\") pod \"00e1797d-0629-4b79-8073-eaa093f48f2a\" (UID: \"00e1797d-0629-4b79-8073-eaa093f48f2a\") "
Oct 30 01:03:28.797491 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:28.797416    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/00e1797d-0629-4b79-8073-eaa093f48f2a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00e1797d-0629-4b79-8073-eaa093f48f2a-istio-podinfo\") pod \"00e1797d-0629-4b79-8073-eaa093f48f2a\" (UID: \"00e1797d-0629-4b79-8073-eaa093f48f2a\") "
Oct 30 01:03:28.798707 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:28.798053    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00e1797d-0629-4b79-8073-eaa093f48f2a-istiod-ca-cert" pod "00e1797d-0629-4b79-8073-eaa093f48f2a" (UID: "00e1797d-0629-4b79-8073-eaa093f48f2a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:03:28.798707 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:28.798243    5002 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00e1797d-0629-4b79-8073-eaa093f48f2a-istiod-ca-cert\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:03:28.798707 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:28.798264    5002 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00e1797d-0629-4b79-8073-eaa093f48f2a-istio-data\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:03:28.798707 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:28.798279    5002 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/543ddf6f-c221-48c0-b5d5-881c23397832-istio-podinfo\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:03:28.798707 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:28.798295    5002 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/543ddf6f-c221-48c0-b5d5-881c23397832-istio-envoy\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:03:28.798707 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:28.798307    5002 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/543ddf6f-c221-48c0-b5d5-881c23397832-istio-token\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:03:28.798707 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:28.798321    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8n4rl\" (UniqueName: \"kubernetes.io/projected/543ddf6f-c221-48c0-b5d5-881c23397832-kube-api-access-8n4rl\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:03:28.798707 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:28.798400    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00e1797d-0629-4b79-8073-eaa093f48f2a-workload-socket" pod "00e1797d-0629-4b79-8073-eaa093f48f2a" (UID: "00e1797d-0629-4b79-8073-eaa093f48f2a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:03:28.798707 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:28.798684    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00e1797d-0629-4b79-8073-eaa093f48f2a-credential-socket" pod "00e1797d-0629-4b79-8073-eaa093f48f2a" (UID: "00e1797d-0629-4b79-8073-eaa093f48f2a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:03:28.799334 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:28.799295    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00e1797d-0629-4b79-8073-eaa093f48f2a-workload-certs" pod "00e1797d-0629-4b79-8073-eaa093f48f2a" (UID: "00e1797d-0629-4b79-8073-eaa093f48f2a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:03:28.802897 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:28.802800    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00e1797d-0629-4b79-8073-eaa093f48f2a-istio-envoy" pod "00e1797d-0629-4b79-8073-eaa093f48f2a" (UID: "00e1797d-0629-4b79-8073-eaa093f48f2a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:03:28.802897 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:28.802810    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00e1797d-0629-4b79-8073-eaa093f48f2a-kube-api-access-4mqb2" pod "00e1797d-0629-4b79-8073-eaa093f48f2a" (UID: "00e1797d-0629-4b79-8073-eaa093f48f2a"). InnerVolumeSpecName "kube-api-access-4mqb2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:03:28.802897 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:28.802862    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00e1797d-0629-4b79-8073-eaa093f48f2a-istio-token" pod "00e1797d-0629-4b79-8073-eaa093f48f2a" (UID: "00e1797d-0629-4b79-8073-eaa093f48f2a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:03:28.804617 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:28.804592    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/00e1797d-0629-4b79-8073-eaa093f48f2a-istio-podinfo" pod "00e1797d-0629-4b79-8073-eaa093f48f2a" (UID: "00e1797d-0629-4b79-8073-eaa093f48f2a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:03:28.899821 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:28.899787    5002 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00e1797d-0629-4b79-8073-eaa093f48f2a-credential-socket\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:03:28.899914 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:28.899827    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4mqb2\" (UniqueName: \"kubernetes.io/projected/00e1797d-0629-4b79-8073-eaa093f48f2a-kube-api-access-4mqb2\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:03:28.899914 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:28.899842    5002 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00e1797d-0629-4b79-8073-eaa093f48f2a-workload-certs\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:03:28.899914 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:28.899857    5002 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00e1797d-0629-4b79-8073-eaa093f48f2a-istio-podinfo\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:03:28.899914 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:28.899870    5002 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00e1797d-0629-4b79-8073-eaa093f48f2a-workload-socket\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:03:28.899914 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:28.899883    5002 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00e1797d-0629-4b79-8073-eaa093f48f2a-istio-envoy\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:03:28.899914 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:28.899895    5002 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00e1797d-0629-4b79-8073-eaa093f48f2a-istio-token\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:03:29.035845 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:29.035809    5002 generic.go:356] "Generic (PLEG): container finished" podID="543ddf6f-c221-48c0-b5d5-881c23397832" containerID="c2e515361511c699a057ce13a30e41a1bba07e19a8d1ea26054712ab170ff601" exitCode=0
Oct 30 01:03:29.036006 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:29.035860    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-lhfxq"
Oct 30 01:03:29.036006 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:29.035985    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-lhfxq" event={"ID":"543ddf6f-c221-48c0-b5d5-881c23397832","Type":"ContainerDied","Data":"c2e515361511c699a057ce13a30e41a1bba07e19a8d1ea26054712ab170ff601"}
Oct 30 01:03:29.036108 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:29.036033    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-lhfxq" event={"ID":"543ddf6f-c221-48c0-b5d5-881c23397832","Type":"ContainerDied","Data":"a262e7019ae4dfbaa0a3390837256a9dc8189e36c1871caff7df0323f2844f75"}
Oct 30 01:03:29.036108 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:29.036063    5002 scope.go:122] "RemoveContainer" containerID="c2e515361511c699a057ce13a30e41a1bba07e19a8d1ea26054712ab170ff601"
Oct 30 01:03:29.038649 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:29.038622    5002 generic.go:356] "Generic (PLEG): container finished" podID="9c20eb99-f057-44bd-94e2-1972b67aa7f0" containerID="77e997a5afac45add3fe0b7b8a7e5224e426820f35fd950ee3e772038f1b2c5e" exitCode=0
Oct 30 01:03:29.038770 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:29.038661    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-fjlsm" event={"ID":"9c20eb99-f057-44bd-94e2-1972b67aa7f0","Type":"ContainerDied","Data":"77e997a5afac45add3fe0b7b8a7e5224e426820f35fd950ee3e772038f1b2c5e"}
Oct 30 01:03:29.040503 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:29.040476    5002 generic.go:356] "Generic (PLEG): container finished" podID="00e1797d-0629-4b79-8073-eaa093f48f2a" containerID="6a9e84f080cf9b9a34947b12801f2a10a8ada55922ca91adf63d0e553378486a" exitCode=0
Oct 30 01:03:29.040617 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:29.040531    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-vqjxg" event={"ID":"00e1797d-0629-4b79-8073-eaa093f48f2a","Type":"ContainerDied","Data":"6a9e84f080cf9b9a34947b12801f2a10a8ada55922ca91adf63d0e553378486a"}
Oct 30 01:03:29.040617 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:29.040555    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-vqjxg"
Oct 30 01:03:29.040617 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:29.040571    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-vqjxg" event={"ID":"00e1797d-0629-4b79-8073-eaa093f48f2a","Type":"ContainerDied","Data":"1bf3e181715b960439a93de324f1da7cfcd1197f464b7d0caf84a501b5688011"}
Oct 30 01:03:29.053080 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:29.053057    5002 scope.go:122] "RemoveContainer" containerID="c2e515361511c699a057ce13a30e41a1bba07e19a8d1ea26054712ab170ff601"
Oct 30 01:03:29.054300 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:03:29.054251    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2e515361511c699a057ce13a30e41a1bba07e19a8d1ea26054712ab170ff601\": container with ID starting with c2e515361511c699a057ce13a30e41a1bba07e19a8d1ea26054712ab170ff601 not found: ID does not exist" containerID="c2e515361511c699a057ce13a30e41a1bba07e19a8d1ea26054712ab170ff601"
Oct 30 01:03:29.054300 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:29.054288    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2e515361511c699a057ce13a30e41a1bba07e19a8d1ea26054712ab170ff601"} err="failed to get container status \"c2e515361511c699a057ce13a30e41a1bba07e19a8d1ea26054712ab170ff601\": rpc error: code = NotFound desc = could not find container \"c2e515361511c699a057ce13a30e41a1bba07e19a8d1ea26054712ab170ff601\": container with ID starting with c2e515361511c699a057ce13a30e41a1bba07e19a8d1ea26054712ab170ff601 not found: ID does not exist"
Oct 30 01:03:29.054552 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:29.054312    5002 scope.go:122] "RemoveContainer" containerID="6a9e84f080cf9b9a34947b12801f2a10a8ada55922ca91adf63d0e553378486a"
Oct 30 01:03:29.099286 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:29.098745    5002 scope.go:122] "RemoveContainer" containerID="6a9e84f080cf9b9a34947b12801f2a10a8ada55922ca91adf63d0e553378486a"
Oct 30 01:03:29.100292 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:03:29.099555    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a9e84f080cf9b9a34947b12801f2a10a8ada55922ca91adf63d0e553378486a\": container with ID starting with 6a9e84f080cf9b9a34947b12801f2a10a8ada55922ca91adf63d0e553378486a not found: ID does not exist" containerID="6a9e84f080cf9b9a34947b12801f2a10a8ada55922ca91adf63d0e553378486a"
Oct 30 01:03:29.100292 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:29.099596    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a9e84f080cf9b9a34947b12801f2a10a8ada55922ca91adf63d0e553378486a"} err="failed to get container status \"6a9e84f080cf9b9a34947b12801f2a10a8ada55922ca91adf63d0e553378486a\": rpc error: code = NotFound desc = could not find container \"6a9e84f080cf9b9a34947b12801f2a10a8ada55922ca91adf63d0e553378486a\": container with ID starting with 6a9e84f080cf9b9a34947b12801f2a10a8ada55922ca91adf63d0e553378486a not found: ID does not exist"
Oct 30 01:03:29.112887 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:29.112849    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-lhfxq"]
Oct 30 01:03:29.137400 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:29.118648    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-lhfxq"]
Oct 30 01:03:29.188894 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:29.185476    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-vqjxg"]
Oct 30 01:03:29.193646 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:29.193619    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-vqjxg"]
Oct 30 01:03:29.194419 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:29.194308    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-fjlsm"
Oct 30 01:03:29.305262 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:29.305223    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c20eb99-f057-44bd-94e2-1972b67aa7f0-istio-token\" (UniqueName: \"kubernetes.io/projected/9c20eb99-f057-44bd-94e2-1972b67aa7f0-istio-token\") pod \"9c20eb99-f057-44bd-94e2-1972b67aa7f0\" (UID: \"9c20eb99-f057-44bd-94e2-1972b67aa7f0\") "
Oct 30 01:03:29.305524 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:29.305279    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c20eb99-f057-44bd-94e2-1972b67aa7f0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c20eb99-f057-44bd-94e2-1972b67aa7f0-credential-socket\") pod \"9c20eb99-f057-44bd-94e2-1972b67aa7f0\" (UID: \"9c20eb99-f057-44bd-94e2-1972b67aa7f0\") "
Oct 30 01:03:29.305524 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:29.305334    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c20eb99-f057-44bd-94e2-1972b67aa7f0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c20eb99-f057-44bd-94e2-1972b67aa7f0-istio-data\") pod \"9c20eb99-f057-44bd-94e2-1972b67aa7f0\" (UID: \"9c20eb99-f057-44bd-94e2-1972b67aa7f0\") "
Oct 30 01:03:29.305524 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:29.305364    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c20eb99-f057-44bd-94e2-1972b67aa7f0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c20eb99-f057-44bd-94e2-1972b67aa7f0-istiod-ca-cert\") pod \"9c20eb99-f057-44bd-94e2-1972b67aa7f0\" (UID: \"9c20eb99-f057-44bd-94e2-1972b67aa7f0\") "
Oct 30 01:03:29.305524 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:29.305412    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9c20eb99-f057-44bd-94e2-1972b67aa7f0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c20eb99-f057-44bd-94e2-1972b67aa7f0-istio-podinfo\") pod \"9c20eb99-f057-44bd-94e2-1972b67aa7f0\" (UID: \"9c20eb99-f057-44bd-94e2-1972b67aa7f0\") "
Oct 30 01:03:29.305524 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:29.305451    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c20eb99-f057-44bd-94e2-1972b67aa7f0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c20eb99-f057-44bd-94e2-1972b67aa7f0-workload-socket\") pod \"9c20eb99-f057-44bd-94e2-1972b67aa7f0\" (UID: \"9c20eb99-f057-44bd-94e2-1972b67aa7f0\") "
Oct 30 01:03:29.305524 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:29.305491    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c20eb99-f057-44bd-94e2-1972b67aa7f0-kube-api-access-kxkq7\" (UniqueName: \"kubernetes.io/projected/9c20eb99-f057-44bd-94e2-1972b67aa7f0-kube-api-access-kxkq7\") pod \"9c20eb99-f057-44bd-94e2-1972b67aa7f0\" (UID: \"9c20eb99-f057-44bd-94e2-1972b67aa7f0\") "
Oct 30 01:03:29.305897 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:29.305544    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c20eb99-f057-44bd-94e2-1972b67aa7f0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c20eb99-f057-44bd-94e2-1972b67aa7f0-istio-envoy\") pod \"9c20eb99-f057-44bd-94e2-1972b67aa7f0\" (UID: \"9c20eb99-f057-44bd-94e2-1972b67aa7f0\") "
Oct 30 01:03:29.305897 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:29.305729    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c20eb99-f057-44bd-94e2-1972b67aa7f0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c20eb99-f057-44bd-94e2-1972b67aa7f0-workload-certs\") pod \"9c20eb99-f057-44bd-94e2-1972b67aa7f0\" (UID: \"9c20eb99-f057-44bd-94e2-1972b67aa7f0\") "
Oct 30 01:03:29.305897 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:29.305737    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c20eb99-f057-44bd-94e2-1972b67aa7f0-credential-socket" pod "9c20eb99-f057-44bd-94e2-1972b67aa7f0" (UID: "9c20eb99-f057-44bd-94e2-1972b67aa7f0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:03:29.308305 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:29.308276    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9c20eb99-f057-44bd-94e2-1972b67aa7f0-istio-podinfo" pod "9c20eb99-f057-44bd-94e2-1972b67aa7f0" (UID: "9c20eb99-f057-44bd-94e2-1972b67aa7f0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:03:29.308569 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:29.308543    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c20eb99-f057-44bd-94e2-1972b67aa7f0-istio-token" pod "9c20eb99-f057-44bd-94e2-1972b67aa7f0" (UID: "9c20eb99-f057-44bd-94e2-1972b67aa7f0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:03:29.308659 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:29.308644    5002 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c20eb99-f057-44bd-94e2-1972b67aa7f0-credential-socket\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:03:29.308715 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:29.308664    5002 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c20eb99-f057-44bd-94e2-1972b67aa7f0-istio-podinfo\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:03:29.309104 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:29.309076    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c20eb99-f057-44bd-94e2-1972b67aa7f0-workload-certs" pod "9c20eb99-f057-44bd-94e2-1972b67aa7f0" (UID: "9c20eb99-f057-44bd-94e2-1972b67aa7f0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:03:29.309274 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:29.309226    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c20eb99-f057-44bd-94e2-1972b67aa7f0-istiod-ca-cert" pod "9c20eb99-f057-44bd-94e2-1972b67aa7f0" (UID: "9c20eb99-f057-44bd-94e2-1972b67aa7f0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:03:29.311147 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:29.309472    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c20eb99-f057-44bd-94e2-1972b67aa7f0-workload-socket" pod "9c20eb99-f057-44bd-94e2-1972b67aa7f0" (UID: "9c20eb99-f057-44bd-94e2-1972b67aa7f0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:03:29.311447 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:29.311400    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c20eb99-f057-44bd-94e2-1972b67aa7f0-istio-envoy" pod "9c20eb99-f057-44bd-94e2-1972b67aa7f0" (UID: "9c20eb99-f057-44bd-94e2-1972b67aa7f0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:03:29.312149 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:29.312129    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c20eb99-f057-44bd-94e2-1972b67aa7f0-istio-data" pod "9c20eb99-f057-44bd-94e2-1972b67aa7f0" (UID: "9c20eb99-f057-44bd-94e2-1972b67aa7f0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:03:29.313749 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:29.313709    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c20eb99-f057-44bd-94e2-1972b67aa7f0-kube-api-access-kxkq7" pod "9c20eb99-f057-44bd-94e2-1972b67aa7f0" (UID: "9c20eb99-f057-44bd-94e2-1972b67aa7f0"). InnerVolumeSpecName "kube-api-access-kxkq7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:03:29.423343 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:29.416986    5002 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c20eb99-f057-44bd-94e2-1972b67aa7f0-istio-token\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:03:29.423818 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:29.423784    5002 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c20eb99-f057-44bd-94e2-1972b67aa7f0-istio-data\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:03:29.430336 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:29.424475    5002 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c20eb99-f057-44bd-94e2-1972b67aa7f0-istiod-ca-cert\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:03:29.430575 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:29.430556    5002 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c20eb99-f057-44bd-94e2-1972b67aa7f0-workload-socket\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:03:29.430679 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:29.430666    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kxkq7\" (UniqueName: \"kubernetes.io/projected/9c20eb99-f057-44bd-94e2-1972b67aa7f0-kube-api-access-kxkq7\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:03:29.430854 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:29.430820    5002 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c20eb99-f057-44bd-94e2-1972b67aa7f0-istio-envoy\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:03:29.430854 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:29.430846    5002 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c20eb99-f057-44bd-94e2-1972b67aa7f0-workload-certs\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:03:30.063010 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:30.062968    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-fjlsm" event={"ID":"9c20eb99-f057-44bd-94e2-1972b67aa7f0","Type":"ContainerDied","Data":"74b5e630cd91f614eafd425f493ba3b70e9d665b92376541263d0dc1f7f8b470"}
Oct 30 01:03:30.063219 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:30.063067    5002 scope.go:122] "RemoveContainer" containerID="77e997a5afac45add3fe0b7b8a7e5224e426820f35fd950ee3e772038f1b2c5e"
Oct 30 01:03:30.063219 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:30.062982    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-fjlsm"
Oct 30 01:03:30.126345 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:30.126307    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-8dbe73/test-model-8dbe73-kserve-78dbd9d5cc-5zdxp" podUID="f3cd6ac5-b9a0-4096-94ee-851e2042e9fa" containerName="main" containerID="cri-o://dced08146389e2646b17865cb9b689035cb4d3defe5a42def75fd1c96b6f67c3" gracePeriod=45
Oct 30 01:03:30.140185 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:30.140160    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-fjlsm"]
Oct 30 01:03:30.145450 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:30.145423    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-fjlsm"]
Oct 30 01:03:30.197067 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:03:30.197030    5002 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf3cd6ac5_b9a0_4096_94ee_851e2042e9fa.slice/crio-conmon-dced08146389e2646b17865cb9b689035cb4d3defe5a42def75fd1c96b6f67c3.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 01:03:30.431008 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:30.430979    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-8dbe73/test-model-8dbe73-kserve-78dbd9d5cc-5zdxp"
Oct 30 01:03:30.498969 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:30.498931    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00e1797d-0629-4b79-8073-eaa093f48f2a" path="/var/lib/kubelet/pods/00e1797d-0629-4b79-8073-eaa093f48f2a/volumes"
Oct 30 01:03:30.499614 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:30.499590    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="543ddf6f-c221-48c0-b5d5-881c23397832" path="/var/lib/kubelet/pods/543ddf6f-c221-48c0-b5d5-881c23397832/volumes"
Oct 30 01:03:30.500182 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:30.500160    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c20eb99-f057-44bd-94e2-1972b67aa7f0" path="/var/lib/kubelet/pods/9c20eb99-f057-44bd-94e2-1972b67aa7f0/volumes"
Oct 30 01:03:30.547123 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:30.547094    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3cd6ac5-b9a0-4096-94ee-851e2042e9fa-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3cd6ac5-b9a0-4096-94ee-851e2042e9fa-tmp-dir\") pod \"f3cd6ac5-b9a0-4096-94ee-851e2042e9fa\" (UID: \"f3cd6ac5-b9a0-4096-94ee-851e2042e9fa\") "
Oct 30 01:03:30.547279 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:30.547155    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3cd6ac5-b9a0-4096-94ee-851e2042e9fa-home\" (UniqueName: \"kubernetes.io/empty-dir/f3cd6ac5-b9a0-4096-94ee-851e2042e9fa-home\") pod \"f3cd6ac5-b9a0-4096-94ee-851e2042e9fa\" (UID: \"f3cd6ac5-b9a0-4096-94ee-851e2042e9fa\") "
Oct 30 01:03:30.547279 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:30.547183    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3cd6ac5-b9a0-4096-94ee-851e2042e9fa-kube-api-access-6tctb\" (UniqueName: \"kubernetes.io/projected/f3cd6ac5-b9a0-4096-94ee-851e2042e9fa-kube-api-access-6tctb\") pod \"f3cd6ac5-b9a0-4096-94ee-851e2042e9fa\" (UID: \"f3cd6ac5-b9a0-4096-94ee-851e2042e9fa\") "
Oct 30 01:03:30.547279 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:30.547253    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f3cd6ac5-b9a0-4096-94ee-851e2042e9fa-tls-certs\" (UniqueName: \"kubernetes.io/secret/f3cd6ac5-b9a0-4096-94ee-851e2042e9fa-tls-certs\") pod \"f3cd6ac5-b9a0-4096-94ee-851e2042e9fa\" (UID: \"f3cd6ac5-b9a0-4096-94ee-851e2042e9fa\") "
Oct 30 01:03:30.547518 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:30.547372    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3cd6ac5-b9a0-4096-94ee-851e2042e9fa-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3cd6ac5-b9a0-4096-94ee-851e2042e9fa-model-cache\") pod \"f3cd6ac5-b9a0-4096-94ee-851e2042e9fa\" (UID: \"f3cd6ac5-b9a0-4096-94ee-851e2042e9fa\") "
Oct 30 01:03:30.547518 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:30.547421    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3cd6ac5-b9a0-4096-94ee-851e2042e9fa-tmp-dir" pod "f3cd6ac5-b9a0-4096-94ee-851e2042e9fa" (UID: "f3cd6ac5-b9a0-4096-94ee-851e2042e9fa"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:03:30.547518 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:30.547490    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3cd6ac5-b9a0-4096-94ee-851e2042e9fa-home" pod "f3cd6ac5-b9a0-4096-94ee-851e2042e9fa" (UID: "f3cd6ac5-b9a0-4096-94ee-851e2042e9fa"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:03:30.547672 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:30.547570    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3cd6ac5-b9a0-4096-94ee-851e2042e9fa-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3cd6ac5-b9a0-4096-94ee-851e2042e9fa-dshm\") pod \"f3cd6ac5-b9a0-4096-94ee-851e2042e9fa\" (UID: \"f3cd6ac5-b9a0-4096-94ee-851e2042e9fa\") "
Oct 30 01:03:30.547672 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:30.547612    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3cd6ac5-b9a0-4096-94ee-851e2042e9fa-model-cache" pod "f3cd6ac5-b9a0-4096-94ee-851e2042e9fa" (UID: "f3cd6ac5-b9a0-4096-94ee-851e2042e9fa"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:03:30.548058 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:30.548035    5002 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3cd6ac5-b9a0-4096-94ee-851e2042e9fa-model-cache\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:03:30.548157 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:30.548082    5002 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3cd6ac5-b9a0-4096-94ee-851e2042e9fa-tmp-dir\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:03:30.548157 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:30.548095    5002 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3cd6ac5-b9a0-4096-94ee-851e2042e9fa-home\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:03:30.549609 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:30.549583    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3cd6ac5-b9a0-4096-94ee-851e2042e9fa-dshm" pod "f3cd6ac5-b9a0-4096-94ee-851e2042e9fa" (UID: "f3cd6ac5-b9a0-4096-94ee-851e2042e9fa"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:03:30.549717 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:30.549665    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3cd6ac5-b9a0-4096-94ee-851e2042e9fa-tls-certs" pod "f3cd6ac5-b9a0-4096-94ee-851e2042e9fa" (UID: "f3cd6ac5-b9a0-4096-94ee-851e2042e9fa"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:03:30.549717 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:30.549698    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3cd6ac5-b9a0-4096-94ee-851e2042e9fa-kube-api-access-6tctb" pod "f3cd6ac5-b9a0-4096-94ee-851e2042e9fa" (UID: "f3cd6ac5-b9a0-4096-94ee-851e2042e9fa"). InnerVolumeSpecName "kube-api-access-6tctb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:03:30.573686 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:30.573651    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-03d892/test-model-03d892-kserve-7f8c7cb699-kp2mb" podUID="b3e31ee3-d2eb-42b1-84dc-173ed4434f84" containerName="main" containerID="cri-o://15b3aa67bbfbaebf2826c2b7f5c0eed9397463d1f8b9a8e96d4995cc869f44b0" gracePeriod=45
Oct 30 01:03:30.650490 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:30.650413    5002 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3cd6ac5-b9a0-4096-94ee-851e2042e9fa-tls-certs\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:03:30.650490 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:30.650452    5002 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3cd6ac5-b9a0-4096-94ee-851e2042e9fa-dshm\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:03:30.650490 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:30.650470    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6tctb\" (UniqueName: \"kubernetes.io/projected/f3cd6ac5-b9a0-4096-94ee-851e2042e9fa-kube-api-access-6tctb\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:03:30.858555 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:30.858521    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-03d892/test-model-03d892-kserve-7f8c7cb699-kp2mb"
Oct 30 01:03:30.958921 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:30.958882    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3e31ee3-d2eb-42b1-84dc-173ed4434f84-kube-api-access-8l9n7\" (UniqueName: \"kubernetes.io/projected/b3e31ee3-d2eb-42b1-84dc-173ed4434f84-kube-api-access-8l9n7\") pod \"b3e31ee3-d2eb-42b1-84dc-173ed4434f84\" (UID: \"b3e31ee3-d2eb-42b1-84dc-173ed4434f84\") "
Oct 30 01:03:30.959323 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:30.959296    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3e31ee3-d2eb-42b1-84dc-173ed4434f84-home\" (UniqueName: \"kubernetes.io/empty-dir/b3e31ee3-d2eb-42b1-84dc-173ed4434f84-home\") pod \"b3e31ee3-d2eb-42b1-84dc-173ed4434f84\" (UID: \"b3e31ee3-d2eb-42b1-84dc-173ed4434f84\") "
Oct 30 01:03:30.959811 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:30.959789    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3e31ee3-d2eb-42b1-84dc-173ed4434f84-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3e31ee3-d2eb-42b1-84dc-173ed4434f84-tmp-dir\") pod \"b3e31ee3-d2eb-42b1-84dc-173ed4434f84\" (UID: \"b3e31ee3-d2eb-42b1-84dc-173ed4434f84\") "
Oct 30 01:03:30.960034 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:30.960005    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3e31ee3-d2eb-42b1-84dc-173ed4434f84-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3e31ee3-d2eb-42b1-84dc-173ed4434f84-model-cache\") pod \"b3e31ee3-d2eb-42b1-84dc-173ed4434f84\" (UID: \"b3e31ee3-d2eb-42b1-84dc-173ed4434f84\") "
Oct 30 01:03:30.960139 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:30.960104    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b3e31ee3-d2eb-42b1-84dc-173ed4434f84-tls-certs\" (UniqueName: \"kubernetes.io/secret/b3e31ee3-d2eb-42b1-84dc-173ed4434f84-tls-certs\") pod \"b3e31ee3-d2eb-42b1-84dc-173ed4434f84\" (UID: \"b3e31ee3-d2eb-42b1-84dc-173ed4434f84\") "
Oct 30 01:03:30.961167 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:30.960229    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3e31ee3-d2eb-42b1-84dc-173ed4434f84-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b3e31ee3-d2eb-42b1-84dc-173ed4434f84-dshm\") pod \"b3e31ee3-d2eb-42b1-84dc-173ed4434f84\" (UID: \"b3e31ee3-d2eb-42b1-84dc-173ed4434f84\") "
Oct 30 01:03:30.961167 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:30.960462    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3e31ee3-d2eb-42b1-84dc-173ed4434f84-home" pod "b3e31ee3-d2eb-42b1-84dc-173ed4434f84" (UID: "b3e31ee3-d2eb-42b1-84dc-173ed4434f84"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:03:30.961167 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:30.961096    5002 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b3e31ee3-d2eb-42b1-84dc-173ed4434f84-home\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:03:30.961405 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:30.961358    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3e31ee3-d2eb-42b1-84dc-173ed4434f84-model-cache" pod "b3e31ee3-d2eb-42b1-84dc-173ed4434f84" (UID: "b3e31ee3-d2eb-42b1-84dc-173ed4434f84"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:03:30.961579 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:30.961549    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3e31ee3-d2eb-42b1-84dc-173ed4434f84-tmp-dir" pod "b3e31ee3-d2eb-42b1-84dc-173ed4434f84" (UID: "b3e31ee3-d2eb-42b1-84dc-173ed4434f84"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:03:30.962680 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:30.962648    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3e31ee3-d2eb-42b1-84dc-173ed4434f84-kube-api-access-8l9n7" pod "b3e31ee3-d2eb-42b1-84dc-173ed4434f84" (UID: "b3e31ee3-d2eb-42b1-84dc-173ed4434f84"). InnerVolumeSpecName "kube-api-access-8l9n7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:03:30.963267 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:30.963247    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3e31ee3-d2eb-42b1-84dc-173ed4434f84-dshm" pod "b3e31ee3-d2eb-42b1-84dc-173ed4434f84" (UID: "b3e31ee3-d2eb-42b1-84dc-173ed4434f84"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:03:30.963780 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:30.963757    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3e31ee3-d2eb-42b1-84dc-173ed4434f84-tls-certs" pod "b3e31ee3-d2eb-42b1-84dc-173ed4434f84" (UID: "b3e31ee3-d2eb-42b1-84dc-173ed4434f84"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:03:31.063036 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:31.063001    5002 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3e31ee3-d2eb-42b1-84dc-173ed4434f84-tls-certs\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:03:31.063036 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:31.063031    5002 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b3e31ee3-d2eb-42b1-84dc-173ed4434f84-dshm\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:03:31.063036 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:31.063041    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8l9n7\" (UniqueName: \"kubernetes.io/projected/b3e31ee3-d2eb-42b1-84dc-173ed4434f84-kube-api-access-8l9n7\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:03:31.063332 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:31.063050    5002 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3e31ee3-d2eb-42b1-84dc-173ed4434f84-tmp-dir\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:03:31.063332 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:31.063060    5002 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3e31ee3-d2eb-42b1-84dc-173ed4434f84-model-cache\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:03:31.079481 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:31.079447    5002 generic.go:356] "Generic (PLEG): container finished" podID="b3e31ee3-d2eb-42b1-84dc-173ed4434f84" containerID="15b3aa67bbfbaebf2826c2b7f5c0eed9397463d1f8b9a8e96d4995cc869f44b0" exitCode=0
Oct 30 01:03:31.079655 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:31.079504    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-03d892/test-model-03d892-kserve-7f8c7cb699-kp2mb"
Oct 30 01:03:31.079655 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:31.079543    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-03d892/test-model-03d892-kserve-7f8c7cb699-kp2mb" event={"ID":"b3e31ee3-d2eb-42b1-84dc-173ed4434f84","Type":"ContainerDied","Data":"15b3aa67bbfbaebf2826c2b7f5c0eed9397463d1f8b9a8e96d4995cc869f44b0"}
Oct 30 01:03:31.079655 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:31.079582    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-03d892/test-model-03d892-kserve-7f8c7cb699-kp2mb" event={"ID":"b3e31ee3-d2eb-42b1-84dc-173ed4434f84","Type":"ContainerDied","Data":"21a76f73e177bad352b4f33477c04d62cd011d33833f0b8e4a2dd1334ac17658"}
Oct 30 01:03:31.079655 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:31.079603    5002 scope.go:122] "RemoveContainer" containerID="15b3aa67bbfbaebf2826c2b7f5c0eed9397463d1f8b9a8e96d4995cc869f44b0"
Oct 30 01:03:31.081628 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:31.081584    5002 generic.go:356] "Generic (PLEG): container finished" podID="f3cd6ac5-b9a0-4096-94ee-851e2042e9fa" containerID="dced08146389e2646b17865cb9b689035cb4d3defe5a42def75fd1c96b6f67c3" exitCode=0
Oct 30 01:03:31.081771 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:31.081653    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-8dbe73/test-model-8dbe73-kserve-78dbd9d5cc-5zdxp"
Oct 30 01:03:31.081771 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:31.081678    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-8dbe73/test-model-8dbe73-kserve-78dbd9d5cc-5zdxp" event={"ID":"f3cd6ac5-b9a0-4096-94ee-851e2042e9fa","Type":"ContainerDied","Data":"dced08146389e2646b17865cb9b689035cb4d3defe5a42def75fd1c96b6f67c3"}
Oct 30 01:03:31.081771 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:31.081710    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-8dbe73/test-model-8dbe73-kserve-78dbd9d5cc-5zdxp" event={"ID":"f3cd6ac5-b9a0-4096-94ee-851e2042e9fa","Type":"ContainerDied","Data":"7ae2474e8693bc900dc0769ef918dee3c9da684ac70c652c9571481b8c014215"}
Oct 30 01:03:31.092514 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:31.092370    5002 scope.go:122] "RemoveContainer" containerID="15b3aa67bbfbaebf2826c2b7f5c0eed9397463d1f8b9a8e96d4995cc869f44b0"
Oct 30 01:03:31.092821 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:03:31.092717    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15b3aa67bbfbaebf2826c2b7f5c0eed9397463d1f8b9a8e96d4995cc869f44b0\": container with ID starting with 15b3aa67bbfbaebf2826c2b7f5c0eed9397463d1f8b9a8e96d4995cc869f44b0 not found: ID does not exist" containerID="15b3aa67bbfbaebf2826c2b7f5c0eed9397463d1f8b9a8e96d4995cc869f44b0"
Oct 30 01:03:31.092821 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:31.092758    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15b3aa67bbfbaebf2826c2b7f5c0eed9397463d1f8b9a8e96d4995cc869f44b0"} err="failed to get container status \"15b3aa67bbfbaebf2826c2b7f5c0eed9397463d1f8b9a8e96d4995cc869f44b0\": rpc error: code = NotFound desc = could not find container \"15b3aa67bbfbaebf2826c2b7f5c0eed9397463d1f8b9a8e96d4995cc869f44b0\": container with ID starting with 15b3aa67bbfbaebf2826c2b7f5c0eed9397463d1f8b9a8e96d4995cc869f44b0 not found: ID does not exist"
Oct 30 01:03:31.092821 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:31.092784    5002 scope.go:122] "RemoveContainer" containerID="dced08146389e2646b17865cb9b689035cb4d3defe5a42def75fd1c96b6f67c3"
Oct 30 01:03:31.103487 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:31.103466    5002 scope.go:122] "RemoveContainer" containerID="dced08146389e2646b17865cb9b689035cb4d3defe5a42def75fd1c96b6f67c3"
Oct 30 01:03:31.103740 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:03:31.103724    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dced08146389e2646b17865cb9b689035cb4d3defe5a42def75fd1c96b6f67c3\": container with ID starting with dced08146389e2646b17865cb9b689035cb4d3defe5a42def75fd1c96b6f67c3 not found: ID does not exist" containerID="dced08146389e2646b17865cb9b689035cb4d3defe5a42def75fd1c96b6f67c3"
Oct 30 01:03:31.103792 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:31.103746    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dced08146389e2646b17865cb9b689035cb4d3defe5a42def75fd1c96b6f67c3"} err="failed to get container status \"dced08146389e2646b17865cb9b689035cb4d3defe5a42def75fd1c96b6f67c3\": rpc error: code = NotFound desc = could not find container \"dced08146389e2646b17865cb9b689035cb4d3defe5a42def75fd1c96b6f67c3\": container with ID starting with dced08146389e2646b17865cb9b689035cb4d3defe5a42def75fd1c96b6f67c3 not found: ID does not exist"
Oct 30 01:03:31.139173 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:31.139141    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-8dbe73/test-model-8dbe73-kserve-78dbd9d5cc-5zdxp"]
Oct 30 01:03:31.142731 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:31.142686    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-8dbe73/test-model-8dbe73-kserve-78dbd9d5cc-5zdxp"]
Oct 30 01:03:31.215502 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:31.198912    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-03d892/test-model-03d892-kserve-7f8c7cb699-kp2mb"]
Oct 30 01:03:31.219582 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:31.219536    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-03d892/test-model-03d892-kserve-7f8c7cb699-kp2mb"]
Oct 30 01:03:31.389565 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:31.389510    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-8dbe73/test-model-8dbe73-kserve-78dbd9d5cc-5zdxp" podUID="f3cd6ac5-b9a0-4096-94ee-851e2042e9fa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.137:8000/ready\": context deadline exceeded"
Oct 30 01:03:32.498830 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:32.498729    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3e31ee3-d2eb-42b1-84dc-173ed4434f84" path="/var/lib/kubelet/pods/b3e31ee3-d2eb-42b1-84dc-173ed4434f84/volumes"
Oct 30 01:03:32.499444 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:32.499374    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3cd6ac5-b9a0-4096-94ee-851e2042e9fa" path="/var/lib/kubelet/pods/f3cd6ac5-b9a0-4096-94ee-851e2042e9fa/volumes"
Oct 30 01:03:34.969410 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:34.967180    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7d8z4"]
Oct 30 01:03:38.913404 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:38.909516    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-84b89f/test-model-84b89f-kserve-d8c475485-5prlt"]
Oct 30 01:03:39.338746 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:39.338713    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-481ab7-m4gqt"]
Oct 30 01:03:39.342189 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:39.342159    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-481ab7-m4gqt"]
Oct 30 01:03:39.444699 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:39.444661    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8dbe73-wj7p4"]
Oct 30 01:03:39.478797 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:39.476656    5002 cpu_manager.go:414] "containerMap: removing container" podUID="00e1797d-0629-4b79-8073-eaa093f48f2a" containerName="istio-proxy"
Oct 30 01:03:39.478797 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:39.476685    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00e1797d-0629-4b79-8073-eaa093f48f2a" containerName="istio-proxy"
Oct 30 01:03:39.478797 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:39.476697    5002 cpu_manager.go:414] "containerMap: removing container" podUID="a8cef991-175d-4b30-8d0b-33f05e92a992" containerName="revoke-keys"
Oct 30 01:03:39.478797 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:39.476703    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8cef991-175d-4b30-8d0b-33f05e92a992" containerName="revoke-keys"
Oct 30 01:03:39.478797 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:39.476711    5002 cpu_manager.go:414] "containerMap: removing container" podUID="0f9d2df7-6b86-4f0a-8f75-9a3d54f09aed" containerName="revoke-keys"
Oct 30 01:03:39.478797 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:39.476716    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f9d2df7-6b86-4f0a-8f75-9a3d54f09aed" containerName="revoke-keys"
Oct 30 01:03:39.478797 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:39.476726    5002 cpu_manager.go:414] "containerMap: removing container" podUID="9c20eb99-f057-44bd-94e2-1972b67aa7f0" containerName="istio-proxy"
Oct 30 01:03:39.478797 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:39.476732    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c20eb99-f057-44bd-94e2-1972b67aa7f0" containerName="istio-proxy"
Oct 30 01:03:39.478797 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:39.476771    5002 cpu_manager.go:414] "containerMap: removing container" podUID="543ddf6f-c221-48c0-b5d5-881c23397832" containerName="istio-proxy"
Oct 30 01:03:39.478797 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:39.476779    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="543ddf6f-c221-48c0-b5d5-881c23397832" containerName="istio-proxy"
Oct 30 01:03:39.478797 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:39.476793    5002 cpu_manager.go:414] "containerMap: removing container" podUID="f3cd6ac5-b9a0-4096-94ee-851e2042e9fa" containerName="main"
Oct 30 01:03:39.478797 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:39.476798    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3cd6ac5-b9a0-4096-94ee-851e2042e9fa" containerName="main"
Oct 30 01:03:39.478797 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:39.476819    5002 cpu_manager.go:414] "containerMap: removing container" podUID="b3e31ee3-d2eb-42b1-84dc-173ed4434f84" containerName="main"
Oct 30 01:03:39.478797 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:39.476823    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3e31ee3-d2eb-42b1-84dc-173ed4434f84" containerName="main"
Oct 30 01:03:39.478797 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:39.476943    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="00e1797d-0629-4b79-8073-eaa093f48f2a" containerName="istio-proxy"
Oct 30 01:03:39.478797 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:39.476954    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8cef991-175d-4b30-8d0b-33f05e92a992" containerName="revoke-keys"
Oct 30 01:03:39.478797 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:39.476961    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f9d2df7-6b86-4f0a-8f75-9a3d54f09aed" containerName="revoke-keys"
Oct 30 01:03:39.478797 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:39.476969    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c20eb99-f057-44bd-94e2-1972b67aa7f0" containerName="istio-proxy"
Oct 30 01:03:39.478797 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:39.476979    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="543ddf6f-c221-48c0-b5d5-881c23397832" containerName="istio-proxy"
Oct 30 01:03:39.478797 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:39.476985    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3cd6ac5-b9a0-4096-94ee-851e2042e9fa" containerName="main"
Oct 30 01:03:39.478797 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:39.476993    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3e31ee3-d2eb-42b1-84dc-173ed4434f84" containerName="main"
Oct 30 01:03:39.486524 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:39.485149    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8dbe73-wj7p4"
Oct 30 01:03:39.486524 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:39.485045    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8dbe73-wj7p4"]
Oct 30 01:03:39.502507 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:39.502479    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:03:39.502685 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:39.502516    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5phsh\""
Oct 30 01:03:39.507003 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:39.506977    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a58c0b7-f707-45fc-9c2c-38dd671fb592-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-8dbe73-wj7p4\" (UID: \"0a58c0b7-f707-45fc-9c2c-38dd671fb592\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8dbe73-wj7p4"
Oct 30 01:03:39.507271 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:39.507251    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bd4b8\" (UniqueName: \"kubernetes.io/projected/0a58c0b7-f707-45fc-9c2c-38dd671fb592-kube-api-access-bd4b8\") pod \"maas-api-revoke-keys-e2e-inf-a-8dbe73-wj7p4\" (UID: \"0a58c0b7-f707-45fc-9c2c-38dd671fb592\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8dbe73-wj7p4"
Oct 30 01:03:39.507635 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:39.507609    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a58c0b7-f707-45fc-9c2c-38dd671fb592-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-8dbe73-wj7p4\" (UID: \"0a58c0b7-f707-45fc-9c2c-38dd671fb592\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8dbe73-wj7p4"
Oct 30 01:03:39.593504 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:39.592672    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:03:39.593504 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:39.592831    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:03:39.608907 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:39.608876    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a58c0b7-f707-45fc-9c2c-38dd671fb592-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-8dbe73-wj7p4\" (UID: \"0a58c0b7-f707-45fc-9c2c-38dd671fb592\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8dbe73-wj7p4"
Oct 30 01:03:39.609089 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:39.609029    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a58c0b7-f707-45fc-9c2c-38dd671fb592-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-8dbe73-wj7p4\" (UID: \"0a58c0b7-f707-45fc-9c2c-38dd671fb592\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8dbe73-wj7p4"
Oct 30 01:03:39.609159 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:39.609116    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bd4b8\" (UniqueName: \"kubernetes.io/projected/0a58c0b7-f707-45fc-9c2c-38dd671fb592-kube-api-access-bd4b8\") pod \"maas-api-revoke-keys-e2e-inf-a-8dbe73-wj7p4\" (UID: \"0a58c0b7-f707-45fc-9c2c-38dd671fb592\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8dbe73-wj7p4"
Oct 30 01:03:39.609717 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:39.609695    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a58c0b7-f707-45fc-9c2c-38dd671fb592-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-8dbe73-wj7p4\" (UID: \"0a58c0b7-f707-45fc-9c2c-38dd671fb592\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8dbe73-wj7p4"
Oct 30 01:03:39.611557 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:39.611534    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a58c0b7-f707-45fc-9c2c-38dd671fb592-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-8dbe73-wj7p4\" (UID: \"0a58c0b7-f707-45fc-9c2c-38dd671fb592\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8dbe73-wj7p4"
Oct 30 01:03:39.634214 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:39.634184    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bd4b8\" (UniqueName: \"kubernetes.io/projected/0a58c0b7-f707-45fc-9c2c-38dd671fb592-kube-api-access-bd4b8\") pod \"maas-api-revoke-keys-e2e-inf-a-8dbe73-wj7p4\" (UID: \"0a58c0b7-f707-45fc-9c2c-38dd671fb592\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8dbe73-wj7p4"
Oct 30 01:03:39.807272 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:39.807230    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8dbe73-wj7p4"
Oct 30 01:03:40.150948 ip-10-0-128-7 kubenswrapper[5002]: W1030 01:03:40.150911    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a58c0b7_f707_45fc_9c2c_38dd671fb592.slice/crio-ab8af99ea4ffb86417abcb076117e49bdc4c73c44e1eabd5cca764464606a1e6 WatchSource:0}: Error finding container ab8af99ea4ffb86417abcb076117e49bdc4c73c44e1eabd5cca764464606a1e6: Status 404 returned error can't find the container with id ab8af99ea4ffb86417abcb076117e49bdc4c73c44e1eabd5cca764464606a1e6
Oct 30 01:03:40.153127 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:40.153100    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8dbe73-wj7p4"]
Oct 30 01:03:40.307137 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:40.307093    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8dbe73-wj7p4" event={"ID":"0a58c0b7-f707-45fc-9c2c-38dd671fb592","Type":"ContainerStarted","Data":"ab8af99ea4ffb86417abcb076117e49bdc4c73c44e1eabd5cca764464606a1e6"}
Oct 30 01:03:40.498282 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:40.498244    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f9d2df7-6b86-4f0a-8f75-9a3d54f09aed" path="/var/lib/kubelet/pods/0f9d2df7-6b86-4f0a-8f75-9a3d54f09aed/volumes"
Oct 30 01:03:41.317088 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:41.317044    5002 generic.go:356] "Generic (PLEG): container finished" podID="0a58c0b7-f707-45fc-9c2c-38dd671fb592" containerID="8aa4f34cd77904f69d199495513315de890394b15aade7ecaf8c503e29ced588" exitCode=0
Oct 30 01:03:41.317577 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:41.317222    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8dbe73-wj7p4" event={"ID":"0a58c0b7-f707-45fc-9c2c-38dd671fb592","Type":"ContainerDied","Data":"8aa4f34cd77904f69d199495513315de890394b15aade7ecaf8c503e29ced588"}
Oct 30 01:03:42.521670 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:42.521638    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8dbe73-wj7p4"
Oct 30 01:03:42.555908 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:42.555875    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a58c0b7-f707-45fc-9c2c-38dd671fb592-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a58c0b7-f707-45fc-9c2c-38dd671fb592-maas-api-openshift-ca\") pod \"0a58c0b7-f707-45fc-9c2c-38dd671fb592\" (UID: \"0a58c0b7-f707-45fc-9c2c-38dd671fb592\") "
Oct 30 01:03:42.556100 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:42.555945    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0a58c0b7-f707-45fc-9c2c-38dd671fb592-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a58c0b7-f707-45fc-9c2c-38dd671fb592-maas-api-xks-ca\") pod \"0a58c0b7-f707-45fc-9c2c-38dd671fb592\" (UID: \"0a58c0b7-f707-45fc-9c2c-38dd671fb592\") "
Oct 30 01:03:42.556239 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:42.556207    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a58c0b7-f707-45fc-9c2c-38dd671fb592-kube-api-access-bd4b8\" (UniqueName: \"kubernetes.io/projected/0a58c0b7-f707-45fc-9c2c-38dd671fb592-kube-api-access-bd4b8\") pod \"0a58c0b7-f707-45fc-9c2c-38dd671fb592\" (UID: \"0a58c0b7-f707-45fc-9c2c-38dd671fb592\") "
Oct 30 01:03:42.556305 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:42.556259    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a58c0b7-f707-45fc-9c2c-38dd671fb592-maas-api-openshift-ca" pod "0a58c0b7-f707-45fc-9c2c-38dd671fb592" (UID: "0a58c0b7-f707-45fc-9c2c-38dd671fb592"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:03:42.557461 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:42.557435    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a58c0b7-f707-45fc-9c2c-38dd671fb592-maas-api-openshift-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:03:42.559098 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:42.559065    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a58c0b7-f707-45fc-9c2c-38dd671fb592-kube-api-access-bd4b8" pod "0a58c0b7-f707-45fc-9c2c-38dd671fb592" (UID: "0a58c0b7-f707-45fc-9c2c-38dd671fb592"). InnerVolumeSpecName "kube-api-access-bd4b8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:03:42.559536 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:42.559337    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a58c0b7-f707-45fc-9c2c-38dd671fb592-maas-api-xks-ca" pod "0a58c0b7-f707-45fc-9c2c-38dd671fb592" (UID: "0a58c0b7-f707-45fc-9c2c-38dd671fb592"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:03:42.659355 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:42.659254    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bd4b8\" (UniqueName: \"kubernetes.io/projected/0a58c0b7-f707-45fc-9c2c-38dd671fb592-kube-api-access-bd4b8\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:03:42.659355 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:42.659289    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a58c0b7-f707-45fc-9c2c-38dd671fb592-maas-api-xks-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:03:42.866183 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:03:42.866140    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbjdq" containerName="extractor"
Oct 30 01:03:42.900993 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:42.900949    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f874e14-m6h2d"]
Oct 30 01:03:42.901775 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:42.901752    5002 cpu_manager.go:414] "containerMap: removing container" podUID="0a58c0b7-f707-45fc-9c2c-38dd671fb592" containerName="revoke-keys"
Oct 30 01:03:42.901775 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:42.901772    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a58c0b7-f707-45fc-9c2c-38dd671fb592" containerName="revoke-keys"
Oct 30 01:03:42.901957 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:42.901891    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a58c0b7-f707-45fc-9c2c-38dd671fb592" containerName="revoke-keys"
Oct 30 01:03:42.905566 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:42.905547    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f874e14-m6h2d"
Oct 30 01:03:42.923168 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:42.923142    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f874e14-m6h2d"]
Oct 30 01:03:42.964308 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:42.964065    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f879b587-9122-48b1-bca2-cecb3d22067e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2f874e14-m6h2d\" (UID: \"f879b587-9122-48b1-bca2-cecb3d22067e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f874e14-m6h2d"
Oct 30 01:03:42.964308 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:42.964206    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pxvk\" (UniqueName: \"kubernetes.io/projected/f879b587-9122-48b1-bca2-cecb3d22067e-kube-api-access-6pxvk\") pod \"maas-api-revoke-sub-2f874e14-m6h2d\" (UID: \"f879b587-9122-48b1-bca2-cecb3d22067e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f874e14-m6h2d"
Oct 30 01:03:42.965569 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:42.964435    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f879b587-9122-48b1-bca2-cecb3d22067e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2f874e14-m6h2d\" (UID: \"f879b587-9122-48b1-bca2-cecb3d22067e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f874e14-m6h2d"
Oct 30 01:03:43.066198 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:43.066162    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f879b587-9122-48b1-bca2-cecb3d22067e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2f874e14-m6h2d\" (UID: \"f879b587-9122-48b1-bca2-cecb3d22067e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f874e14-m6h2d"
Oct 30 01:03:43.066445 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:43.066276    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f879b587-9122-48b1-bca2-cecb3d22067e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2f874e14-m6h2d\" (UID: \"f879b587-9122-48b1-bca2-cecb3d22067e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f874e14-m6h2d"
Oct 30 01:03:43.066445 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:43.066370    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pxvk\" (UniqueName: \"kubernetes.io/projected/f879b587-9122-48b1-bca2-cecb3d22067e-kube-api-access-6pxvk\") pod \"maas-api-revoke-sub-2f874e14-m6h2d\" (UID: \"f879b587-9122-48b1-bca2-cecb3d22067e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f874e14-m6h2d"
Oct 30 01:03:43.067053 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:43.067027    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f879b587-9122-48b1-bca2-cecb3d22067e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2f874e14-m6h2d\" (UID: \"f879b587-9122-48b1-bca2-cecb3d22067e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f874e14-m6h2d"
Oct 30 01:03:43.068984 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:43.068957    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f879b587-9122-48b1-bca2-cecb3d22067e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2f874e14-m6h2d\" (UID: \"f879b587-9122-48b1-bca2-cecb3d22067e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f874e14-m6h2d"
Oct 30 01:03:43.093664 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:43.093635    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pxvk\" (UniqueName: \"kubernetes.io/projected/f879b587-9122-48b1-bca2-cecb3d22067e-kube-api-access-6pxvk\") pod \"maas-api-revoke-sub-2f874e14-m6h2d\" (UID: \"f879b587-9122-48b1-bca2-cecb3d22067e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f874e14-m6h2d"
Oct 30 01:03:43.224590 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:43.223235    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f874e14-m6h2d"
Oct 30 01:03:43.338549 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:43.338513    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8dbe73-wj7p4" event={"ID":"0a58c0b7-f707-45fc-9c2c-38dd671fb592","Type":"ContainerDied","Data":"ab8af99ea4ffb86417abcb076117e49bdc4c73c44e1eabd5cca764464606a1e6"}
Oct 30 01:03:43.338720 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:43.338557    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8dbe73-wj7p4"
Oct 30 01:03:43.338720 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:43.338569    5002 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ab8af99ea4ffb86417abcb076117e49bdc4c73c44e1eabd5cca764464606a1e6"
Oct 30 01:03:43.556329 ip-10-0-128-7 kubenswrapper[5002]: W1030 01:03:43.556298    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf879b587_9122_48b1_bca2_cecb3d22067e.slice/crio-34bc1ed483482aea0f1bb5b5684477d8f14fbec59b02710360d048f46822c8dc WatchSource:0}: Error finding container 34bc1ed483482aea0f1bb5b5684477d8f14fbec59b02710360d048f46822c8dc: Status 404 returned error can't find the container with id 34bc1ed483482aea0f1bb5b5684477d8f14fbec59b02710360d048f46822c8dc
Oct 30 01:03:43.558718 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:43.558688    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f874e14-m6h2d"]
Oct 30 01:03:44.349027 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:44.348937    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f874e14-m6h2d" event={"ID":"f879b587-9122-48b1-bca2-cecb3d22067e","Type":"ContainerStarted","Data":"1e7b12d93baea8c06d9e54fc2de5d6d472a12460c643a242b77a4099c94600f3"}
Oct 30 01:03:44.349027 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:44.348998    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f874e14-m6h2d" event={"ID":"f879b587-9122-48b1-bca2-cecb3d22067e","Type":"ContainerStarted","Data":"34bc1ed483482aea0f1bb5b5684477d8f14fbec59b02710360d048f46822c8dc"}
Oct 30 01:03:44.397297 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:44.397232    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f874e14-m6h2d" podStartSLOduration=2.397212652 podStartE2EDuration="2.397212652s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:03:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:03:44.391374216 +0000 UTC m=+1920.007807096" watchObservedRunningTime="2026-10-30 01:03:44.397212652 +0000 UTC m=+1920.013645532"
Oct 30 01:03:45.359975 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:45.359940    5002 generic.go:356] "Generic (PLEG): container finished" podID="f879b587-9122-48b1-bca2-cecb3d22067e" containerID="1e7b12d93baea8c06d9e54fc2de5d6d472a12460c643a242b77a4099c94600f3" exitCode=0
Oct 30 01:03:45.360543 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:45.360017    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f874e14-m6h2d" event={"ID":"f879b587-9122-48b1-bca2-cecb3d22067e","Type":"ContainerDied","Data":"1e7b12d93baea8c06d9e54fc2de5d6d472a12460c643a242b77a4099c94600f3"}
Oct 30 01:03:46.658410 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:46.656282    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f874e14-m6h2d"
Oct 30 01:03:46.735074 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:46.735022    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f879b587-9122-48b1-bca2-cecb3d22067e-kube-api-access-6pxvk\" (UniqueName: \"kubernetes.io/projected/f879b587-9122-48b1-bca2-cecb3d22067e-kube-api-access-6pxvk\") pod \"f879b587-9122-48b1-bca2-cecb3d22067e\" (UID: \"f879b587-9122-48b1-bca2-cecb3d22067e\") "
Oct 30 01:03:46.735623 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:46.735545    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f879b587-9122-48b1-bca2-cecb3d22067e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f879b587-9122-48b1-bca2-cecb3d22067e-maas-api-openshift-ca\") pod \"f879b587-9122-48b1-bca2-cecb3d22067e\" (UID: \"f879b587-9122-48b1-bca2-cecb3d22067e\") "
Oct 30 01:03:46.735928 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:46.735907    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f879b587-9122-48b1-bca2-cecb3d22067e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f879b587-9122-48b1-bca2-cecb3d22067e-maas-api-xks-ca\") pod \"f879b587-9122-48b1-bca2-cecb3d22067e\" (UID: \"f879b587-9122-48b1-bca2-cecb3d22067e\") "
Oct 30 01:03:46.738567 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:46.738543    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f879b587-9122-48b1-bca2-cecb3d22067e-maas-api-openshift-ca" pod "f879b587-9122-48b1-bca2-cecb3d22067e" (UID: "f879b587-9122-48b1-bca2-cecb3d22067e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:03:46.740747 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:46.740725    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f879b587-9122-48b1-bca2-cecb3d22067e-maas-api-xks-ca" pod "f879b587-9122-48b1-bca2-cecb3d22067e" (UID: "f879b587-9122-48b1-bca2-cecb3d22067e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:03:46.740996 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:46.740976    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f879b587-9122-48b1-bca2-cecb3d22067e-kube-api-access-6pxvk" pod "f879b587-9122-48b1-bca2-cecb3d22067e" (UID: "f879b587-9122-48b1-bca2-cecb3d22067e"). InnerVolumeSpecName "kube-api-access-6pxvk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:03:46.843852 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:46.843792    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f879b587-9122-48b1-bca2-cecb3d22067e-maas-api-openshift-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:03:46.844163 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:46.844139    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f879b587-9122-48b1-bca2-cecb3d22067e-maas-api-xks-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:03:46.844412 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:46.844370    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6pxvk\" (UniqueName: \"kubernetes.io/projected/f879b587-9122-48b1-bca2-cecb3d22067e-kube-api-access-6pxvk\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:03:47.257839 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:47.257787    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-03d892-x4vt9"]
Oct 30 01:03:47.258988 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:47.258958    5002 cpu_manager.go:414] "containerMap: removing container" podUID="f879b587-9122-48b1-bca2-cecb3d22067e" containerName="revoke-keys"
Oct 30 01:03:47.258988 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:47.258985    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f879b587-9122-48b1-bca2-cecb3d22067e" containerName="revoke-keys"
Oct 30 01:03:47.259451 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:47.259426    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="f879b587-9122-48b1-bca2-cecb3d22067e" containerName="revoke-keys"
Oct 30 01:03:47.263849 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:47.263826    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-03d892-x4vt9"
Oct 30 01:03:47.277495 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:47.277334    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-03d892-x4vt9"]
Oct 30 01:03:47.312864 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:47.312821    5002 scope.go:122] "RemoveContainer" containerID="2d20eef6aad7b011766f73825c68964c489c598cbcf672010ff9475ac569a9de"
Oct 30 01:03:47.324545 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:47.324521    5002 scope.go:122] "RemoveContainer" containerID="924f68c091f4b8ac7157a073873b7ba1d194ba0e4eb23252c60a9ff8c07d5fc9"
Oct 30 01:03:47.352199 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:47.352165    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/304bba5e-2064-4901-bc87-4a770f57e5f9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-03d892-x4vt9\" (UID: \"304bba5e-2064-4901-bc87-4a770f57e5f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-03d892-x4vt9"
Oct 30 01:03:47.352320 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:47.352266    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5d6z\" (UniqueName: \"kubernetes.io/projected/304bba5e-2064-4901-bc87-4a770f57e5f9-kube-api-access-k5d6z\") pod \"maas-api-revoke-keys-e2e-inf-b-03d892-x4vt9\" (UID: \"304bba5e-2064-4901-bc87-4a770f57e5f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-03d892-x4vt9"
Oct 30 01:03:47.352320 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:47.352293    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/304bba5e-2064-4901-bc87-4a770f57e5f9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-03d892-x4vt9\" (UID: \"304bba5e-2064-4901-bc87-4a770f57e5f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-03d892-x4vt9"
Oct 30 01:03:47.381589 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:47.381552    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f874e14-m6h2d" event={"ID":"f879b587-9122-48b1-bca2-cecb3d22067e","Type":"ContainerDied","Data":"34bc1ed483482aea0f1bb5b5684477d8f14fbec59b02710360d048f46822c8dc"}
Oct 30 01:03:47.381694 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:47.381619    5002 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="34bc1ed483482aea0f1bb5b5684477d8f14fbec59b02710360d048f46822c8dc"
Oct 30 01:03:47.381733 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:47.381696    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f874e14-m6h2d"
Oct 30 01:03:47.454048 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:47.454006    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/304bba5e-2064-4901-bc87-4a770f57e5f9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-03d892-x4vt9\" (UID: \"304bba5e-2064-4901-bc87-4a770f57e5f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-03d892-x4vt9"
Oct 30 01:03:47.454243 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:47.454101    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5d6z\" (UniqueName: \"kubernetes.io/projected/304bba5e-2064-4901-bc87-4a770f57e5f9-kube-api-access-k5d6z\") pod \"maas-api-revoke-keys-e2e-inf-b-03d892-x4vt9\" (UID: \"304bba5e-2064-4901-bc87-4a770f57e5f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-03d892-x4vt9"
Oct 30 01:03:47.454243 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:47.454136    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/304bba5e-2064-4901-bc87-4a770f57e5f9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-03d892-x4vt9\" (UID: \"304bba5e-2064-4901-bc87-4a770f57e5f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-03d892-x4vt9"
Oct 30 01:03:47.454845 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:47.454818    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/304bba5e-2064-4901-bc87-4a770f57e5f9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-03d892-x4vt9\" (UID: \"304bba5e-2064-4901-bc87-4a770f57e5f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-03d892-x4vt9"
Oct 30 01:03:47.457156 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:47.457122    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/304bba5e-2064-4901-bc87-4a770f57e5f9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-03d892-x4vt9\" (UID: \"304bba5e-2064-4901-bc87-4a770f57e5f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-03d892-x4vt9"
Oct 30 01:03:47.479511 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:47.479469    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5d6z\" (UniqueName: \"kubernetes.io/projected/304bba5e-2064-4901-bc87-4a770f57e5f9-kube-api-access-k5d6z\") pod \"maas-api-revoke-keys-e2e-inf-b-03d892-x4vt9\" (UID: \"304bba5e-2064-4901-bc87-4a770f57e5f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-03d892-x4vt9"
Oct 30 01:03:47.600861 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:47.600756    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-03d892-x4vt9"
Oct 30 01:03:47.918693 ip-10-0-128-7 kubenswrapper[5002]: W1030 01:03:47.918665    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod304bba5e_2064_4901_bc87_4a770f57e5f9.slice/crio-0c64ae454768a93a6a58e212de19daa0f62be955fa2cbdb4945953906801fd46 WatchSource:0}: Error finding container 0c64ae454768a93a6a58e212de19daa0f62be955fa2cbdb4945953906801fd46: Status 404 returned error can't find the container with id 0c64ae454768a93a6a58e212de19daa0f62be955fa2cbdb4945953906801fd46
Oct 30 01:03:47.921015 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:47.920987    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-03d892-x4vt9"]
Oct 30 01:03:48.393015 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:48.392966    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-03d892-x4vt9" event={"ID":"304bba5e-2064-4901-bc87-4a770f57e5f9","Type":"ContainerStarted","Data":"0c64ae454768a93a6a58e212de19daa0f62be955fa2cbdb4945953906801fd46"}
Oct 30 01:03:49.402674 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:49.402590    5002 generic.go:356] "Generic (PLEG): container finished" podID="304bba5e-2064-4901-bc87-4a770f57e5f9" containerID="c45dd3879f6956061f289a74598e4b795875b85985bd04394cb62d73e3f7687e" exitCode=0
Oct 30 01:03:49.403051 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:49.402671    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-03d892-x4vt9" event={"ID":"304bba5e-2064-4901-bc87-4a770f57e5f9","Type":"ContainerDied","Data":"c45dd3879f6956061f289a74598e4b795875b85985bd04394cb62d73e3f7687e"}
Oct 30 01:03:49.816522 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:49.816482    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-8dbe73-698f74c77d-l7tpx"]
Oct 30 01:03:49.851484 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:49.851446    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-8dbe73-788cc7d8c9-qnxcz"]
Oct 30 01:03:50.830450 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:50.830425    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-03d892-x4vt9"
Oct 30 01:03:50.961404 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:50.959655    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/304bba5e-2064-4901-bc87-4a770f57e5f9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/304bba5e-2064-4901-bc87-4a770f57e5f9-maas-api-xks-ca\") pod \"304bba5e-2064-4901-bc87-4a770f57e5f9\" (UID: \"304bba5e-2064-4901-bc87-4a770f57e5f9\") "
Oct 30 01:03:50.961404 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:50.959800    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/304bba5e-2064-4901-bc87-4a770f57e5f9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/304bba5e-2064-4901-bc87-4a770f57e5f9-maas-api-openshift-ca\") pod \"304bba5e-2064-4901-bc87-4a770f57e5f9\" (UID: \"304bba5e-2064-4901-bc87-4a770f57e5f9\") "
Oct 30 01:03:50.961404 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:50.959854    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/304bba5e-2064-4901-bc87-4a770f57e5f9-kube-api-access-k5d6z\" (UniqueName: \"kubernetes.io/projected/304bba5e-2064-4901-bc87-4a770f57e5f9-kube-api-access-k5d6z\") pod \"304bba5e-2064-4901-bc87-4a770f57e5f9\" (UID: \"304bba5e-2064-4901-bc87-4a770f57e5f9\") "
Oct 30 01:03:50.970414 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:50.967035    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/304bba5e-2064-4901-bc87-4a770f57e5f9-kube-api-access-k5d6z" pod "304bba5e-2064-4901-bc87-4a770f57e5f9" (UID: "304bba5e-2064-4901-bc87-4a770f57e5f9"). InnerVolumeSpecName "kube-api-access-k5d6z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:03:50.970414 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:50.967130    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/304bba5e-2064-4901-bc87-4a770f57e5f9-maas-api-xks-ca" pod "304bba5e-2064-4901-bc87-4a770f57e5f9" (UID: "304bba5e-2064-4901-bc87-4a770f57e5f9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:03:50.970414 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:50.967442    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/304bba5e-2064-4901-bc87-4a770f57e5f9-maas-api-openshift-ca" pod "304bba5e-2064-4901-bc87-4a770f57e5f9" (UID: "304bba5e-2064-4901-bc87-4a770f57e5f9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:03:51.061500 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:51.061410    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/304bba5e-2064-4901-bc87-4a770f57e5f9-maas-api-openshift-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:03:51.061500 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:51.061442    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k5d6z\" (UniqueName: \"kubernetes.io/projected/304bba5e-2064-4901-bc87-4a770f57e5f9-kube-api-access-k5d6z\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:03:51.061500 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:51.061456    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/304bba5e-2064-4901-bc87-4a770f57e5f9-maas-api-xks-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:03:51.479041 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:51.478994    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-03d892-x4vt9" event={"ID":"304bba5e-2064-4901-bc87-4a770f57e5f9","Type":"ContainerDied","Data":"0c64ae454768a93a6a58e212de19daa0f62be955fa2cbdb4945953906801fd46"}
Oct 30 01:03:51.479041 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:51.479020    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-03d892-x4vt9"
Oct 30 01:03:51.479301 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:51.479051    5002 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0c64ae454768a93a6a58e212de19daa0f62be955fa2cbdb4945953906801fd46"
Oct 30 01:03:53.266840 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:53.266798    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82b7e656-hb79m"]
Oct 30 01:03:53.267789 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:53.267767    5002 cpu_manager.go:414] "containerMap: removing container" podUID="304bba5e-2064-4901-bc87-4a770f57e5f9" containerName="revoke-keys"
Oct 30 01:03:53.267849 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:53.267792    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="304bba5e-2064-4901-bc87-4a770f57e5f9" containerName="revoke-keys"
Oct 30 01:03:53.268024 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:53.268012    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="304bba5e-2064-4901-bc87-4a770f57e5f9" containerName="revoke-keys"
Oct 30 01:03:53.272448 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:53.272423    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82b7e656-hb79m"
Oct 30 01:03:53.289657 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:53.289626    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82b7e656-hb79m"]
Oct 30 01:03:53.292459 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:53.292428    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xd72\" (UniqueName: \"kubernetes.io/projected/ccd498da-017b-4a42-a806-21b31d3fe326-kube-api-access-8xd72\") pod \"maas-api-revoke-sub-82b7e656-hb79m\" (UID: \"ccd498da-017b-4a42-a806-21b31d3fe326\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82b7e656-hb79m"
Oct 30 01:03:53.292600 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:53.292495    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ccd498da-017b-4a42-a806-21b31d3fe326-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-82b7e656-hb79m\" (UID: \"ccd498da-017b-4a42-a806-21b31d3fe326\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82b7e656-hb79m"
Oct 30 01:03:53.292660 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:53.292617    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ccd498da-017b-4a42-a806-21b31d3fe326-maas-api-xks-ca\") pod \"maas-api-revoke-sub-82b7e656-hb79m\" (UID: \"ccd498da-017b-4a42-a806-21b31d3fe326\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82b7e656-hb79m"
Oct 30 01:03:53.298443 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:53.297475    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:03:53.298443 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:53.297537    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:03:53.298443 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:53.297623    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:03:53.302705 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:53.302681    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5phsh\""
Oct 30 01:03:53.393942 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:53.393896    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xd72\" (UniqueName: \"kubernetes.io/projected/ccd498da-017b-4a42-a806-21b31d3fe326-kube-api-access-8xd72\") pod \"maas-api-revoke-sub-82b7e656-hb79m\" (UID: \"ccd498da-017b-4a42-a806-21b31d3fe326\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82b7e656-hb79m"
Oct 30 01:03:53.394130 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:53.393961    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ccd498da-017b-4a42-a806-21b31d3fe326-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-82b7e656-hb79m\" (UID: \"ccd498da-017b-4a42-a806-21b31d3fe326\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82b7e656-hb79m"
Oct 30 01:03:53.394130 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:53.394039    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ccd498da-017b-4a42-a806-21b31d3fe326-maas-api-xks-ca\") pod \"maas-api-revoke-sub-82b7e656-hb79m\" (UID: \"ccd498da-017b-4a42-a806-21b31d3fe326\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82b7e656-hb79m"
Oct 30 01:03:53.394802 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:53.394775    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ccd498da-017b-4a42-a806-21b31d3fe326-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-82b7e656-hb79m\" (UID: \"ccd498da-017b-4a42-a806-21b31d3fe326\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82b7e656-hb79m"
Oct 30 01:03:53.396693 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:53.396664    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ccd498da-017b-4a42-a806-21b31d3fe326-maas-api-xks-ca\") pod \"maas-api-revoke-sub-82b7e656-hb79m\" (UID: \"ccd498da-017b-4a42-a806-21b31d3fe326\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82b7e656-hb79m"
Oct 30 01:03:53.420023 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:53.419997    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xd72\" (UniqueName: \"kubernetes.io/projected/ccd498da-017b-4a42-a806-21b31d3fe326-kube-api-access-8xd72\") pod \"maas-api-revoke-sub-82b7e656-hb79m\" (UID: \"ccd498da-017b-4a42-a806-21b31d3fe326\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82b7e656-hb79m"
Oct 30 01:03:53.590302 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:53.590205    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82b7e656-hb79m"
Oct 30 01:03:53.895509 ip-10-0-128-7 kubenswrapper[5002]: W1030 01:03:53.895464    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccd498da_017b_4a42_a806_21b31d3fe326.slice/crio-0cdf983afda007a1aa95ec3ec9c65d2e544ce67ed0c3e4fb1384307faeb0b865 WatchSource:0}: Error finding container 0cdf983afda007a1aa95ec3ec9c65d2e544ce67ed0c3e4fb1384307faeb0b865: Status 404 returned error can't find the container with id 0cdf983afda007a1aa95ec3ec9c65d2e544ce67ed0c3e4fb1384307faeb0b865
Oct 30 01:03:53.898269 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:53.898236    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82b7e656-hb79m"]
Oct 30 01:03:54.325729 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:54.325691    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-84b89f/test-model-84b89f-kserve-d8c475485-5prlt" podUID="00a3e39a-01f0-4d98-9350-aa0b17a6c434" containerName="main" containerID="cri-o://09c94c5020da7b32f56961507c85a26041e42f9685c73aa2348c2fc0bb9e0e22" gracePeriod=45
Oct 30 01:03:54.429956 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:54.429915    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-84b89f/test-model-84b89f-kserve-d8c475485-5prlt" podUID="00a3e39a-01f0-4d98-9350-aa0b17a6c434" containerName="main" probeResult="failure" output="Get \"https://10.132.0.138:8000/ready\": dial tcp 10.132.0.138:8000: connect: connection refused"
Oct 30 01:03:54.534251 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:54.534158    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82b7e656-hb79m" event={"ID":"ccd498da-017b-4a42-a806-21b31d3fe326","Type":"ContainerStarted","Data":"3573151956bf14c3538772080527f3d46f8b4b787b1743248e4e500c1dbdb5ff"}
Oct 30 01:03:54.534251 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:54.534229    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82b7e656-hb79m" event={"ID":"ccd498da-017b-4a42-a806-21b31d3fe326","Type":"ContainerStarted","Data":"0cdf983afda007a1aa95ec3ec9c65d2e544ce67ed0c3e4fb1384307faeb0b865"}
Oct 30 01:03:54.537164 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:54.537139    5002 generic.go:356] "Generic (PLEG): container finished" podID="00a3e39a-01f0-4d98-9350-aa0b17a6c434" containerID="09c94c5020da7b32f56961507c85a26041e42f9685c73aa2348c2fc0bb9e0e22" exitCode=0
Oct 30 01:03:54.537280 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:54.537182    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-84b89f/test-model-84b89f-kserve-d8c475485-5prlt" event={"ID":"00a3e39a-01f0-4d98-9350-aa0b17a6c434","Type":"ContainerDied","Data":"09c94c5020da7b32f56961507c85a26041e42f9685c73aa2348c2fc0bb9e0e22"}
Oct 30 01:03:54.580866 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:54.580725    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82b7e656-hb79m" podStartSLOduration=1.580705912 podStartE2EDuration="1.580705912s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:03:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:03:54.577994691 +0000 UTC m=+1930.194427572" watchObservedRunningTime="2026-10-30 01:03:54.580705912 +0000 UTC m=+1930.197138791"
Oct 30 01:03:54.687877 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:54.687845    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-84b89f/test-model-84b89f-kserve-d8c475485-5prlt"
Oct 30 01:03:54.714922 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:54.714886    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00a3e39a-01f0-4d98-9350-aa0b17a6c434-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00a3e39a-01f0-4d98-9350-aa0b17a6c434-tmp-dir\") pod \"00a3e39a-01f0-4d98-9350-aa0b17a6c434\" (UID: \"00a3e39a-01f0-4d98-9350-aa0b17a6c434\") "
Oct 30 01:03:54.715085 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:54.715015    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00a3e39a-01f0-4d98-9350-aa0b17a6c434-kube-api-access-k5n62\" (UniqueName: \"kubernetes.io/projected/00a3e39a-01f0-4d98-9350-aa0b17a6c434-kube-api-access-k5n62\") pod \"00a3e39a-01f0-4d98-9350-aa0b17a6c434\" (UID: \"00a3e39a-01f0-4d98-9350-aa0b17a6c434\") "
Oct 30 01:03:54.715171 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:54.715154    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00a3e39a-01f0-4d98-9350-aa0b17a6c434-home\" (UniqueName: \"kubernetes.io/empty-dir/00a3e39a-01f0-4d98-9350-aa0b17a6c434-home\") pod \"00a3e39a-01f0-4d98-9350-aa0b17a6c434\" (UID: \"00a3e39a-01f0-4d98-9350-aa0b17a6c434\") "
Oct 30 01:03:54.715228 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:54.715187    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00a3e39a-01f0-4d98-9350-aa0b17a6c434-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00a3e39a-01f0-4d98-9350-aa0b17a6c434-model-cache\") pod \"00a3e39a-01f0-4d98-9350-aa0b17a6c434\" (UID: \"00a3e39a-01f0-4d98-9350-aa0b17a6c434\") "
Oct 30 01:03:54.715228 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:54.715191    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00a3e39a-01f0-4d98-9350-aa0b17a6c434-tmp-dir" pod "00a3e39a-01f0-4d98-9350-aa0b17a6c434" (UID: "00a3e39a-01f0-4d98-9350-aa0b17a6c434"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:03:54.715228 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:54.715211    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/00a3e39a-01f0-4d98-9350-aa0b17a6c434-tls-certs\" (UniqueName: \"kubernetes.io/secret/00a3e39a-01f0-4d98-9350-aa0b17a6c434-tls-certs\") pod \"00a3e39a-01f0-4d98-9350-aa0b17a6c434\" (UID: \"00a3e39a-01f0-4d98-9350-aa0b17a6c434\") "
Oct 30 01:03:54.715434 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:54.715237    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00a3e39a-01f0-4d98-9350-aa0b17a6c434-dshm\" (UniqueName: \"kubernetes.io/empty-dir/00a3e39a-01f0-4d98-9350-aa0b17a6c434-dshm\") pod \"00a3e39a-01f0-4d98-9350-aa0b17a6c434\" (UID: \"00a3e39a-01f0-4d98-9350-aa0b17a6c434\") "
Oct 30 01:03:54.715434 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:54.715375    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00a3e39a-01f0-4d98-9350-aa0b17a6c434-model-cache" pod "00a3e39a-01f0-4d98-9350-aa0b17a6c434" (UID: "00a3e39a-01f0-4d98-9350-aa0b17a6c434"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:03:54.715732 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:54.715704    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00a3e39a-01f0-4d98-9350-aa0b17a6c434-home" pod "00a3e39a-01f0-4d98-9350-aa0b17a6c434" (UID: "00a3e39a-01f0-4d98-9350-aa0b17a6c434"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:03:54.715946 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:54.715927    5002 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/00a3e39a-01f0-4d98-9350-aa0b17a6c434-home\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:03:54.716020 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:54.715952    5002 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00a3e39a-01f0-4d98-9350-aa0b17a6c434-model-cache\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:03:54.716020 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:54.715967    5002 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00a3e39a-01f0-4d98-9350-aa0b17a6c434-tmp-dir\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:03:54.717531 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:54.717498    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00a3e39a-01f0-4d98-9350-aa0b17a6c434-kube-api-access-k5n62" pod "00a3e39a-01f0-4d98-9350-aa0b17a6c434" (UID: "00a3e39a-01f0-4d98-9350-aa0b17a6c434"). InnerVolumeSpecName "kube-api-access-k5n62". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:03:54.717947 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:54.717922    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00a3e39a-01f0-4d98-9350-aa0b17a6c434-dshm" pod "00a3e39a-01f0-4d98-9350-aa0b17a6c434" (UID: "00a3e39a-01f0-4d98-9350-aa0b17a6c434"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:03:54.718028 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:54.717944    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00a3e39a-01f0-4d98-9350-aa0b17a6c434-tls-certs" pod "00a3e39a-01f0-4d98-9350-aa0b17a6c434" (UID: "00a3e39a-01f0-4d98-9350-aa0b17a6c434"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:03:54.817403 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:54.817292    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k5n62\" (UniqueName: \"kubernetes.io/projected/00a3e39a-01f0-4d98-9350-aa0b17a6c434-kube-api-access-k5n62\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:03:54.817403 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:54.817326    5002 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00a3e39a-01f0-4d98-9350-aa0b17a6c434-tls-certs\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:03:54.817403 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:54.817335    5002 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/00a3e39a-01f0-4d98-9350-aa0b17a6c434-dshm\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:03:54.817630 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:54.817367    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-8dbe73-698f74c77d-l7tpx" podUID="dd1c2b8c-8b70-4583-bdec-d1901c0bd640" containerName="payload-processing" containerID="cri-o://a21e598f815033054af077d05ebe0bea9f35846b54db470069f55dece4d9daa4" gracePeriod=25
Oct 30 01:03:54.852147 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:54.852095    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8dbe73-788cc7d8c9-qnxcz" podUID="4521f1a9-13c3-409b-b0f3-d636722108a5" containerName="payload-pre-processing" containerID="cri-o://40402337a9a3a2dd5bcc1186f43b22c64efbb7956c12a4eddbfc2a1381888920" gracePeriod=25
Oct 30 01:03:55.546712 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:55.546679    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-84b89f/test-model-84b89f-kserve-d8c475485-5prlt"
Oct 30 01:03:55.547140 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:55.546716    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-84b89f/test-model-84b89f-kserve-d8c475485-5prlt" event={"ID":"00a3e39a-01f0-4d98-9350-aa0b17a6c434","Type":"ContainerDied","Data":"fefc066dec24ba55b9133855ddb925c8d54386a2e8b68b8fddf18ea6e733d6c2"}
Oct 30 01:03:55.547140 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:55.546779    5002 scope.go:122] "RemoveContainer" containerID="09c94c5020da7b32f56961507c85a26041e42f9685c73aa2348c2fc0bb9e0e22"
Oct 30 01:03:55.548774 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:55.548749    5002 generic.go:356] "Generic (PLEG): container finished" podID="ccd498da-017b-4a42-a806-21b31d3fe326" containerID="3573151956bf14c3538772080527f3d46f8b4b787b1743248e4e500c1dbdb5ff" exitCode=0
Oct 30 01:03:55.548908 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:55.548832    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82b7e656-hb79m" event={"ID":"ccd498da-017b-4a42-a806-21b31d3fe326","Type":"ContainerDied","Data":"3573151956bf14c3538772080527f3d46f8b4b787b1743248e4e500c1dbdb5ff"}
Oct 30 01:03:55.610646 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:55.610058    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-84b89f/test-model-84b89f-kserve-d8c475485-5prlt"]
Oct 30 01:03:55.620096 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:55.620062    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-84b89f/test-model-84b89f-kserve-d8c475485-5prlt"]
Oct 30 01:03:56.502055 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:56.501979    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00a3e39a-01f0-4d98-9350-aa0b17a6c434" path="/var/lib/kubelet/pods/00a3e39a-01f0-4d98-9350-aa0b17a6c434/volumes"
Oct 30 01:03:56.752572 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:56.752492    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82b7e656-hb79m"
Oct 30 01:03:56.847925 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:56.847871    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ccd498da-017b-4a42-a806-21b31d3fe326-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ccd498da-017b-4a42-a806-21b31d3fe326-maas-api-xks-ca\") pod \"ccd498da-017b-4a42-a806-21b31d3fe326\" (UID: \"ccd498da-017b-4a42-a806-21b31d3fe326\") "
Oct 30 01:03:56.848100 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:56.847948    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ccd498da-017b-4a42-a806-21b31d3fe326-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ccd498da-017b-4a42-a806-21b31d3fe326-maas-api-openshift-ca\") pod \"ccd498da-017b-4a42-a806-21b31d3fe326\" (UID: \"ccd498da-017b-4a42-a806-21b31d3fe326\") "
Oct 30 01:03:56.848227 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:56.848208    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ccd498da-017b-4a42-a806-21b31d3fe326-kube-api-access-8xd72\" (UniqueName: \"kubernetes.io/projected/ccd498da-017b-4a42-a806-21b31d3fe326-kube-api-access-8xd72\") pod \"ccd498da-017b-4a42-a806-21b31d3fe326\" (UID: \"ccd498da-017b-4a42-a806-21b31d3fe326\") "
Oct 30 01:03:56.848643 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:56.848611    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccd498da-017b-4a42-a806-21b31d3fe326-maas-api-openshift-ca" pod "ccd498da-017b-4a42-a806-21b31d3fe326" (UID: "ccd498da-017b-4a42-a806-21b31d3fe326"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:03:56.848918 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:56.848895    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ccd498da-017b-4a42-a806-21b31d3fe326-maas-api-openshift-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:03:56.860458 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:56.858941    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ccd498da-017b-4a42-a806-21b31d3fe326-maas-api-xks-ca" pod "ccd498da-017b-4a42-a806-21b31d3fe326" (UID: "ccd498da-017b-4a42-a806-21b31d3fe326"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:03:56.860458 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:56.859082    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccd498da-017b-4a42-a806-21b31d3fe326-kube-api-access-8xd72" pod "ccd498da-017b-4a42-a806-21b31d3fe326" (UID: "ccd498da-017b-4a42-a806-21b31d3fe326"). InnerVolumeSpecName "kube-api-access-8xd72". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:03:56.950945 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:56.950901    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ccd498da-017b-4a42-a806-21b31d3fe326-maas-api-xks-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:03:56.950945 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:56.950934    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8xd72\" (UniqueName: \"kubernetes.io/projected/ccd498da-017b-4a42-a806-21b31d3fe326-kube-api-access-8xd72\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:03:57.593840 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:57.593806    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82b7e656-hb79m" event={"ID":"ccd498da-017b-4a42-a806-21b31d3fe326","Type":"ContainerDied","Data":"0cdf983afda007a1aa95ec3ec9c65d2e544ce67ed0c3e4fb1384307faeb0b865"}
Oct 30 01:03:57.594040 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:57.593856    5002 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0cdf983afda007a1aa95ec3ec9c65d2e544ce67ed0c3e4fb1384307faeb0b865"
Oct 30 01:03:57.594040 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:57.593912    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82b7e656-hb79m"
Oct 30 01:03:58.192405 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:58.191490    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-8dbe73-698f74c77d-l7tpx" podUID="dd1c2b8c-8b70-4583-bdec-d1901c0bd640" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:03:58.192405 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:03:58.191858    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8dbe73-788cc7d8c9-qnxcz" podUID="4521f1a9-13c3-409b-b0f3-d636722108a5" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:04:01.681029 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:01.680991    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24080105-hzdnf"]
Oct 30 01:04:01.682017 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:01.681989    5002 cpu_manager.go:414] "containerMap: removing container" podUID="00a3e39a-01f0-4d98-9350-aa0b17a6c434" containerName="main"
Oct 30 01:04:01.682017 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:01.682014    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00a3e39a-01f0-4d98-9350-aa0b17a6c434" containerName="main"
Oct 30 01:04:01.682195 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:01.682051    5002 cpu_manager.go:414] "containerMap: removing container" podUID="ccd498da-017b-4a42-a806-21b31d3fe326" containerName="revoke-keys"
Oct 30 01:04:01.682195 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:01.682059    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ccd498da-017b-4a42-a806-21b31d3fe326" containerName="revoke-keys"
Oct 30 01:04:01.682283 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:01.682229    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="00a3e39a-01f0-4d98-9350-aa0b17a6c434" containerName="main"
Oct 30 01:04:01.682283 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:01.682246    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="ccd498da-017b-4a42-a806-21b31d3fe326" containerName="revoke-keys"
Oct 30 01:04:01.686861 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:01.686839    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24080105-hzdnf"
Oct 30 01:04:01.704173 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:01.704137    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24080105-hzdnf"]
Oct 30 01:04:01.712737 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:01.712706    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:04:01.712961 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:01.712940    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:04:01.718836 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:01.718814    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:04:01.719068 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:01.719048    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5phsh\""
Oct 30 01:04:01.834279 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:01.834232    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8adb29be-3746-4500-9889-ea465d532657-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-24080105-hzdnf\" (UID: \"8adb29be-3746-4500-9889-ea465d532657\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24080105-hzdnf"
Oct 30 01:04:01.834522 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:01.834358    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87wt9\" (UniqueName: \"kubernetes.io/projected/8adb29be-3746-4500-9889-ea465d532657-kube-api-access-87wt9\") pod \"maas-api-revoke-sub-24080105-hzdnf\" (UID: \"8adb29be-3746-4500-9889-ea465d532657\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24080105-hzdnf"
Oct 30 01:04:01.834601 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:01.834529    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8adb29be-3746-4500-9889-ea465d532657-maas-api-xks-ca\") pod \"maas-api-revoke-sub-24080105-hzdnf\" (UID: \"8adb29be-3746-4500-9889-ea465d532657\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24080105-hzdnf"
Oct 30 01:04:01.938464 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:01.938375    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8adb29be-3746-4500-9889-ea465d532657-maas-api-xks-ca\") pod \"maas-api-revoke-sub-24080105-hzdnf\" (UID: \"8adb29be-3746-4500-9889-ea465d532657\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24080105-hzdnf"
Oct 30 01:04:01.940600 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:01.940563    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8adb29be-3746-4500-9889-ea465d532657-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-24080105-hzdnf\" (UID: \"8adb29be-3746-4500-9889-ea465d532657\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24080105-hzdnf"
Oct 30 01:04:01.940898 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:01.940872    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87wt9\" (UniqueName: \"kubernetes.io/projected/8adb29be-3746-4500-9889-ea465d532657-kube-api-access-87wt9\") pod \"maas-api-revoke-sub-24080105-hzdnf\" (UID: \"8adb29be-3746-4500-9889-ea465d532657\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24080105-hzdnf"
Oct 30 01:04:01.941527 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:01.941504    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8adb29be-3746-4500-9889-ea465d532657-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-24080105-hzdnf\" (UID: \"8adb29be-3746-4500-9889-ea465d532657\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24080105-hzdnf"
Oct 30 01:04:01.943787 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:01.943762    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8adb29be-3746-4500-9889-ea465d532657-maas-api-xks-ca\") pod \"maas-api-revoke-sub-24080105-hzdnf\" (UID: \"8adb29be-3746-4500-9889-ea465d532657\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24080105-hzdnf"
Oct 30 01:04:01.966340 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:01.966316    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-87wt9\" (UniqueName: \"kubernetes.io/projected/8adb29be-3746-4500-9889-ea465d532657-kube-api-access-87wt9\") pod \"maas-api-revoke-sub-24080105-hzdnf\" (UID: \"8adb29be-3746-4500-9889-ea465d532657\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24080105-hzdnf"
Oct 30 01:04:02.003832 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:02.003806    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24080105-hzdnf"
Oct 30 01:04:02.310406 ip-10-0-128-7 kubenswrapper[5002]: W1030 01:04:02.310354    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8adb29be_3746_4500_9889_ea465d532657.slice/crio-1a31f32f6452f7e49c802e09e8285c3b013b896f40b6af0792c97e4615be035f WatchSource:0}: Error finding container 1a31f32f6452f7e49c802e09e8285c3b013b896f40b6af0792c97e4615be035f: Status 404 returned error can't find the container with id 1a31f32f6452f7e49c802e09e8285c3b013b896f40b6af0792c97e4615be035f
Oct 30 01:04:02.311299 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:02.311265    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24080105-hzdnf"]
Oct 30 01:04:02.633355 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:02.633315    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24080105-hzdnf" event={"ID":"8adb29be-3746-4500-9889-ea465d532657","Type":"ContainerStarted","Data":"dd192abbc69c3be96a2c91b78b7245eda34d0e5eba6a8f299080e86ae72a7c62"}
Oct 30 01:04:02.633658 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:02.633394    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24080105-hzdnf" event={"ID":"8adb29be-3746-4500-9889-ea465d532657","Type":"ContainerStarted","Data":"1a31f32f6452f7e49c802e09e8285c3b013b896f40b6af0792c97e4615be035f"}
Oct 30 01:04:02.681117 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:02.680462    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24080105-hzdnf" podStartSLOduration=1.6804445289999999 podStartE2EDuration="1.680444529s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:04:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:04:02.675988846 +0000 UTC m=+1938.292421748" watchObservedRunningTime="2026-10-30 01:04:02.680444529 +0000 UTC m=+1938.296877409"
Oct 30 01:04:03.657561 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:03.657521    5002 generic.go:356] "Generic (PLEG): container finished" podID="8adb29be-3746-4500-9889-ea465d532657" containerID="dd192abbc69c3be96a2c91b78b7245eda34d0e5eba6a8f299080e86ae72a7c62" exitCode=0
Oct 30 01:04:03.657778 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:03.657642    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24080105-hzdnf" event={"ID":"8adb29be-3746-4500-9889-ea465d532657","Type":"ContainerDied","Data":"dd192abbc69c3be96a2c91b78b7245eda34d0e5eba6a8f299080e86ae72a7c62"}
Oct 30 01:04:04.837720 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:04.837691    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24080105-hzdnf"
Oct 30 01:04:04.993105 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:04.993070    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8adb29be-3746-4500-9889-ea465d532657-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8adb29be-3746-4500-9889-ea465d532657-maas-api-xks-ca\") pod \"8adb29be-3746-4500-9889-ea465d532657\" (UID: \"8adb29be-3746-4500-9889-ea465d532657\") "
Oct 30 01:04:04.993316 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:04.993122    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8adb29be-3746-4500-9889-ea465d532657-kube-api-access-87wt9\" (UniqueName: \"kubernetes.io/projected/8adb29be-3746-4500-9889-ea465d532657-kube-api-access-87wt9\") pod \"8adb29be-3746-4500-9889-ea465d532657\" (UID: \"8adb29be-3746-4500-9889-ea465d532657\") "
Oct 30 01:04:04.993316 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:04.993294    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8adb29be-3746-4500-9889-ea465d532657-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8adb29be-3746-4500-9889-ea465d532657-maas-api-openshift-ca\") pod \"8adb29be-3746-4500-9889-ea465d532657\" (UID: \"8adb29be-3746-4500-9889-ea465d532657\") "
Oct 30 01:04:04.993654 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:04.993626    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8adb29be-3746-4500-9889-ea465d532657-maas-api-openshift-ca" pod "8adb29be-3746-4500-9889-ea465d532657" (UID: "8adb29be-3746-4500-9889-ea465d532657"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:04:04.993934 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:04.993914    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8adb29be-3746-4500-9889-ea465d532657-maas-api-openshift-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:04:04.995431 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:04.995398    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8adb29be-3746-4500-9889-ea465d532657-kube-api-access-87wt9" pod "8adb29be-3746-4500-9889-ea465d532657" (UID: "8adb29be-3746-4500-9889-ea465d532657"). InnerVolumeSpecName "kube-api-access-87wt9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:04:04.995648 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:04.995626    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8adb29be-3746-4500-9889-ea465d532657-maas-api-xks-ca" pod "8adb29be-3746-4500-9889-ea465d532657" (UID: "8adb29be-3746-4500-9889-ea465d532657"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:04:05.095613 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:05.095574    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8adb29be-3746-4500-9889-ea465d532657-maas-api-xks-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:04:05.095613 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:05.095614    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-87wt9\" (UniqueName: \"kubernetes.io/projected/8adb29be-3746-4500-9889-ea465d532657-kube-api-access-87wt9\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:04:05.675024 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:05.674990    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24080105-hzdnf"
Oct 30 01:04:05.675024 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:05.675012    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24080105-hzdnf" event={"ID":"8adb29be-3746-4500-9889-ea465d532657","Type":"ContainerDied","Data":"1a31f32f6452f7e49c802e09e8285c3b013b896f40b6af0792c97e4615be035f"}
Oct 30 01:04:05.675291 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:05.675064    5002 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1a31f32f6452f7e49c802e09e8285c3b013b896f40b6af0792c97e4615be035f"
Oct 30 01:04:08.171723 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:08.171677    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8dbe73-788cc7d8c9-qnxcz" podUID="4521f1a9-13c3-409b-b0f3-d636722108a5" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:04:08.172155 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:08.171919    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-8dbe73-698f74c77d-l7tpx" podUID="dd1c2b8c-8b70-4583-bdec-d1901c0bd640" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:04:08.481167 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:08.481122    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-ef1084/test-model-ef1084-kserve-6fb474f595-xm568"]
Oct 30 01:04:08.482101 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:08.482080    5002 cpu_manager.go:414] "containerMap: removing container" podUID="8adb29be-3746-4500-9889-ea465d532657" containerName="revoke-keys"
Oct 30 01:04:08.482101 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:08.482101    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8adb29be-3746-4500-9889-ea465d532657" containerName="revoke-keys"
Oct 30 01:04:08.482413 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:08.482393    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="8adb29be-3746-4500-9889-ea465d532657" containerName="revoke-keys"
Oct 30 01:04:08.486733 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:08.486713    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-ef1084/test-model-ef1084-kserve-6fb474f595-xm568"
Oct 30 01:04:08.509829 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:08.509798    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-ef1084/test-model-ef1084-kserve-6fb474f595-xm568"]
Oct 30 01:04:08.513698 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:08.513670    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-ef1084\"/\"default-dockercfg-72dqx\""
Oct 30 01:04:08.514008 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:08.513990    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-ef1084\"/\"kube-root-ca.crt\""
Oct 30 01:04:08.514221 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:08.514191    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-ef1084\"/\"openshift-service-ca.crt\""
Oct 30 01:04:08.521534 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:08.521513    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-ef1084\"/\"test-model-ef1084-kserve-self-signed-certs\""
Oct 30 01:04:08.560713 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:08.560667    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/652cd481-aaf4-4f3d-8532-978fe17201c8-model-cache\") pod \"test-model-ef1084-kserve-6fb474f595-xm568\" (UID: \"652cd481-aaf4-4f3d-8532-978fe17201c8\") " pod="ai-tenant-e2e-inf-a-ef1084/test-model-ef1084-kserve-6fb474f595-xm568"
Oct 30 01:04:08.560919 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:08.560768    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/652cd481-aaf4-4f3d-8532-978fe17201c8-tmp-dir\") pod \"test-model-ef1084-kserve-6fb474f595-xm568\" (UID: \"652cd481-aaf4-4f3d-8532-978fe17201c8\") " pod="ai-tenant-e2e-inf-a-ef1084/test-model-ef1084-kserve-6fb474f595-xm568"
Oct 30 01:04:08.561002 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:08.560924    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/652cd481-aaf4-4f3d-8532-978fe17201c8-dshm\") pod \"test-model-ef1084-kserve-6fb474f595-xm568\" (UID: \"652cd481-aaf4-4f3d-8532-978fe17201c8\") " pod="ai-tenant-e2e-inf-a-ef1084/test-model-ef1084-kserve-6fb474f595-xm568"
Oct 30 01:04:08.561002 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:08.560949    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/652cd481-aaf4-4f3d-8532-978fe17201c8-tls-certs\") pod \"test-model-ef1084-kserve-6fb474f595-xm568\" (UID: \"652cd481-aaf4-4f3d-8532-978fe17201c8\") " pod="ai-tenant-e2e-inf-a-ef1084/test-model-ef1084-kserve-6fb474f595-xm568"
Oct 30 01:04:08.561130 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:08.561105    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zh8v5\" (UniqueName: \"kubernetes.io/projected/652cd481-aaf4-4f3d-8532-978fe17201c8-kube-api-access-zh8v5\") pod \"test-model-ef1084-kserve-6fb474f595-xm568\" (UID: \"652cd481-aaf4-4f3d-8532-978fe17201c8\") " pod="ai-tenant-e2e-inf-a-ef1084/test-model-ef1084-kserve-6fb474f595-xm568"
Oct 30 01:04:08.561203 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:08.561186    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/652cd481-aaf4-4f3d-8532-978fe17201c8-home\") pod \"test-model-ef1084-kserve-6fb474f595-xm568\" (UID: \"652cd481-aaf4-4f3d-8532-978fe17201c8\") " pod="ai-tenant-e2e-inf-a-ef1084/test-model-ef1084-kserve-6fb474f595-xm568"
Oct 30 01:04:08.662776 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:08.662733    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/652cd481-aaf4-4f3d-8532-978fe17201c8-dshm\") pod \"test-model-ef1084-kserve-6fb474f595-xm568\" (UID: \"652cd481-aaf4-4f3d-8532-978fe17201c8\") " pod="ai-tenant-e2e-inf-a-ef1084/test-model-ef1084-kserve-6fb474f595-xm568"
Oct 30 01:04:08.662974 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:08.662788    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/652cd481-aaf4-4f3d-8532-978fe17201c8-tls-certs\") pod \"test-model-ef1084-kserve-6fb474f595-xm568\" (UID: \"652cd481-aaf4-4f3d-8532-978fe17201c8\") " pod="ai-tenant-e2e-inf-a-ef1084/test-model-ef1084-kserve-6fb474f595-xm568"
Oct 30 01:04:08.662974 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:08.662901    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zh8v5\" (UniqueName: \"kubernetes.io/projected/652cd481-aaf4-4f3d-8532-978fe17201c8-kube-api-access-zh8v5\") pod \"test-model-ef1084-kserve-6fb474f595-xm568\" (UID: \"652cd481-aaf4-4f3d-8532-978fe17201c8\") " pod="ai-tenant-e2e-inf-a-ef1084/test-model-ef1084-kserve-6fb474f595-xm568"
Oct 30 01:04:08.662974 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:08.662958    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/652cd481-aaf4-4f3d-8532-978fe17201c8-home\") pod \"test-model-ef1084-kserve-6fb474f595-xm568\" (UID: \"652cd481-aaf4-4f3d-8532-978fe17201c8\") " pod="ai-tenant-e2e-inf-a-ef1084/test-model-ef1084-kserve-6fb474f595-xm568"
Oct 30 01:04:08.663188 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:08.663168    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/652cd481-aaf4-4f3d-8532-978fe17201c8-model-cache\") pod \"test-model-ef1084-kserve-6fb474f595-xm568\" (UID: \"652cd481-aaf4-4f3d-8532-978fe17201c8\") " pod="ai-tenant-e2e-inf-a-ef1084/test-model-ef1084-kserve-6fb474f595-xm568"
Oct 30 01:04:08.663278 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:08.663212    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/652cd481-aaf4-4f3d-8532-978fe17201c8-tmp-dir\") pod \"test-model-ef1084-kserve-6fb474f595-xm568\" (UID: \"652cd481-aaf4-4f3d-8532-978fe17201c8\") " pod="ai-tenant-e2e-inf-a-ef1084/test-model-ef1084-kserve-6fb474f595-xm568"
Oct 30 01:04:08.663418 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:08.663350    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/652cd481-aaf4-4f3d-8532-978fe17201c8-home\") pod \"test-model-ef1084-kserve-6fb474f595-xm568\" (UID: \"652cd481-aaf4-4f3d-8532-978fe17201c8\") " pod="ai-tenant-e2e-inf-a-ef1084/test-model-ef1084-kserve-6fb474f595-xm568"
Oct 30 01:04:08.663622 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:08.663584    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/652cd481-aaf4-4f3d-8532-978fe17201c8-model-cache\") pod \"test-model-ef1084-kserve-6fb474f595-xm568\" (UID: \"652cd481-aaf4-4f3d-8532-978fe17201c8\") " pod="ai-tenant-e2e-inf-a-ef1084/test-model-ef1084-kserve-6fb474f595-xm568"
Oct 30 01:04:08.663703 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:08.663629    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/652cd481-aaf4-4f3d-8532-978fe17201c8-tmp-dir\") pod \"test-model-ef1084-kserve-6fb474f595-xm568\" (UID: \"652cd481-aaf4-4f3d-8532-978fe17201c8\") " pod="ai-tenant-e2e-inf-a-ef1084/test-model-ef1084-kserve-6fb474f595-xm568"
Oct 30 01:04:08.665406 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:08.665354    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/652cd481-aaf4-4f3d-8532-978fe17201c8-dshm\") pod \"test-model-ef1084-kserve-6fb474f595-xm568\" (UID: \"652cd481-aaf4-4f3d-8532-978fe17201c8\") " pod="ai-tenant-e2e-inf-a-ef1084/test-model-ef1084-kserve-6fb474f595-xm568"
Oct 30 01:04:08.665619 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:08.665600    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/652cd481-aaf4-4f3d-8532-978fe17201c8-tls-certs\") pod \"test-model-ef1084-kserve-6fb474f595-xm568\" (UID: \"652cd481-aaf4-4f3d-8532-978fe17201c8\") " pod="ai-tenant-e2e-inf-a-ef1084/test-model-ef1084-kserve-6fb474f595-xm568"
Oct 30 01:04:08.688177 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:08.688145    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zh8v5\" (UniqueName: \"kubernetes.io/projected/652cd481-aaf4-4f3d-8532-978fe17201c8-kube-api-access-zh8v5\") pod \"test-model-ef1084-kserve-6fb474f595-xm568\" (UID: \"652cd481-aaf4-4f3d-8532-978fe17201c8\") " pod="ai-tenant-e2e-inf-a-ef1084/test-model-ef1084-kserve-6fb474f595-xm568"
Oct 30 01:04:08.830505 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:08.829703    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-ef1084/test-model-ef1084-kserve-6fb474f595-xm568"
Oct 30 01:04:09.224843 ip-10-0-128-7 kubenswrapper[5002]: W1030 01:04:09.224804    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod652cd481_aaf4_4f3d_8532_978fe17201c8.slice/crio-6645ee10e1614835f0ea1814757486afdb65be359e5077ba22b99663a41304b7 WatchSource:0}: Error finding container 6645ee10e1614835f0ea1814757486afdb65be359e5077ba22b99663a41304b7: Status 404 returned error can't find the container with id 6645ee10e1614835f0ea1814757486afdb65be359e5077ba22b99663a41304b7
Oct 30 01:04:09.225284 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:09.225163    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-ef1084/test-model-ef1084-kserve-6fb474f595-xm568"]
Oct 30 01:04:09.711937 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:09.711851    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-ef1084/test-model-ef1084-kserve-6fb474f595-xm568" event={"ID":"652cd481-aaf4-4f3d-8532-978fe17201c8","Type":"ContainerStarted","Data":"856e94692202d759fc4f8ab9d6e93edcafdbf33abdabda95860b2eef0173879e"}
Oct 30 01:04:09.711937 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:09.711909    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-ef1084/test-model-ef1084-kserve-6fb474f595-xm568" event={"ID":"652cd481-aaf4-4f3d-8532-978fe17201c8","Type":"ContainerStarted","Data":"6645ee10e1614835f0ea1814757486afdb65be359e5077ba22b99663a41304b7"}
Oct 30 01:04:09.756619 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:09.756566    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-ef1084/test-model-ef1084-kserve-6fb474f595-xm568" podStartSLOduration=1.75655254 podStartE2EDuration="1.75655254s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:04:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:04:09.755129947 +0000 UTC m=+1945.371562823" watchObservedRunningTime="2026-10-30 01:04:09.75655254 +0000 UTC m=+1945.372985416"
Oct 30 01:04:09.831191 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:09.831152    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-ef1084/test-model-ef1084-kserve-6fb474f595-xm568"
Oct 30 01:04:12.864514 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:04:12.864479    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbjdq" containerName="extractor"
Oct 30 01:04:14.906523 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:14.903864    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-48df96/auth-test-model-48df96-kserve-9bb4b5585-cvvln"]
Oct 30 01:04:15.396824 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:15.396786    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8dbe73-788cc7d8c9-qnxcz"
Oct 30 01:04:15.456908 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:15.456871    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-8dbe73-698f74c77d-l7tpx"
Oct 30 01:04:15.476782 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:15.476699    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd1c2b8c-8b70-4583-bdec-d1901c0bd640-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd1c2b8c-8b70-4583-bdec-d1901c0bd640-plugins-config-volume\") pod \"dd1c2b8c-8b70-4583-bdec-d1901c0bd640\" (UID: \"dd1c2b8c-8b70-4583-bdec-d1901c0bd640\") "
Oct 30 01:04:15.476782 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:15.476741    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4521f1a9-13c3-409b-b0f3-d636722108a5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4521f1a9-13c3-409b-b0f3-d636722108a5-plugins-config-volume\") pod \"4521f1a9-13c3-409b-b0f3-d636722108a5\" (UID: \"4521f1a9-13c3-409b-b0f3-d636722108a5\") "
Oct 30 01:04:15.476956 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:15.476816    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4521f1a9-13c3-409b-b0f3-d636722108a5-kube-api-access-zlkk9\" (UniqueName: \"kubernetes.io/projected/4521f1a9-13c3-409b-b0f3-d636722108a5-kube-api-access-zlkk9\") pod \"4521f1a9-13c3-409b-b0f3-d636722108a5\" (UID: \"4521f1a9-13c3-409b-b0f3-d636722108a5\") "
Oct 30 01:04:15.476956 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:15.476840    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd1c2b8c-8b70-4583-bdec-d1901c0bd640-kube-api-access-fg2tp\" (UniqueName: \"kubernetes.io/projected/dd1c2b8c-8b70-4583-bdec-d1901c0bd640-kube-api-access-fg2tp\") pod \"dd1c2b8c-8b70-4583-bdec-d1901c0bd640\" (UID: \"dd1c2b8c-8b70-4583-bdec-d1901c0bd640\") "
Oct 30 01:04:15.477208 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:15.477174    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd1c2b8c-8b70-4583-bdec-d1901c0bd640-plugins-config-volume" pod "dd1c2b8c-8b70-4583-bdec-d1901c0bd640" (UID: "dd1c2b8c-8b70-4583-bdec-d1901c0bd640"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:04:15.477946 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:15.477915    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4521f1a9-13c3-409b-b0f3-d636722108a5-plugins-config-volume" pod "4521f1a9-13c3-409b-b0f3-d636722108a5" (UID: "4521f1a9-13c3-409b-b0f3-d636722108a5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:04:15.478117 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:15.478098    5002 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd1c2b8c-8b70-4583-bdec-d1901c0bd640-plugins-config-volume\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:04:15.478117 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:15.478117    5002 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4521f1a9-13c3-409b-b0f3-d636722108a5-plugins-config-volume\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:04:15.487451 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:15.479855    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd1c2b8c-8b70-4583-bdec-d1901c0bd640-kube-api-access-fg2tp" pod "dd1c2b8c-8b70-4583-bdec-d1901c0bd640" (UID: "dd1c2b8c-8b70-4583-bdec-d1901c0bd640"). InnerVolumeSpecName "kube-api-access-fg2tp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:04:15.526640 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:15.523951    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4521f1a9-13c3-409b-b0f3-d636722108a5-kube-api-access-zlkk9" pod "4521f1a9-13c3-409b-b0f3-d636722108a5" (UID: "4521f1a9-13c3-409b-b0f3-d636722108a5"). InnerVolumeSpecName "kube-api-access-zlkk9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:04:15.579799 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:15.579758    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zlkk9\" (UniqueName: \"kubernetes.io/projected/4521f1a9-13c3-409b-b0f3-d636722108a5-kube-api-access-zlkk9\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:04:15.579799 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:15.579797    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fg2tp\" (UniqueName: \"kubernetes.io/projected/dd1c2b8c-8b70-4583-bdec-d1901c0bd640-kube-api-access-fg2tp\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:04:15.764007 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:15.763910    5002 generic.go:356] "Generic (PLEG): container finished" podID="4521f1a9-13c3-409b-b0f3-d636722108a5" containerID="40402337a9a3a2dd5bcc1186f43b22c64efbb7956c12a4eddbfc2a1381888920" exitCode=0
Oct 30 01:04:15.764007 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:15.763980    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8dbe73-788cc7d8c9-qnxcz"
Oct 30 01:04:15.764259 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:15.763994    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8dbe73-788cc7d8c9-qnxcz" event={"ID":"4521f1a9-13c3-409b-b0f3-d636722108a5","Type":"ContainerDied","Data":"40402337a9a3a2dd5bcc1186f43b22c64efbb7956c12a4eddbfc2a1381888920"}
Oct 30 01:04:15.764259 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:15.764064    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8dbe73-788cc7d8c9-qnxcz" event={"ID":"4521f1a9-13c3-409b-b0f3-d636722108a5","Type":"ContainerDied","Data":"bf238e6eb8f275f4f6e6cf5094c885e61d6e59d9c09aeddd631c71ffef20ea4d"}
Oct 30 01:04:15.764259 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:15.764093    5002 scope.go:122] "RemoveContainer" containerID="40402337a9a3a2dd5bcc1186f43b22c64efbb7956c12a4eddbfc2a1381888920"
Oct 30 01:04:15.765812 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:15.765789    5002 generic.go:356] "Generic (PLEG): container finished" podID="dd1c2b8c-8b70-4583-bdec-d1901c0bd640" containerID="a21e598f815033054af077d05ebe0bea9f35846b54db470069f55dece4d9daa4" exitCode=0
Oct 30 01:04:15.765919 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:15.765844    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-8dbe73-698f74c77d-l7tpx"
Oct 30 01:04:15.765919 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:15.765864    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-8dbe73-698f74c77d-l7tpx" event={"ID":"dd1c2b8c-8b70-4583-bdec-d1901c0bd640","Type":"ContainerDied","Data":"a21e598f815033054af077d05ebe0bea9f35846b54db470069f55dece4d9daa4"}
Oct 30 01:04:15.765919 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:15.765905    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-8dbe73-698f74c77d-l7tpx" event={"ID":"dd1c2b8c-8b70-4583-bdec-d1901c0bd640","Type":"ContainerDied","Data":"b94b9099aac02494ab45dda93600ada102d2e788743824f8dfd4cf32a46f0919"}
Oct 30 01:04:15.778694 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:15.778672    5002 scope.go:122] "RemoveContainer" containerID="40402337a9a3a2dd5bcc1186f43b22c64efbb7956c12a4eddbfc2a1381888920"
Oct 30 01:04:15.779025 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:04:15.779005    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40402337a9a3a2dd5bcc1186f43b22c64efbb7956c12a4eddbfc2a1381888920\": container with ID starting with 40402337a9a3a2dd5bcc1186f43b22c64efbb7956c12a4eddbfc2a1381888920 not found: ID does not exist" containerID="40402337a9a3a2dd5bcc1186f43b22c64efbb7956c12a4eddbfc2a1381888920"
Oct 30 01:04:15.779068 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:15.779033    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40402337a9a3a2dd5bcc1186f43b22c64efbb7956c12a4eddbfc2a1381888920"} err="failed to get container status \"40402337a9a3a2dd5bcc1186f43b22c64efbb7956c12a4eddbfc2a1381888920\": rpc error: code = NotFound desc = could not find container \"40402337a9a3a2dd5bcc1186f43b22c64efbb7956c12a4eddbfc2a1381888920\": container with ID starting with 40402337a9a3a2dd5bcc1186f43b22c64efbb7956c12a4eddbfc2a1381888920 not found: ID does not exist"
Oct 30 01:04:15.779068 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:15.779050    5002 scope.go:122] "RemoveContainer" containerID="a21e598f815033054af077d05ebe0bea9f35846b54db470069f55dece4d9daa4"
Oct 30 01:04:15.791755 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:15.791734    5002 scope.go:122] "RemoveContainer" containerID="a21e598f815033054af077d05ebe0bea9f35846b54db470069f55dece4d9daa4"
Oct 30 01:04:15.792077 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:04:15.792051    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a21e598f815033054af077d05ebe0bea9f35846b54db470069f55dece4d9daa4\": container with ID starting with a21e598f815033054af077d05ebe0bea9f35846b54db470069f55dece4d9daa4 not found: ID does not exist" containerID="a21e598f815033054af077d05ebe0bea9f35846b54db470069f55dece4d9daa4"
Oct 30 01:04:15.792170 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:15.792084    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a21e598f815033054af077d05ebe0bea9f35846b54db470069f55dece4d9daa4"} err="failed to get container status \"a21e598f815033054af077d05ebe0bea9f35846b54db470069f55dece4d9daa4\": rpc error: code = NotFound desc = could not find container \"a21e598f815033054af077d05ebe0bea9f35846b54db470069f55dece4d9daa4\": container with ID starting with a21e598f815033054af077d05ebe0bea9f35846b54db470069f55dece4d9daa4 not found: ID does not exist"
Oct 30 01:04:15.827806 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:15.827770    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-8dbe73-788cc7d8c9-qnxcz"]
Oct 30 01:04:15.831310 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:15.831287    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-8dbe73-788cc7d8c9-qnxcz"]
Oct 30 01:04:15.872962 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:15.872925    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-8dbe73-698f74c77d-l7tpx"]
Oct 30 01:04:15.878920 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:15.878892    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-8dbe73-698f74c77d-l7tpx"]
Oct 30 01:04:16.500192 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:16.500158    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4521f1a9-13c3-409b-b0f3-d636722108a5" path="/var/lib/kubelet/pods/4521f1a9-13c3-409b-b0f3-d636722108a5/volumes"
Oct 30 01:04:16.500755 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:16.500737    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd1c2b8c-8b70-4583-bdec-d1901c0bd640" path="/var/lib/kubelet/pods/dd1c2b8c-8b70-4583-bdec-d1901c0bd640/volumes"
Oct 30 01:04:18.830638 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:18.830600    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-ef1084/test-model-ef1084-kserve-6fb474f595-xm568"
Oct 30 01:04:18.843518 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:18.843487    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-ef1084/test-model-ef1084-kserve-6fb474f595-xm568"
Oct 30 01:04:19.815127 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:19.815094    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-ef1084/test-model-ef1084-kserve-6fb474f595-xm568"
Oct 30 01:04:26.705189 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:26.705138    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8dbe73-wj7p4"]
Oct 30 01:04:26.708775 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:26.708741    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8dbe73-wj7p4"]
Oct 30 01:04:26.891092 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:26.891052    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-73435a-openshift-default-fc454477f-x6g2b"]
Oct 30 01:04:26.892293 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:26.892263    5002 cpu_manager.go:414] "containerMap: removing container" podUID="dd1c2b8c-8b70-4583-bdec-d1901c0bd640" containerName="payload-processing"
Oct 30 01:04:26.892481 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:26.892464    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd1c2b8c-8b70-4583-bdec-d1901c0bd640" containerName="payload-processing"
Oct 30 01:04:26.892581 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:26.892569    5002 cpu_manager.go:414] "containerMap: removing container" podUID="4521f1a9-13c3-409b-b0f3-d636722108a5" containerName="payload-pre-processing"
Oct 30 01:04:26.892668 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:26.892656    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4521f1a9-13c3-409b-b0f3-d636722108a5" containerName="payload-pre-processing"
Oct 30 01:04:26.893026 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:26.893012    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd1c2b8c-8b70-4583-bdec-d1901c0bd640" containerName="payload-processing"
Oct 30 01:04:26.893130 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:26.893120    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="4521f1a9-13c3-409b-b0f3-d636722108a5" containerName="payload-pre-processing"
Oct 30 01:04:26.904424 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:26.904396    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-73435a-openshift-default-fc454477f-x6g2b"
Oct 30 01:04:26.951550 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:26.951518    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b38dbda1-9a06-461d-8d0a-a9413b76d333-istio-data\") pod \"e2e-shared-b-w4-73435a-openshift-default-fc454477f-x6g2b\" (UID: \"b38dbda1-9a06-461d-8d0a-a9413b76d333\") " pod="openshift-ingress/e2e-shared-b-w4-73435a-openshift-default-fc454477f-x6g2b"
Oct 30 01:04:26.951719 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:26.951578    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b38dbda1-9a06-461d-8d0a-a9413b76d333-istio-token\") pod \"e2e-shared-b-w4-73435a-openshift-default-fc454477f-x6g2b\" (UID: \"b38dbda1-9a06-461d-8d0a-a9413b76d333\") " pod="openshift-ingress/e2e-shared-b-w4-73435a-openshift-default-fc454477f-x6g2b"
Oct 30 01:04:26.951719 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:26.951672    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b38dbda1-9a06-461d-8d0a-a9413b76d333-istio-podinfo\") pod \"e2e-shared-b-w4-73435a-openshift-default-fc454477f-x6g2b\" (UID: \"b38dbda1-9a06-461d-8d0a-a9413b76d333\") " pod="openshift-ingress/e2e-shared-b-w4-73435a-openshift-default-fc454477f-x6g2b"
Oct 30 01:04:26.951795 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:26.951721    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b38dbda1-9a06-461d-8d0a-a9413b76d333-istio-envoy\") pod \"e2e-shared-b-w4-73435a-openshift-default-fc454477f-x6g2b\" (UID: \"b38dbda1-9a06-461d-8d0a-a9413b76d333\") " pod="openshift-ingress/e2e-shared-b-w4-73435a-openshift-default-fc454477f-x6g2b"
Oct 30 01:04:26.951830 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:26.951805    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b38dbda1-9a06-461d-8d0a-a9413b76d333-workload-certs\") pod \"e2e-shared-b-w4-73435a-openshift-default-fc454477f-x6g2b\" (UID: \"b38dbda1-9a06-461d-8d0a-a9413b76d333\") " pod="openshift-ingress/e2e-shared-b-w4-73435a-openshift-default-fc454477f-x6g2b"
Oct 30 01:04:26.951889 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:26.951871    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vt5w8\" (UniqueName: \"kubernetes.io/projected/b38dbda1-9a06-461d-8d0a-a9413b76d333-kube-api-access-vt5w8\") pod \"e2e-shared-b-w4-73435a-openshift-default-fc454477f-x6g2b\" (UID: \"b38dbda1-9a06-461d-8d0a-a9413b76d333\") " pod="openshift-ingress/e2e-shared-b-w4-73435a-openshift-default-fc454477f-x6g2b"
Oct 30 01:04:26.951941 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:26.951912    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b38dbda1-9a06-461d-8d0a-a9413b76d333-credential-socket\") pod \"e2e-shared-b-w4-73435a-openshift-default-fc454477f-x6g2b\" (UID: \"b38dbda1-9a06-461d-8d0a-a9413b76d333\") " pod="openshift-ingress/e2e-shared-b-w4-73435a-openshift-default-fc454477f-x6g2b"
Oct 30 01:04:26.952070 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:26.952056    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b38dbda1-9a06-461d-8d0a-a9413b76d333-istiod-ca-cert\") pod \"e2e-shared-b-w4-73435a-openshift-default-fc454477f-x6g2b\" (UID: \"b38dbda1-9a06-461d-8d0a-a9413b76d333\") " pod="openshift-ingress/e2e-shared-b-w4-73435a-openshift-default-fc454477f-x6g2b"
Oct 30 01:04:26.952143 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:26.952130    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b38dbda1-9a06-461d-8d0a-a9413b76d333-workload-socket\") pod \"e2e-shared-b-w4-73435a-openshift-default-fc454477f-x6g2b\" (UID: \"b38dbda1-9a06-461d-8d0a-a9413b76d333\") " pod="openshift-ingress/e2e-shared-b-w4-73435a-openshift-default-fc454477f-x6g2b"
Oct 30 01:04:26.967048 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:26.966969    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-73435a-openshift-default-fc454477f-x6g2b"]
Oct 30 01:04:26.997961 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:26.982595    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-73435a-openshift-default-dockercfg-whl5c\""
Oct 30 01:04:27.095409 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:27.085814    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b38dbda1-9a06-461d-8d0a-a9413b76d333-istiod-ca-cert\") pod \"e2e-shared-b-w4-73435a-openshift-default-fc454477f-x6g2b\" (UID: \"b38dbda1-9a06-461d-8d0a-a9413b76d333\") " pod="openshift-ingress/e2e-shared-b-w4-73435a-openshift-default-fc454477f-x6g2b"
Oct 30 01:04:27.095409 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:27.085912    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b38dbda1-9a06-461d-8d0a-a9413b76d333-workload-socket\") pod \"e2e-shared-b-w4-73435a-openshift-default-fc454477f-x6g2b\" (UID: \"b38dbda1-9a06-461d-8d0a-a9413b76d333\") " pod="openshift-ingress/e2e-shared-b-w4-73435a-openshift-default-fc454477f-x6g2b"
Oct 30 01:04:27.095409 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:27.086025    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b38dbda1-9a06-461d-8d0a-a9413b76d333-istio-data\") pod \"e2e-shared-b-w4-73435a-openshift-default-fc454477f-x6g2b\" (UID: \"b38dbda1-9a06-461d-8d0a-a9413b76d333\") " pod="openshift-ingress/e2e-shared-b-w4-73435a-openshift-default-fc454477f-x6g2b"
Oct 30 01:04:27.095409 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:27.086070    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b38dbda1-9a06-461d-8d0a-a9413b76d333-istio-token\") pod \"e2e-shared-b-w4-73435a-openshift-default-fc454477f-x6g2b\" (UID: \"b38dbda1-9a06-461d-8d0a-a9413b76d333\") " pod="openshift-ingress/e2e-shared-b-w4-73435a-openshift-default-fc454477f-x6g2b"
Oct 30 01:04:27.095409 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:27.086101    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b38dbda1-9a06-461d-8d0a-a9413b76d333-istio-podinfo\") pod \"e2e-shared-b-w4-73435a-openshift-default-fc454477f-x6g2b\" (UID: \"b38dbda1-9a06-461d-8d0a-a9413b76d333\") " pod="openshift-ingress/e2e-shared-b-w4-73435a-openshift-default-fc454477f-x6g2b"
Oct 30 01:04:27.095409 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:27.086133    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b38dbda1-9a06-461d-8d0a-a9413b76d333-istio-envoy\") pod \"e2e-shared-b-w4-73435a-openshift-default-fc454477f-x6g2b\" (UID: \"b38dbda1-9a06-461d-8d0a-a9413b76d333\") " pod="openshift-ingress/e2e-shared-b-w4-73435a-openshift-default-fc454477f-x6g2b"
Oct 30 01:04:27.095409 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:27.086181    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b38dbda1-9a06-461d-8d0a-a9413b76d333-workload-certs\") pod \"e2e-shared-b-w4-73435a-openshift-default-fc454477f-x6g2b\" (UID: \"b38dbda1-9a06-461d-8d0a-a9413b76d333\") " pod="openshift-ingress/e2e-shared-b-w4-73435a-openshift-default-fc454477f-x6g2b"
Oct 30 01:04:27.095409 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:27.086246    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vt5w8\" (UniqueName: \"kubernetes.io/projected/b38dbda1-9a06-461d-8d0a-a9413b76d333-kube-api-access-vt5w8\") pod \"e2e-shared-b-w4-73435a-openshift-default-fc454477f-x6g2b\" (UID: \"b38dbda1-9a06-461d-8d0a-a9413b76d333\") " pod="openshift-ingress/e2e-shared-b-w4-73435a-openshift-default-fc454477f-x6g2b"
Oct 30 01:04:27.095409 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:27.086287    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b38dbda1-9a06-461d-8d0a-a9413b76d333-credential-socket\") pod \"e2e-shared-b-w4-73435a-openshift-default-fc454477f-x6g2b\" (UID: \"b38dbda1-9a06-461d-8d0a-a9413b76d333\") " pod="openshift-ingress/e2e-shared-b-w4-73435a-openshift-default-fc454477f-x6g2b"
Oct 30 01:04:27.095409 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:27.086819    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b38dbda1-9a06-461d-8d0a-a9413b76d333-credential-socket\") pod \"e2e-shared-b-w4-73435a-openshift-default-fc454477f-x6g2b\" (UID: \"b38dbda1-9a06-461d-8d0a-a9413b76d333\") " pod="openshift-ingress/e2e-shared-b-w4-73435a-openshift-default-fc454477f-x6g2b"
Oct 30 01:04:27.095409 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:27.087541    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b38dbda1-9a06-461d-8d0a-a9413b76d333-istiod-ca-cert\") pod \"e2e-shared-b-w4-73435a-openshift-default-fc454477f-x6g2b\" (UID: \"b38dbda1-9a06-461d-8d0a-a9413b76d333\") " pod="openshift-ingress/e2e-shared-b-w4-73435a-openshift-default-fc454477f-x6g2b"
Oct 30 01:04:27.095409 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:27.087804    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b38dbda1-9a06-461d-8d0a-a9413b76d333-workload-socket\") pod \"e2e-shared-b-w4-73435a-openshift-default-fc454477f-x6g2b\" (UID: \"b38dbda1-9a06-461d-8d0a-a9413b76d333\") " pod="openshift-ingress/e2e-shared-b-w4-73435a-openshift-default-fc454477f-x6g2b"
Oct 30 01:04:27.095409 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:27.088047    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b38dbda1-9a06-461d-8d0a-a9413b76d333-istio-data\") pod \"e2e-shared-b-w4-73435a-openshift-default-fc454477f-x6g2b\" (UID: \"b38dbda1-9a06-461d-8d0a-a9413b76d333\") " pod="openshift-ingress/e2e-shared-b-w4-73435a-openshift-default-fc454477f-x6g2b"
Oct 30 01:04:27.214585 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:27.196033    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b38dbda1-9a06-461d-8d0a-a9413b76d333-istio-token\") pod \"e2e-shared-b-w4-73435a-openshift-default-fc454477f-x6g2b\" (UID: \"b38dbda1-9a06-461d-8d0a-a9413b76d333\") " pod="openshift-ingress/e2e-shared-b-w4-73435a-openshift-default-fc454477f-x6g2b"
Oct 30 01:04:27.215746 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:27.215697    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b38dbda1-9a06-461d-8d0a-a9413b76d333-workload-certs\") pod \"e2e-shared-b-w4-73435a-openshift-default-fc454477f-x6g2b\" (UID: \"b38dbda1-9a06-461d-8d0a-a9413b76d333\") " pod="openshift-ingress/e2e-shared-b-w4-73435a-openshift-default-fc454477f-x6g2b"
Oct 30 01:04:27.280540 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:27.259427    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vt5w8\" (UniqueName: \"kubernetes.io/projected/b38dbda1-9a06-461d-8d0a-a9413b76d333-kube-api-access-vt5w8\") pod \"e2e-shared-b-w4-73435a-openshift-default-fc454477f-x6g2b\" (UID: \"b38dbda1-9a06-461d-8d0a-a9413b76d333\") " pod="openshift-ingress/e2e-shared-b-w4-73435a-openshift-default-fc454477f-x6g2b"
Oct 30 01:04:27.307560 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:27.307245    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b38dbda1-9a06-461d-8d0a-a9413b76d333-istio-envoy\") pod \"e2e-shared-b-w4-73435a-openshift-default-fc454477f-x6g2b\" (UID: \"b38dbda1-9a06-461d-8d0a-a9413b76d333\") " pod="openshift-ingress/e2e-shared-b-w4-73435a-openshift-default-fc454477f-x6g2b"
Oct 30 01:04:27.336999 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:27.335128    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b38dbda1-9a06-461d-8d0a-a9413b76d333-istio-podinfo\") pod \"e2e-shared-b-w4-73435a-openshift-default-fc454477f-x6g2b\" (UID: \"b38dbda1-9a06-461d-8d0a-a9413b76d333\") " pod="openshift-ingress/e2e-shared-b-w4-73435a-openshift-default-fc454477f-x6g2b"
Oct 30 01:04:27.521359 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:27.521322    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-73435a-openshift-default-fc454477f-x6g2b"
Oct 30 01:04:27.859432 ip-10-0-128-7 kubenswrapper[5002]: W1030 01:04:27.858348    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb38dbda1_9a06_461d_8d0a_a9413b76d333.slice/crio-a7f730f453f33fe99b5976f4ee5b7f95453d1879fb7ee44d631f94659d76ca1f WatchSource:0}: Error finding container a7f730f453f33fe99b5976f4ee5b7f95453d1879fb7ee44d631f94659d76ca1f: Status 404 returned error can't find the container with id a7f730f453f33fe99b5976f4ee5b7f95453d1879fb7ee44d631f94659d76ca1f
Oct 30 01:04:27.860439 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:27.860394    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-73435a-openshift-default-fc454477f-x6g2b"]
Oct 30 01:04:27.861926 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:27.861891    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:04:27.862026 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:27.861969    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:04:27.862026 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:27.862001    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:04:27.940352 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:27.940317    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-73435a-openshift-default-fc454477f-x6g2b" event={"ID":"b38dbda1-9a06-461d-8d0a-a9413b76d333","Type":"ContainerStarted","Data":"a7f730f453f33fe99b5976f4ee5b7f95453d1879fb7ee44d631f94659d76ca1f"}
Oct 30 01:04:28.499139 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:28.499103    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a58c0b7-f707-45fc-9c2c-38dd671fb592" path="/var/lib/kubelet/pods/0a58c0b7-f707-45fc-9c2c-38dd671fb592/volumes"
Oct 30 01:04:28.953950 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:28.951017    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-73435a-openshift-default-fc454477f-x6g2b" event={"ID":"b38dbda1-9a06-461d-8d0a-a9413b76d333","Type":"ContainerStarted","Data":"b245cf8d44445c2ed5ef47ce7669b2c108f4555f746c312e4322c4795e538938"}
Oct 30 01:04:29.015022 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:29.014964    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-73435a-openshift-default-fc454477f-x6g2b" podStartSLOduration=3.014951625 podStartE2EDuration="3.014951625s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:04:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:04:29.013331335 +0000 UTC m=+1964.629764227" watchObservedRunningTime="2026-10-30 01:04:29.014951625 +0000 UTC m=+1964.631384503"
Oct 30 01:04:29.085197 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:29.085153    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7d8z4"]
Oct 30 01:04:29.521598 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:29.521563    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-73435a-openshift-default-fc454477f-x6g2b"
Oct 30 01:04:29.529770 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:29.529734    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-73435a-openshift-default-fc454477f-x6g2b"
Oct 30 01:04:29.931672 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:29.931570    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w1-48df96/auth-test-model-48df96-kserve-9bb4b5585-cvvln" podUID="22351ddc-9884-4058-b71e-7c320c2da701" containerName="main" containerID="cri-o://578e94fe330d154d28c00680bf3416b451f2a56bc0672cde0b1e37b779581440" gracePeriod=45
Oct 30 01:04:29.959751 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:29.959722    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-73435a-openshift-default-fc454477f-x6g2b"
Oct 30 01:04:29.960993 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:29.960966    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-73435a-openshift-default-fc454477f-x6g2b"
Oct 30 01:04:30.276856 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:30.276827    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-48df96/auth-test-model-48df96-kserve-9bb4b5585-cvvln"
Oct 30 01:04:30.380206 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:30.380171    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22351ddc-9884-4058-b71e-7c320c2da701-kube-api-access-4g4k4\" (UniqueName: \"kubernetes.io/projected/22351ddc-9884-4058-b71e-7c320c2da701-kube-api-access-4g4k4\") pod \"22351ddc-9884-4058-b71e-7c320c2da701\" (UID: \"22351ddc-9884-4058-b71e-7c320c2da701\") "
Oct 30 01:04:30.380437 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:30.380247    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22351ddc-9884-4058-b71e-7c320c2da701-home\" (UniqueName: \"kubernetes.io/empty-dir/22351ddc-9884-4058-b71e-7c320c2da701-home\") pod \"22351ddc-9884-4058-b71e-7c320c2da701\" (UID: \"22351ddc-9884-4058-b71e-7c320c2da701\") "
Oct 30 01:04:30.380437 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:30.380341    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22351ddc-9884-4058-b71e-7c320c2da701-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22351ddc-9884-4058-b71e-7c320c2da701-model-cache\") pod \"22351ddc-9884-4058-b71e-7c320c2da701\" (UID: \"22351ddc-9884-4058-b71e-7c320c2da701\") "
Oct 30 01:04:30.380437 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:30.380374    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22351ddc-9884-4058-b71e-7c320c2da701-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22351ddc-9884-4058-b71e-7c320c2da701-tmp-dir\") pod \"22351ddc-9884-4058-b71e-7c320c2da701\" (UID: \"22351ddc-9884-4058-b71e-7c320c2da701\") "
Oct 30 01:04:30.380628 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:30.380457    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/22351ddc-9884-4058-b71e-7c320c2da701-tls-certs\" (UniqueName: \"kubernetes.io/secret/22351ddc-9884-4058-b71e-7c320c2da701-tls-certs\") pod \"22351ddc-9884-4058-b71e-7c320c2da701\" (UID: \"22351ddc-9884-4058-b71e-7c320c2da701\") "
Oct 30 01:04:30.380628 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:30.380519    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22351ddc-9884-4058-b71e-7c320c2da701-dshm\" (UniqueName: \"kubernetes.io/empty-dir/22351ddc-9884-4058-b71e-7c320c2da701-dshm\") pod \"22351ddc-9884-4058-b71e-7c320c2da701\" (UID: \"22351ddc-9884-4058-b71e-7c320c2da701\") "
Oct 30 01:04:30.381021 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:30.380993    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22351ddc-9884-4058-b71e-7c320c2da701-model-cache" pod "22351ddc-9884-4058-b71e-7c320c2da701" (UID: "22351ddc-9884-4058-b71e-7c320c2da701"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:04:30.381554 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:30.381462    5002 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22351ddc-9884-4058-b71e-7c320c2da701-model-cache\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:04:30.381651 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:30.381569    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22351ddc-9884-4058-b71e-7c320c2da701-tmp-dir" pod "22351ddc-9884-4058-b71e-7c320c2da701" (UID: "22351ddc-9884-4058-b71e-7c320c2da701"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:04:30.381747 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:30.381678    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22351ddc-9884-4058-b71e-7c320c2da701-home" pod "22351ddc-9884-4058-b71e-7c320c2da701" (UID: "22351ddc-9884-4058-b71e-7c320c2da701"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:04:30.383464 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:30.383435    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22351ddc-9884-4058-b71e-7c320c2da701-kube-api-access-4g4k4" pod "22351ddc-9884-4058-b71e-7c320c2da701" (UID: "22351ddc-9884-4058-b71e-7c320c2da701"). InnerVolumeSpecName "kube-api-access-4g4k4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:04:30.383553 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:30.383525    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22351ddc-9884-4058-b71e-7c320c2da701-dshm" pod "22351ddc-9884-4058-b71e-7c320c2da701" (UID: "22351ddc-9884-4058-b71e-7c320c2da701"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:04:30.383765 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:30.383740    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22351ddc-9884-4058-b71e-7c320c2da701-tls-certs" pod "22351ddc-9884-4058-b71e-7c320c2da701" (UID: "22351ddc-9884-4058-b71e-7c320c2da701"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:04:30.483366 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:30.483329    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4g4k4\" (UniqueName: \"kubernetes.io/projected/22351ddc-9884-4058-b71e-7c320c2da701-kube-api-access-4g4k4\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:04:30.483366 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:30.483363    5002 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22351ddc-9884-4058-b71e-7c320c2da701-home\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:04:30.483598 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:30.483417    5002 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22351ddc-9884-4058-b71e-7c320c2da701-tmp-dir\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:04:30.483598 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:30.483428    5002 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22351ddc-9884-4058-b71e-7c320c2da701-tls-certs\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:04:30.483598 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:30.483436    5002 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22351ddc-9884-4058-b71e-7c320c2da701-dshm\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:04:30.979652 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:30.977712    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-19bc68/test-model-19bc68-kserve-77fd689c7c-qw7lc"]
Oct 30 01:04:30.979652 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:30.978681    5002 cpu_manager.go:414] "containerMap: removing container" podUID="22351ddc-9884-4058-b71e-7c320c2da701" containerName="main"
Oct 30 01:04:30.979652 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:30.978701    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22351ddc-9884-4058-b71e-7c320c2da701" containerName="main"
Oct 30 01:04:30.979652 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:30.978863    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="22351ddc-9884-4058-b71e-7c320c2da701" containerName="main"
Oct 30 01:04:30.986834 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:30.985673    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-19bc68/test-model-19bc68-kserve-77fd689c7c-qw7lc"]
Oct 30 01:04:30.986834 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:30.985759    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-19bc68/test-model-19bc68-kserve-77fd689c7c-qw7lc"
Oct 30 01:04:31.006434 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:31.006188    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-19bc68\"/\"test-model-19bc68-kserve-self-signed-certs\""
Oct 30 01:04:31.006434 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:31.006245    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-19bc68\"/\"openshift-service-ca.crt\""
Oct 30 01:04:31.006434 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:31.006368    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-19bc68\"/\"default-dockercfg-9pnwp\""
Oct 30 01:04:31.012136 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:31.010312    5002 generic.go:356] "Generic (PLEG): container finished" podID="22351ddc-9884-4058-b71e-7c320c2da701" containerID="578e94fe330d154d28c00680bf3416b451f2a56bc0672cde0b1e37b779581440" exitCode=0
Oct 30 01:04:31.012136 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:31.010526    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-48df96/auth-test-model-48df96-kserve-9bb4b5585-cvvln" event={"ID":"22351ddc-9884-4058-b71e-7c320c2da701","Type":"ContainerDied","Data":"578e94fe330d154d28c00680bf3416b451f2a56bc0672cde0b1e37b779581440"}
Oct 30 01:04:31.012136 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:31.010563    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-48df96/auth-test-model-48df96-kserve-9bb4b5585-cvvln" event={"ID":"22351ddc-9884-4058-b71e-7c320c2da701","Type":"ContainerDied","Data":"8ab78a1df6e272ee61949526d60496490e917347d3748d45b4d3aeb97c217a58"}
Oct 30 01:04:31.012136 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:31.010581    5002 scope.go:122] "RemoveContainer" containerID="578e94fe330d154d28c00680bf3416b451f2a56bc0672cde0b1e37b779581440"
Oct 30 01:04:31.012136 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:31.010715    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-48df96/auth-test-model-48df96-kserve-9bb4b5585-cvvln"
Oct 30 01:04:31.022398 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:31.020085    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-19bc68\"/\"kube-root-ca.crt\""
Oct 30 01:04:31.031516 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:31.031494    5002 scope.go:122] "RemoveContainer" containerID="578e94fe330d154d28c00680bf3416b451f2a56bc0672cde0b1e37b779581440"
Oct 30 01:04:31.031855 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:04:31.031832    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"578e94fe330d154d28c00680bf3416b451f2a56bc0672cde0b1e37b779581440\": container with ID starting with 578e94fe330d154d28c00680bf3416b451f2a56bc0672cde0b1e37b779581440 not found: ID does not exist" containerID="578e94fe330d154d28c00680bf3416b451f2a56bc0672cde0b1e37b779581440"
Oct 30 01:04:31.031943 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:31.031868    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"578e94fe330d154d28c00680bf3416b451f2a56bc0672cde0b1e37b779581440"} err="failed to get container status \"578e94fe330d154d28c00680bf3416b451f2a56bc0672cde0b1e37b779581440\": rpc error: code = NotFound desc = could not find container \"578e94fe330d154d28c00680bf3416b451f2a56bc0672cde0b1e37b779581440\": container with ID starting with 578e94fe330d154d28c00680bf3416b451f2a56bc0672cde0b1e37b779581440 not found: ID does not exist"
Oct 30 01:04:31.094081 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:31.093467    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-48df96/auth-test-model-48df96-kserve-9bb4b5585-cvvln"]
Oct 30 01:04:31.099338 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:31.099304    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w1-48df96/auth-test-model-48df96-kserve-9bb4b5585-cvvln"]
Oct 30 01:04:31.141400 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:31.139871    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/076dfc74-4c57-46c9-946f-f06e3fb77f79-dshm\") pod \"test-model-19bc68-kserve-77fd689c7c-qw7lc\" (UID: \"076dfc74-4c57-46c9-946f-f06e3fb77f79\") " pod="ai-tenant-e2e-inf-b-19bc68/test-model-19bc68-kserve-77fd689c7c-qw7lc"
Oct 30 01:04:31.141400 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:31.139948    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28gxr\" (UniqueName: \"kubernetes.io/projected/076dfc74-4c57-46c9-946f-f06e3fb77f79-kube-api-access-28gxr\") pod \"test-model-19bc68-kserve-77fd689c7c-qw7lc\" (UID: \"076dfc74-4c57-46c9-946f-f06e3fb77f79\") " pod="ai-tenant-e2e-inf-b-19bc68/test-model-19bc68-kserve-77fd689c7c-qw7lc"
Oct 30 01:04:31.141400 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:31.139990    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/076dfc74-4c57-46c9-946f-f06e3fb77f79-tmp-dir\") pod \"test-model-19bc68-kserve-77fd689c7c-qw7lc\" (UID: \"076dfc74-4c57-46c9-946f-f06e3fb77f79\") " pod="ai-tenant-e2e-inf-b-19bc68/test-model-19bc68-kserve-77fd689c7c-qw7lc"
Oct 30 01:04:31.141400 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:31.140099    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/076dfc74-4c57-46c9-946f-f06e3fb77f79-model-cache\") pod \"test-model-19bc68-kserve-77fd689c7c-qw7lc\" (UID: \"076dfc74-4c57-46c9-946f-f06e3fb77f79\") " pod="ai-tenant-e2e-inf-b-19bc68/test-model-19bc68-kserve-77fd689c7c-qw7lc"
Oct 30 01:04:31.141400 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:31.140208    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/076dfc74-4c57-46c9-946f-f06e3fb77f79-home\") pod \"test-model-19bc68-kserve-77fd689c7c-qw7lc\" (UID: \"076dfc74-4c57-46c9-946f-f06e3fb77f79\") " pod="ai-tenant-e2e-inf-b-19bc68/test-model-19bc68-kserve-77fd689c7c-qw7lc"
Oct 30 01:04:31.141400 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:31.140570    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/076dfc74-4c57-46c9-946f-f06e3fb77f79-tls-certs\") pod \"test-model-19bc68-kserve-77fd689c7c-qw7lc\" (UID: \"076dfc74-4c57-46c9-946f-f06e3fb77f79\") " pod="ai-tenant-e2e-inf-b-19bc68/test-model-19bc68-kserve-77fd689c7c-qw7lc"
Oct 30 01:04:31.253462 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:31.246355    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/076dfc74-4c57-46c9-946f-f06e3fb77f79-model-cache\") pod \"test-model-19bc68-kserve-77fd689c7c-qw7lc\" (UID: \"076dfc74-4c57-46c9-946f-f06e3fb77f79\") " pod="ai-tenant-e2e-inf-b-19bc68/test-model-19bc68-kserve-77fd689c7c-qw7lc"
Oct 30 01:04:31.253462 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:31.246486    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/076dfc74-4c57-46c9-946f-f06e3fb77f79-home\") pod \"test-model-19bc68-kserve-77fd689c7c-qw7lc\" (UID: \"076dfc74-4c57-46c9-946f-f06e3fb77f79\") " pod="ai-tenant-e2e-inf-b-19bc68/test-model-19bc68-kserve-77fd689c7c-qw7lc"
Oct 30 01:04:31.253462 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:31.246572    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/076dfc74-4c57-46c9-946f-f06e3fb77f79-tls-certs\") pod \"test-model-19bc68-kserve-77fd689c7c-qw7lc\" (UID: \"076dfc74-4c57-46c9-946f-f06e3fb77f79\") " pod="ai-tenant-e2e-inf-b-19bc68/test-model-19bc68-kserve-77fd689c7c-qw7lc"
Oct 30 01:04:31.253462 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:31.246764    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/076dfc74-4c57-46c9-946f-f06e3fb77f79-dshm\") pod \"test-model-19bc68-kserve-77fd689c7c-qw7lc\" (UID: \"076dfc74-4c57-46c9-946f-f06e3fb77f79\") " pod="ai-tenant-e2e-inf-b-19bc68/test-model-19bc68-kserve-77fd689c7c-qw7lc"
Oct 30 01:04:31.253462 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:31.246816    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28gxr\" (UniqueName: \"kubernetes.io/projected/076dfc74-4c57-46c9-946f-f06e3fb77f79-kube-api-access-28gxr\") pod \"test-model-19bc68-kserve-77fd689c7c-qw7lc\" (UID: \"076dfc74-4c57-46c9-946f-f06e3fb77f79\") " pod="ai-tenant-e2e-inf-b-19bc68/test-model-19bc68-kserve-77fd689c7c-qw7lc"
Oct 30 01:04:31.253462 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:31.246871    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/076dfc74-4c57-46c9-946f-f06e3fb77f79-tmp-dir\") pod \"test-model-19bc68-kserve-77fd689c7c-qw7lc\" (UID: \"076dfc74-4c57-46c9-946f-f06e3fb77f79\") " pod="ai-tenant-e2e-inf-b-19bc68/test-model-19bc68-kserve-77fd689c7c-qw7lc"
Oct 30 01:04:31.253462 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:31.247359    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/076dfc74-4c57-46c9-946f-f06e3fb77f79-tmp-dir\") pod \"test-model-19bc68-kserve-77fd689c7c-qw7lc\" (UID: \"076dfc74-4c57-46c9-946f-f06e3fb77f79\") " pod="ai-tenant-e2e-inf-b-19bc68/test-model-19bc68-kserve-77fd689c7c-qw7lc"
Oct 30 01:04:31.253462 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:31.247659    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/076dfc74-4c57-46c9-946f-f06e3fb77f79-model-cache\") pod \"test-model-19bc68-kserve-77fd689c7c-qw7lc\" (UID: \"076dfc74-4c57-46c9-946f-f06e3fb77f79\") " pod="ai-tenant-e2e-inf-b-19bc68/test-model-19bc68-kserve-77fd689c7c-qw7lc"
Oct 30 01:04:31.253462 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:31.247902    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/076dfc74-4c57-46c9-946f-f06e3fb77f79-home\") pod \"test-model-19bc68-kserve-77fd689c7c-qw7lc\" (UID: \"076dfc74-4c57-46c9-946f-f06e3fb77f79\") " pod="ai-tenant-e2e-inf-b-19bc68/test-model-19bc68-kserve-77fd689c7c-qw7lc"
Oct 30 01:04:31.271937 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:31.271889    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/076dfc74-4c57-46c9-946f-f06e3fb77f79-dshm\") pod \"test-model-19bc68-kserve-77fd689c7c-qw7lc\" (UID: \"076dfc74-4c57-46c9-946f-f06e3fb77f79\") " pod="ai-tenant-e2e-inf-b-19bc68/test-model-19bc68-kserve-77fd689c7c-qw7lc"
Oct 30 01:04:31.316402 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:31.301858    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-28gxr\" (UniqueName: \"kubernetes.io/projected/076dfc74-4c57-46c9-946f-f06e3fb77f79-kube-api-access-28gxr\") pod \"test-model-19bc68-kserve-77fd689c7c-qw7lc\" (UID: \"076dfc74-4c57-46c9-946f-f06e3fb77f79\") " pod="ai-tenant-e2e-inf-b-19bc68/test-model-19bc68-kserve-77fd689c7c-qw7lc"
Oct 30 01:04:31.316402 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:31.310825    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/076dfc74-4c57-46c9-946f-f06e3fb77f79-tls-certs\") pod \"test-model-19bc68-kserve-77fd689c7c-qw7lc\" (UID: \"076dfc74-4c57-46c9-946f-f06e3fb77f79\") " pod="ai-tenant-e2e-inf-b-19bc68/test-model-19bc68-kserve-77fd689c7c-qw7lc"
Oct 30 01:04:31.316402 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:31.311097    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-19bc68/test-model-19bc68-kserve-77fd689c7c-qw7lc"
Oct 30 01:04:32.308735 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:32.308694    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1a11cf/distinct-2-w2-1a11cf-kserve-758bb6dd57-j5f2k" podUID="f2d66dea-9f0b-43f3-a774-bb065e88f378" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:04:32.449118 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:32.449052    5002 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-52650f/distinct-w0-52650f-kserve-d85899b5-vlhps" podUID="ea1850d9-abba-4977-a540-f6e0eaefb17d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/health\": context deadline exceeded"
Oct 30 01:04:32.541316 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:32.541280    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22351ddc-9884-4058-b71e-7c320c2da701" path="/var/lib/kubelet/pods/22351ddc-9884-4058-b71e-7c320c2da701/volumes"
Oct 30 01:04:32.949731 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:32.941825    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-19bc68/test-model-19bc68-kserve-77fd689c7c-qw7lc"]
Oct 30 01:04:32.990887 ip-10-0-128-7 kubenswrapper[5002]: W1030 01:04:32.972837    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod076dfc74_4c57_46c9_946f_f06e3fb77f79.slice/crio-5d74017a1b369ab9c5e04a89fdd382eb3e9c66e5f07531c77d5847a96cd3e54b WatchSource:0}: Error finding container 5d74017a1b369ab9c5e04a89fdd382eb3e9c66e5f07531c77d5847a96cd3e54b: Status 404 returned error can't find the container with id 5d74017a1b369ab9c5e04a89fdd382eb3e9c66e5f07531c77d5847a96cd3e54b
Oct 30 01:04:33.201844 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:33.201707    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-19bc68/test-model-19bc68-kserve-77fd689c7c-qw7lc" event={"ID":"076dfc74-4c57-46c9-946f-f06e3fb77f79","Type":"ContainerStarted","Data":"5d74017a1b369ab9c5e04a89fdd382eb3e9c66e5f07531c77d5847a96cd3e54b"}
Oct 30 01:04:33.550825 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:33.550733    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-73435a-7645c9bc64-wcm9t"]
Oct 30 01:04:33.556235 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:33.556211    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-73435a-7645c9bc64-wcm9t"
Oct 30 01:04:33.574454 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:33.574424    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-73435a-7645c9bc64-wcm9t"]
Oct 30 01:04:33.583587 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:33.583561    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w4-73435a-dockercfg-zlsbl\""
Oct 30 01:04:33.584202 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:33.584173    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w4-73435a\""
Oct 30 01:04:33.616574 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:33.616476    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-73435a-766b94d975-d8ztv"]
Oct 30 01:04:33.622800 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:33.622775    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-73435a-766b94d975-d8ztv"
Oct 30 01:04:33.636318 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:33.636280    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-73435a-766b94d975-d8ztv"]
Oct 30 01:04:33.639244 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:33.639213    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60bf7933-a5cb-447a-96cc-e28389894a20-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-73435a-7645c9bc64-wcm9t\" (UID: \"60bf7933-a5cb-447a-96cc-e28389894a20\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-73435a-7645c9bc64-wcm9t"
Oct 30 01:04:33.639642 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:33.639614    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbg9g\" (UniqueName: \"kubernetes.io/projected/60bf7933-a5cb-447a-96cc-e28389894a20-kube-api-access-vbg9g\") pod \"payload-processing-e2e-shared-b-w4-73435a-7645c9bc64-wcm9t\" (UID: \"60bf7933-a5cb-447a-96cc-e28389894a20\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-73435a-7645c9bc64-wcm9t"
Oct 30 01:04:33.769404 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:33.762950    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbg9g\" (UniqueName: \"kubernetes.io/projected/60bf7933-a5cb-447a-96cc-e28389894a20-kube-api-access-vbg9g\") pod \"payload-processing-e2e-shared-b-w4-73435a-7645c9bc64-wcm9t\" (UID: \"60bf7933-a5cb-447a-96cc-e28389894a20\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-73435a-7645c9bc64-wcm9t"
Oct 30 01:04:33.769404 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:33.763193    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a02e27db-2cc5-404f-8881-6efd27b6f48a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-73435a-766b94d975-d8ztv\" (UID: \"a02e27db-2cc5-404f-8881-6efd27b6f48a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-73435a-766b94d975-d8ztv"
Oct 30 01:04:33.769404 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:33.763271    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60bf7933-a5cb-447a-96cc-e28389894a20-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-73435a-7645c9bc64-wcm9t\" (UID: \"60bf7933-a5cb-447a-96cc-e28389894a20\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-73435a-7645c9bc64-wcm9t"
Oct 30 01:04:33.769404 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:33.763328    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrj6d\" (UniqueName: \"kubernetes.io/projected/a02e27db-2cc5-404f-8881-6efd27b6f48a-kube-api-access-jrj6d\") pod \"payload-pre-processing-e2e-shared-b-w4-73435a-766b94d975-d8ztv\" (UID: \"a02e27db-2cc5-404f-8881-6efd27b6f48a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-73435a-766b94d975-d8ztv"
Oct 30 01:04:33.783418 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:33.781645    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60bf7933-a5cb-447a-96cc-e28389894a20-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-73435a-7645c9bc64-wcm9t\" (UID: \"60bf7933-a5cb-447a-96cc-e28389894a20\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-73435a-7645c9bc64-wcm9t"
Oct 30 01:04:33.872065 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:33.871428    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a02e27db-2cc5-404f-8881-6efd27b6f48a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-73435a-766b94d975-d8ztv\" (UID: \"a02e27db-2cc5-404f-8881-6efd27b6f48a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-73435a-766b94d975-d8ztv"
Oct 30 01:04:33.872065 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:33.871561    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrj6d\" (UniqueName: \"kubernetes.io/projected/a02e27db-2cc5-404f-8881-6efd27b6f48a-kube-api-access-jrj6d\") pod \"payload-pre-processing-e2e-shared-b-w4-73435a-766b94d975-d8ztv\" (UID: \"a02e27db-2cc5-404f-8881-6efd27b6f48a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-73435a-766b94d975-d8ztv"
Oct 30 01:04:33.873821 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:33.873053    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a02e27db-2cc5-404f-8881-6efd27b6f48a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-73435a-766b94d975-d8ztv\" (UID: \"a02e27db-2cc5-404f-8881-6efd27b6f48a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-73435a-766b94d975-d8ztv"
Oct 30 01:04:33.889404 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:33.875329    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbg9g\" (UniqueName: \"kubernetes.io/projected/60bf7933-a5cb-447a-96cc-e28389894a20-kube-api-access-vbg9g\") pod \"payload-processing-e2e-shared-b-w4-73435a-7645c9bc64-wcm9t\" (UID: \"60bf7933-a5cb-447a-96cc-e28389894a20\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-73435a-7645c9bc64-wcm9t"
Oct 30 01:04:33.940624 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:33.933878    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrj6d\" (UniqueName: \"kubernetes.io/projected/a02e27db-2cc5-404f-8881-6efd27b6f48a-kube-api-access-jrj6d\") pod \"payload-pre-processing-e2e-shared-b-w4-73435a-766b94d975-d8ztv\" (UID: \"a02e27db-2cc5-404f-8881-6efd27b6f48a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-73435a-766b94d975-d8ztv"
Oct 30 01:04:33.944402 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:33.944017    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-73435a-766b94d975-d8ztv"
Oct 30 01:04:34.184474 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:34.184421    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-73435a-7645c9bc64-wcm9t"
Oct 30 01:04:34.216992 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:34.216956    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-19bc68/test-model-19bc68-kserve-77fd689c7c-qw7lc" event={"ID":"076dfc74-4c57-46c9-946f-f06e3fb77f79","Type":"ContainerStarted","Data":"09f085c8afdf1111d2dc08b2cbc5d3cccc79dec9aad9ad2fcec4e30737e4a26d"}
Oct 30 01:04:34.273477 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:34.264347    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-19bc68/test-model-19bc68-kserve-77fd689c7c-qw7lc" podStartSLOduration=4.264328196 podStartE2EDuration="4.264328196s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:04:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:04:34.262637559 +0000 UTC m=+1969.879070449" watchObservedRunningTime="2026-10-30 01:04:34.264328196 +0000 UTC m=+1969.880761090"
Oct 30 01:04:34.317653 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:34.317624    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-19bc68/test-model-19bc68-kserve-77fd689c7c-qw7lc"
Oct 30 01:04:34.473671 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:34.473638    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-73435a-766b94d975-d8ztv"]
Oct 30 01:04:34.552375 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:34.552337    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-8dbe73-openshift-default-6c44d95ddf-5hz6q"]
Oct 30 01:04:34.552812 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:34.552609    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-8dbe73-openshift-default-6c44d95ddf-5hz6q" podUID="1383e696-ad5a-4b3f-900c-a987399f6bec" containerName="istio-proxy" containerID="cri-o://fda3e2253aa699de01587ab3951226395d7f6cecaa8da8b485e032a93e57be3b" gracePeriod=30
Oct 30 01:04:34.813635 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:34.813603    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-73435a-7645c9bc64-wcm9t"]
Oct 30 01:04:35.237404 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:35.224768    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-84b89f-dknxv"]
Oct 30 01:04:35.495553 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:35.490675    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-84b89f-dknxv"]
Oct 30 01:04:35.495553 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:35.490811    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-84b89f-dknxv"
Oct 30 01:04:35.527350 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:35.525686    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:04:35.527350 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:35.525775    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:04:35.527350 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:35.525888    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5phsh\""
Oct 30 01:04:35.527350 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:35.526652    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:04:35.580538 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:35.554654    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/711f2f5e-e661-488a-be49-6e85d9673f24-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-84b89f-dknxv\" (UID: \"711f2f5e-e661-488a-be49-6e85d9673f24\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-84b89f-dknxv"
Oct 30 01:04:35.580538 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:35.554738    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/711f2f5e-e661-488a-be49-6e85d9673f24-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-84b89f-dknxv\" (UID: \"711f2f5e-e661-488a-be49-6e85d9673f24\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-84b89f-dknxv"
Oct 30 01:04:35.580538 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:35.554805    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kc9xk\" (UniqueName: \"kubernetes.io/projected/711f2f5e-e661-488a-be49-6e85d9673f24-kube-api-access-kc9xk\") pod \"maas-api-revoke-keys-e2e-inf-b-84b89f-dknxv\" (UID: \"711f2f5e-e661-488a-be49-6e85d9673f24\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-84b89f-dknxv"
Oct 30 01:04:35.699921 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:35.680441    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/711f2f5e-e661-488a-be49-6e85d9673f24-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-84b89f-dknxv\" (UID: \"711f2f5e-e661-488a-be49-6e85d9673f24\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-84b89f-dknxv"
Oct 30 01:04:35.699921 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:35.680523    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/711f2f5e-e661-488a-be49-6e85d9673f24-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-84b89f-dknxv\" (UID: \"711f2f5e-e661-488a-be49-6e85d9673f24\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-84b89f-dknxv"
Oct 30 01:04:35.699921 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:35.680597    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kc9xk\" (UniqueName: \"kubernetes.io/projected/711f2f5e-e661-488a-be49-6e85d9673f24-kube-api-access-kc9xk\") pod \"maas-api-revoke-keys-e2e-inf-b-84b89f-dknxv\" (UID: \"711f2f5e-e661-488a-be49-6e85d9673f24\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-84b89f-dknxv"
Oct 30 01:04:35.711115 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:35.710136    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/711f2f5e-e661-488a-be49-6e85d9673f24-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-84b89f-dknxv\" (UID: \"711f2f5e-e661-488a-be49-6e85d9673f24\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-84b89f-dknxv"
Oct 30 01:04:35.725043 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:35.724848    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kc9xk\" (UniqueName: \"kubernetes.io/projected/711f2f5e-e661-488a-be49-6e85d9673f24-kube-api-access-kc9xk\") pod \"maas-api-revoke-keys-e2e-inf-b-84b89f-dknxv\" (UID: \"711f2f5e-e661-488a-be49-6e85d9673f24\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-84b89f-dknxv"
Oct 30 01:04:35.756696 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:35.756615    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-73435a-7645c9bc64-wcm9t" event={"ID":"60bf7933-a5cb-447a-96cc-e28389894a20","Type":"ContainerStarted","Data":"2bf22295b61b1631141841905a8dbc599177db0855ba6397d02287bbbd958e47"}
Oct 30 01:04:35.761827 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:35.761794    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-73435a-766b94d975-d8ztv" event={"ID":"a02e27db-2cc5-404f-8881-6efd27b6f48a","Type":"ContainerStarted","Data":"ae3ab6cc2a1adc5d05878cd80fb856333b98280bfd735d3b21b250cb1185a37a"}
Oct 30 01:04:35.762068 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:35.762048    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-73435a-766b94d975-d8ztv" event={"ID":"a02e27db-2cc5-404f-8881-6efd27b6f48a","Type":"ContainerStarted","Data":"39f0f7487f982bfcd27d88f5ee72f1d68f9a4a8f022111413ec11297d1f372c3"}
Oct 30 01:04:35.762658 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:35.762635    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-73435a-766b94d975-d8ztv"
Oct 30 01:04:35.797869 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:35.797835    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/711f2f5e-e661-488a-be49-6e85d9673f24-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-84b89f-dknxv\" (UID: \"711f2f5e-e661-488a-be49-6e85d9673f24\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-84b89f-dknxv"
Oct 30 01:04:35.835191 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:35.835112    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-73435a-766b94d975-d8ztv" podStartSLOduration=2.835091462 podStartE2EDuration="2.835091462s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:04:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:04:35.809246185 +0000 UTC m=+1971.425679085" watchObservedRunningTime="2026-10-30 01:04:35.835091462 +0000 UTC m=+1971.451524344"
Oct 30 01:04:35.838795 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:35.838760    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-03d892-x4vt9"]
Oct 30 01:04:35.847018 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:35.846987    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-03d892-x4vt9"]
Oct 30 01:04:36.001553 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:36.001517    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0eae29ca-mmdxp"]
Oct 30 01:04:36.005916 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:36.005893    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0eae29ca-mmdxp"
Oct 30 01:04:36.046689 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:36.046647    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0eae29ca-mmdxp"]
Oct 30 01:04:36.053930 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:36.053901    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-84b89f-dknxv"
Oct 30 01:04:36.107730 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:36.107126    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2a2f913-82ba-406c-9cd8-3fb4d39d0763-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0eae29ca-mmdxp\" (UID: \"e2a2f913-82ba-406c-9cd8-3fb4d39d0763\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0eae29ca-mmdxp"
Oct 30 01:04:36.107730 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:36.107247    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svdbj\" (UniqueName: \"kubernetes.io/projected/e2a2f913-82ba-406c-9cd8-3fb4d39d0763-kube-api-access-svdbj\") pod \"maas-api-revoke-sub-0eae29ca-mmdxp\" (UID: \"e2a2f913-82ba-406c-9cd8-3fb4d39d0763\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0eae29ca-mmdxp"
Oct 30 01:04:36.107730 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:36.107512    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2a2f913-82ba-406c-9cd8-3fb4d39d0763-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0eae29ca-mmdxp\" (UID: \"e2a2f913-82ba-406c-9cd8-3fb4d39d0763\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0eae29ca-mmdxp"
Oct 30 01:04:36.210297 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:36.208530    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2a2f913-82ba-406c-9cd8-3fb4d39d0763-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0eae29ca-mmdxp\" (UID: \"e2a2f913-82ba-406c-9cd8-3fb4d39d0763\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0eae29ca-mmdxp"
Oct 30 01:04:36.210297 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:36.208669    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2a2f913-82ba-406c-9cd8-3fb4d39d0763-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0eae29ca-mmdxp\" (UID: \"e2a2f913-82ba-406c-9cd8-3fb4d39d0763\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0eae29ca-mmdxp"
Oct 30 01:04:36.210297 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:36.208738    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svdbj\" (UniqueName: \"kubernetes.io/projected/e2a2f913-82ba-406c-9cd8-3fb4d39d0763-kube-api-access-svdbj\") pod \"maas-api-revoke-sub-0eae29ca-mmdxp\" (UID: \"e2a2f913-82ba-406c-9cd8-3fb4d39d0763\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0eae29ca-mmdxp"
Oct 30 01:04:36.210297 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:36.210087    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2a2f913-82ba-406c-9cd8-3fb4d39d0763-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0eae29ca-mmdxp\" (UID: \"e2a2f913-82ba-406c-9cd8-3fb4d39d0763\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0eae29ca-mmdxp"
Oct 30 01:04:36.222961 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:36.222924    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2a2f913-82ba-406c-9cd8-3fb4d39d0763-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0eae29ca-mmdxp\" (UID: \"e2a2f913-82ba-406c-9cd8-3fb4d39d0763\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0eae29ca-mmdxp"
Oct 30 01:04:36.244163 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:36.244071    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-svdbj\" (UniqueName: \"kubernetes.io/projected/e2a2f913-82ba-406c-9cd8-3fb4d39d0763-kube-api-access-svdbj\") pod \"maas-api-revoke-sub-0eae29ca-mmdxp\" (UID: \"e2a2f913-82ba-406c-9cd8-3fb4d39d0763\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0eae29ca-mmdxp"
Oct 30 01:04:36.329163 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:36.329077    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0eae29ca-mmdxp"
Oct 30 01:04:36.430462 ip-10-0-128-7 kubenswrapper[5002]: W1030 01:04:36.429940    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod711f2f5e_e661_488a_be49_6e85d9673f24.slice/crio-dd768ba33584a120e66a539a1dec6018589c8ba91e6fe2f4a74bf871e8f920d6 WatchSource:0}: Error finding container dd768ba33584a120e66a539a1dec6018589c8ba91e6fe2f4a74bf871e8f920d6: Status 404 returned error can't find the container with id dd768ba33584a120e66a539a1dec6018589c8ba91e6fe2f4a74bf871e8f920d6
Oct 30 01:04:36.437226 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:36.436225    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-84b89f-dknxv"]
Oct 30 01:04:36.505729 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:36.505693    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="304bba5e-2064-4901-bc87-4a770f57e5f9" path="/var/lib/kubelet/pods/304bba5e-2064-4901-bc87-4a770f57e5f9/volumes"
Oct 30 01:04:36.662452 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:36.662415    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0eae29ca-mmdxp"]
Oct 30 01:04:36.663953 ip-10-0-128-7 kubenswrapper[5002]: W1030 01:04:36.663924    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2a2f913_82ba_406c_9cd8_3fb4d39d0763.slice/crio-ff1649371bf8f1a6fb7b58750dbdb3a7251bd8269c1e546d35c0459f75b1d5ff WatchSource:0}: Error finding container ff1649371bf8f1a6fb7b58750dbdb3a7251bd8269c1e546d35c0459f75b1d5ff: Status 404 returned error can't find the container with id ff1649371bf8f1a6fb7b58750dbdb3a7251bd8269c1e546d35c0459f75b1d5ff
Oct 30 01:04:36.769971 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:36.769926    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-73435a-7645c9bc64-wcm9t" event={"ID":"60bf7933-a5cb-447a-96cc-e28389894a20","Type":"ContainerStarted","Data":"3f04f15f187a63d291d20282ba8ba29ee3840ad9cf05ebe52dec8927a08f0e60"}
Oct 30 01:04:36.770172 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:36.770003    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-73435a-7645c9bc64-wcm9t"
Oct 30 01:04:36.771463 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:36.771432    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-84b89f-dknxv" event={"ID":"711f2f5e-e661-488a-be49-6e85d9673f24","Type":"ContainerStarted","Data":"dd768ba33584a120e66a539a1dec6018589c8ba91e6fe2f4a74bf871e8f920d6"}
Oct 30 01:04:36.772877 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:36.772849    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0eae29ca-mmdxp" event={"ID":"e2a2f913-82ba-406c-9cd8-3fb4d39d0763","Type":"ContainerStarted","Data":"ff1649371bf8f1a6fb7b58750dbdb3a7251bd8269c1e546d35c0459f75b1d5ff"}
Oct 30 01:04:36.818015 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:36.817963    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-73435a-7645c9bc64-wcm9t" podStartSLOduration=3.817949624 podStartE2EDuration="3.817949624s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:04:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:04:36.812988201 +0000 UTC m=+1972.429421079" watchObservedRunningTime="2026-10-30 01:04:36.817949624 +0000 UTC m=+1972.434382504"
Oct 30 01:04:37.786418 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:37.786317    5002 generic.go:356] "Generic (PLEG): container finished" podID="711f2f5e-e661-488a-be49-6e85d9673f24" containerID="6353abe176e6dd1b4348592b656844dcb228b9911c894f261b980df7f6cb1a10" exitCode=0
Oct 30 01:04:37.786850 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:37.786407    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-84b89f-dknxv" event={"ID":"711f2f5e-e661-488a-be49-6e85d9673f24","Type":"ContainerDied","Data":"6353abe176e6dd1b4348592b656844dcb228b9911c894f261b980df7f6cb1a10"}
Oct 30 01:04:37.788288 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:37.788261    5002 generic.go:356] "Generic (PLEG): container finished" podID="e2a2f913-82ba-406c-9cd8-3fb4d39d0763" containerID="3bc612d98ed2cfb7adb8e8ca53bf5bbad1f7cf1375bd9b656e839a2c9af965eb" exitCode=0
Oct 30 01:04:37.788457 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:37.788345    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0eae29ca-mmdxp" event={"ID":"e2a2f913-82ba-406c-9cd8-3fb4d39d0763","Type":"ContainerDied","Data":"3bc612d98ed2cfb7adb8e8ca53bf5bbad1f7cf1375bd9b656e839a2c9af965eb"}
Oct 30 01:04:39.013467 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:39.013432    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0eae29ca-mmdxp"
Oct 30 01:04:39.100808 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:39.093774    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2a2f913-82ba-406c-9cd8-3fb4d39d0763-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2a2f913-82ba-406c-9cd8-3fb4d39d0763-maas-api-openshift-ca\") pod \"e2a2f913-82ba-406c-9cd8-3fb4d39d0763\" (UID: \"e2a2f913-82ba-406c-9cd8-3fb4d39d0763\") "
Oct 30 01:04:39.100808 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:39.093829    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2a2f913-82ba-406c-9cd8-3fb4d39d0763-kube-api-access-svdbj\" (UniqueName: \"kubernetes.io/projected/e2a2f913-82ba-406c-9cd8-3fb4d39d0763-kube-api-access-svdbj\") pod \"e2a2f913-82ba-406c-9cd8-3fb4d39d0763\" (UID: \"e2a2f913-82ba-406c-9cd8-3fb4d39d0763\") "
Oct 30 01:04:39.100808 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:39.093903    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e2a2f913-82ba-406c-9cd8-3fb4d39d0763-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2a2f913-82ba-406c-9cd8-3fb4d39d0763-maas-api-xks-ca\") pod \"e2a2f913-82ba-406c-9cd8-3fb4d39d0763\" (UID: \"e2a2f913-82ba-406c-9cd8-3fb4d39d0763\") "
Oct 30 01:04:39.101144 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:39.101109    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2a2f913-82ba-406c-9cd8-3fb4d39d0763-maas-api-openshift-ca" pod "e2a2f913-82ba-406c-9cd8-3fb4d39d0763" (UID: "e2a2f913-82ba-406c-9cd8-3fb4d39d0763"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:04:39.102011 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:39.101985    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-84b89f-dknxv"
Oct 30 01:04:39.103552 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:39.103523    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2a2f913-82ba-406c-9cd8-3fb4d39d0763-maas-api-xks-ca" pod "e2a2f913-82ba-406c-9cd8-3fb4d39d0763" (UID: "e2a2f913-82ba-406c-9cd8-3fb4d39d0763"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:04:39.104939 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:39.104915    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2a2f913-82ba-406c-9cd8-3fb4d39d0763-kube-api-access-svdbj" pod "e2a2f913-82ba-406c-9cd8-3fb4d39d0763" (UID: "e2a2f913-82ba-406c-9cd8-3fb4d39d0763"). InnerVolumeSpecName "kube-api-access-svdbj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:04:39.197276 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:39.197239    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/711f2f5e-e661-488a-be49-6e85d9673f24-kube-api-access-kc9xk\" (UniqueName: \"kubernetes.io/projected/711f2f5e-e661-488a-be49-6e85d9673f24-kube-api-access-kc9xk\") pod \"711f2f5e-e661-488a-be49-6e85d9673f24\" (UID: \"711f2f5e-e661-488a-be49-6e85d9673f24\") "
Oct 30 01:04:39.197474 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:39.197361    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/711f2f5e-e661-488a-be49-6e85d9673f24-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/711f2f5e-e661-488a-be49-6e85d9673f24-maas-api-xks-ca\") pod \"711f2f5e-e661-488a-be49-6e85d9673f24\" (UID: \"711f2f5e-e661-488a-be49-6e85d9673f24\") "
Oct 30 01:04:39.197575 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:39.197533    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/711f2f5e-e661-488a-be49-6e85d9673f24-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/711f2f5e-e661-488a-be49-6e85d9673f24-maas-api-openshift-ca\") pod \"711f2f5e-e661-488a-be49-6e85d9673f24\" (UID: \"711f2f5e-e661-488a-be49-6e85d9673f24\") "
Oct 30 01:04:39.198213 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:39.198193    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2a2f913-82ba-406c-9cd8-3fb4d39d0763-maas-api-openshift-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:04:39.198213 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:39.198221    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-svdbj\" (UniqueName: \"kubernetes.io/projected/e2a2f913-82ba-406c-9cd8-3fb4d39d0763-kube-api-access-svdbj\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:04:39.198341 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:39.198237    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2a2f913-82ba-406c-9cd8-3fb4d39d0763-maas-api-xks-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:04:39.198905 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:39.198881    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/711f2f5e-e661-488a-be49-6e85d9673f24-maas-api-openshift-ca" pod "711f2f5e-e661-488a-be49-6e85d9673f24" (UID: "711f2f5e-e661-488a-be49-6e85d9673f24"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:04:39.201030 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:39.201002    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/711f2f5e-e661-488a-be49-6e85d9673f24-maas-api-xks-ca" pod "711f2f5e-e661-488a-be49-6e85d9673f24" (UID: "711f2f5e-e661-488a-be49-6e85d9673f24"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:04:39.203537 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:39.203509    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/711f2f5e-e661-488a-be49-6e85d9673f24-kube-api-access-kc9xk" pod "711f2f5e-e661-488a-be49-6e85d9673f24" (UID: "711f2f5e-e661-488a-be49-6e85d9673f24"). InnerVolumeSpecName "kube-api-access-kc9xk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:04:39.299681 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:39.299641    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/711f2f5e-e661-488a-be49-6e85d9673f24-maas-api-openshift-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:04:39.299681 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:39.299680    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kc9xk\" (UniqueName: \"kubernetes.io/projected/711f2f5e-e661-488a-be49-6e85d9673f24-kube-api-access-kc9xk\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:04:39.299893 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:39.299693    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/711f2f5e-e661-488a-be49-6e85d9673f24-maas-api-xks-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:04:39.809097 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:39.809066    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-84b89f-dknxv"
Oct 30 01:04:39.809270 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:39.809065    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-84b89f-dknxv" event={"ID":"711f2f5e-e661-488a-be49-6e85d9673f24","Type":"ContainerDied","Data":"dd768ba33584a120e66a539a1dec6018589c8ba91e6fe2f4a74bf871e8f920d6"}
Oct 30 01:04:39.809270 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:39.809218    5002 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dd768ba33584a120e66a539a1dec6018589c8ba91e6fe2f4a74bf871e8f920d6"
Oct 30 01:04:39.810888 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:39.810855    5002 generic.go:356] "Generic (PLEG): container finished" podID="1383e696-ad5a-4b3f-900c-a987399f6bec" containerID="fda3e2253aa699de01587ab3951226395d7f6cecaa8da8b485e032a93e57be3b" exitCode=0
Oct 30 01:04:39.811000 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:39.810924    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8dbe73-openshift-default-6c44d95ddf-5hz6q" event={"ID":"1383e696-ad5a-4b3f-900c-a987399f6bec","Type":"ContainerDied","Data":"fda3e2253aa699de01587ab3951226395d7f6cecaa8da8b485e032a93e57be3b"}
Oct 30 01:04:39.812473 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:39.812448    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0eae29ca-mmdxp" event={"ID":"e2a2f913-82ba-406c-9cd8-3fb4d39d0763","Type":"ContainerDied","Data":"ff1649371bf8f1a6fb7b58750dbdb3a7251bd8269c1e546d35c0459f75b1d5ff"}
Oct 30 01:04:39.812576 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:39.812479    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0eae29ca-mmdxp"
Oct 30 01:04:39.812576 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:39.812493    5002 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ff1649371bf8f1a6fb7b58750dbdb3a7251bd8269c1e546d35c0459f75b1d5ff"
Oct 30 01:04:40.319067 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:40.319040    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-8dbe73-openshift-default-6c44d95ddf-5hz6q"
Oct 30 01:04:40.439416 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:40.439362    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1383e696-ad5a-4b3f-900c-a987399f6bec-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1383e696-ad5a-4b3f-900c-a987399f6bec-istio-podinfo\") pod \"1383e696-ad5a-4b3f-900c-a987399f6bec\" (UID: \"1383e696-ad5a-4b3f-900c-a987399f6bec\") "
Oct 30 01:04:40.439602 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:40.439479    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1383e696-ad5a-4b3f-900c-a987399f6bec-istio-token\" (UniqueName: \"kubernetes.io/projected/1383e696-ad5a-4b3f-900c-a987399f6bec-istio-token\") pod \"1383e696-ad5a-4b3f-900c-a987399f6bec\" (UID: \"1383e696-ad5a-4b3f-900c-a987399f6bec\") "
Oct 30 01:04:40.439602 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:40.439514    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1383e696-ad5a-4b3f-900c-a987399f6bec-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1383e696-ad5a-4b3f-900c-a987399f6bec-istio-envoy\") pod \"1383e696-ad5a-4b3f-900c-a987399f6bec\" (UID: \"1383e696-ad5a-4b3f-900c-a987399f6bec\") "
Oct 30 01:04:40.439790 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:40.439764    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1383e696-ad5a-4b3f-900c-a987399f6bec-kube-api-access-dsqj6\" (UniqueName: \"kubernetes.io/projected/1383e696-ad5a-4b3f-900c-a987399f6bec-kube-api-access-dsqj6\") pod \"1383e696-ad5a-4b3f-900c-a987399f6bec\" (UID: \"1383e696-ad5a-4b3f-900c-a987399f6bec\") "
Oct 30 01:04:40.439947 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:40.439804    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1383e696-ad5a-4b3f-900c-a987399f6bec-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1383e696-ad5a-4b3f-900c-a987399f6bec-workload-certs\") pod \"1383e696-ad5a-4b3f-900c-a987399f6bec\" (UID: \"1383e696-ad5a-4b3f-900c-a987399f6bec\") "
Oct 30 01:04:40.439947 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:40.439839    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1383e696-ad5a-4b3f-900c-a987399f6bec-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1383e696-ad5a-4b3f-900c-a987399f6bec-credential-socket\") pod \"1383e696-ad5a-4b3f-900c-a987399f6bec\" (UID: \"1383e696-ad5a-4b3f-900c-a987399f6bec\") "
Oct 30 01:04:40.439947 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:40.439885    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1383e696-ad5a-4b3f-900c-a987399f6bec-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1383e696-ad5a-4b3f-900c-a987399f6bec-istiod-ca-cert\") pod \"1383e696-ad5a-4b3f-900c-a987399f6bec\" (UID: \"1383e696-ad5a-4b3f-900c-a987399f6bec\") "
Oct 30 01:04:40.440169 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:40.439955    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1383e696-ad5a-4b3f-900c-a987399f6bec-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1383e696-ad5a-4b3f-900c-a987399f6bec-istio-data\") pod \"1383e696-ad5a-4b3f-900c-a987399f6bec\" (UID: \"1383e696-ad5a-4b3f-900c-a987399f6bec\") "
Oct 30 01:04:40.440169 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:40.440001    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1383e696-ad5a-4b3f-900c-a987399f6bec-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1383e696-ad5a-4b3f-900c-a987399f6bec-workload-socket\") pod \"1383e696-ad5a-4b3f-900c-a987399f6bec\" (UID: \"1383e696-ad5a-4b3f-900c-a987399f6bec\") "
Oct 30 01:04:40.441244 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:40.440718    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1383e696-ad5a-4b3f-900c-a987399f6bec-istio-data" pod "1383e696-ad5a-4b3f-900c-a987399f6bec" (UID: "1383e696-ad5a-4b3f-900c-a987399f6bec"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:04:40.441244 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:40.440934    5002 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1383e696-ad5a-4b3f-900c-a987399f6bec-istio-data\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:04:40.441244 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:40.440973    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1383e696-ad5a-4b3f-900c-a987399f6bec-workload-socket" pod "1383e696-ad5a-4b3f-900c-a987399f6bec" (UID: "1383e696-ad5a-4b3f-900c-a987399f6bec"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:04:40.441244 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:40.440994    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1383e696-ad5a-4b3f-900c-a987399f6bec-workload-certs" pod "1383e696-ad5a-4b3f-900c-a987399f6bec" (UID: "1383e696-ad5a-4b3f-900c-a987399f6bec"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:04:40.441244 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:40.441037    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1383e696-ad5a-4b3f-900c-a987399f6bec-credential-socket" pod "1383e696-ad5a-4b3f-900c-a987399f6bec" (UID: "1383e696-ad5a-4b3f-900c-a987399f6bec"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:04:40.441244 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:40.441212    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1383e696-ad5a-4b3f-900c-a987399f6bec-istiod-ca-cert" pod "1383e696-ad5a-4b3f-900c-a987399f6bec" (UID: "1383e696-ad5a-4b3f-900c-a987399f6bec"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:04:40.442253 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:40.442224    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1383e696-ad5a-4b3f-900c-a987399f6bec-kube-api-access-dsqj6" pod "1383e696-ad5a-4b3f-900c-a987399f6bec" (UID: "1383e696-ad5a-4b3f-900c-a987399f6bec"). InnerVolumeSpecName "kube-api-access-dsqj6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:04:40.442357 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:40.442284    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1383e696-ad5a-4b3f-900c-a987399f6bec-istio-token" pod "1383e696-ad5a-4b3f-900c-a987399f6bec" (UID: "1383e696-ad5a-4b3f-900c-a987399f6bec"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:04:40.442649 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:40.442629    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1383e696-ad5a-4b3f-900c-a987399f6bec-istio-envoy" pod "1383e696-ad5a-4b3f-900c-a987399f6bec" (UID: "1383e696-ad5a-4b3f-900c-a987399f6bec"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:04:40.442726 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:40.442641    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1383e696-ad5a-4b3f-900c-a987399f6bec-istio-podinfo" pod "1383e696-ad5a-4b3f-900c-a987399f6bec" (UID: "1383e696-ad5a-4b3f-900c-a987399f6bec"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:04:40.542365 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:40.542293    5002 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1383e696-ad5a-4b3f-900c-a987399f6bec-istio-token\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:04:40.542365 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:40.542321    5002 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1383e696-ad5a-4b3f-900c-a987399f6bec-istio-envoy\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:04:40.542365 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:40.542331    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dsqj6\" (UniqueName: \"kubernetes.io/projected/1383e696-ad5a-4b3f-900c-a987399f6bec-kube-api-access-dsqj6\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:04:40.542365 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:40.542340    5002 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1383e696-ad5a-4b3f-900c-a987399f6bec-workload-certs\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:04:40.542365 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:40.542348    5002 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1383e696-ad5a-4b3f-900c-a987399f6bec-credential-socket\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:04:40.542365 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:40.542356    5002 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1383e696-ad5a-4b3f-900c-a987399f6bec-istiod-ca-cert\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:04:40.542365 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:40.542364    5002 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1383e696-ad5a-4b3f-900c-a987399f6bec-workload-socket\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:04:40.542365 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:40.542372    5002 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1383e696-ad5a-4b3f-900c-a987399f6bec-istio-podinfo\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:04:40.821978 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:40.821898    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-8dbe73-openshift-default-6c44d95ddf-5hz6q"
Oct 30 01:04:40.821978 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:40.821929    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8dbe73-openshift-default-6c44d95ddf-5hz6q" event={"ID":"1383e696-ad5a-4b3f-900c-a987399f6bec","Type":"ContainerDied","Data":"470bcad9733b4a16022f43deb6b5d8a2dda925800a8bd10ebe24591f7f9cc865"}
Oct 30 01:04:40.822176 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:40.821989    5002 scope.go:122] "RemoveContainer" containerID="fda3e2253aa699de01587ab3951226395d7f6cecaa8da8b485e032a93e57be3b"
Oct 30 01:04:40.894542 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:40.894511    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-8dbe73-openshift-default-6c44d95ddf-5hz6q"]
Oct 30 01:04:40.908086 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:40.908059    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-8dbe73-openshift-default-6c44d95ddf-5hz6q"]
Oct 30 01:04:41.324098 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:41.319460    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-19bc68/test-model-19bc68-kserve-77fd689c7c-qw7lc"
Oct 30 01:04:41.343125 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:41.343096    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-19bc68/test-model-19bc68-kserve-77fd689c7c-qw7lc"
Oct 30 01:04:41.842495 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:41.842465    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-19bc68/test-model-19bc68-kserve-77fd689c7c-qw7lc"
Oct 30 01:04:42.498548 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:42.498519    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1383e696-ad5a-4b3f-900c-a987399f6bec" path="/var/lib/kubelet/pods/1383e696-ad5a-4b3f-900c-a987399f6bec/volumes"
Oct 30 01:04:42.823889 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:42.823807    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-03d892-openshift-default-6d4cdfbfd9-qs6xc"]
Oct 30 01:04:42.824047 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:42.824011    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-03d892-openshift-default-6d4cdfbfd9-qs6xc" podUID="7d24afeb-7f16-4e65-b159-9e29b3f32c24" containerName="istio-proxy" containerID="cri-o://a493c7d467e528da883b3ba7767ad707f50543074882fee550e4567563a8bda6" gracePeriod=30
Oct 30 01:04:42.901758 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:04:42.901717    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbjdq" containerName="extractor"
Oct 30 01:04:47.792823 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:47.792676    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-73435a-766b94d975-d8ztv"
Oct 30 01:04:47.793283 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:47.793214    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-73435a-7645c9bc64-wcm9t"
Oct 30 01:04:48.178875 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:48.178842    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-03d892-openshift-default-6d4cdfbfd9-qs6xc"
Oct 30 01:04:48.307926 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:48.307896    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d24afeb-7f16-4e65-b159-9e29b3f32c24-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d24afeb-7f16-4e65-b159-9e29b3f32c24-istiod-ca-cert\") pod \"7d24afeb-7f16-4e65-b159-9e29b3f32c24\" (UID: \"7d24afeb-7f16-4e65-b159-9e29b3f32c24\") "
Oct 30 01:04:48.308123 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:48.307961    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d24afeb-7f16-4e65-b159-9e29b3f32c24-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d24afeb-7f16-4e65-b159-9e29b3f32c24-workload-certs\") pod \"7d24afeb-7f16-4e65-b159-9e29b3f32c24\" (UID: \"7d24afeb-7f16-4e65-b159-9e29b3f32c24\") "
Oct 30 01:04:48.308123 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:48.308036    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d24afeb-7f16-4e65-b159-9e29b3f32c24-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d24afeb-7f16-4e65-b159-9e29b3f32c24-credential-socket\") pod \"7d24afeb-7f16-4e65-b159-9e29b3f32c24\" (UID: \"7d24afeb-7f16-4e65-b159-9e29b3f32c24\") "
Oct 30 01:04:48.308123 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:48.308103    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d24afeb-7f16-4e65-b159-9e29b3f32c24-istio-token\" (UniqueName: \"kubernetes.io/projected/7d24afeb-7f16-4e65-b159-9e29b3f32c24-istio-token\") pod \"7d24afeb-7f16-4e65-b159-9e29b3f32c24\" (UID: \"7d24afeb-7f16-4e65-b159-9e29b3f32c24\") "
Oct 30 01:04:48.308244 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:48.308179    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d24afeb-7f16-4e65-b159-9e29b3f32c24-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d24afeb-7f16-4e65-b159-9e29b3f32c24-workload-socket\") pod \"7d24afeb-7f16-4e65-b159-9e29b3f32c24\" (UID: \"7d24afeb-7f16-4e65-b159-9e29b3f32c24\") "
Oct 30 01:04:48.308244 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:48.308203    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d24afeb-7f16-4e65-b159-9e29b3f32c24-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d24afeb-7f16-4e65-b159-9e29b3f32c24-istio-envoy\") pod \"7d24afeb-7f16-4e65-b159-9e29b3f32c24\" (UID: \"7d24afeb-7f16-4e65-b159-9e29b3f32c24\") "
Oct 30 01:04:48.308317 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:48.308285    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d24afeb-7f16-4e65-b159-9e29b3f32c24-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d24afeb-7f16-4e65-b159-9e29b3f32c24-istio-data\") pod \"7d24afeb-7f16-4e65-b159-9e29b3f32c24\" (UID: \"7d24afeb-7f16-4e65-b159-9e29b3f32c24\") "
Oct 30 01:04:48.308368 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:48.308321    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7d24afeb-7f16-4e65-b159-9e29b3f32c24-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d24afeb-7f16-4e65-b159-9e29b3f32c24-istio-podinfo\") pod \"7d24afeb-7f16-4e65-b159-9e29b3f32c24\" (UID: \"7d24afeb-7f16-4e65-b159-9e29b3f32c24\") "
Oct 30 01:04:48.308484 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:48.308369    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d24afeb-7f16-4e65-b159-9e29b3f32c24-istiod-ca-cert" pod "7d24afeb-7f16-4e65-b159-9e29b3f32c24" (UID: "7d24afeb-7f16-4e65-b159-9e29b3f32c24"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:04:48.308484 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:48.308376    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d24afeb-7f16-4e65-b159-9e29b3f32c24-kube-api-access-zjql8\" (UniqueName: \"kubernetes.io/projected/7d24afeb-7f16-4e65-b159-9e29b3f32c24-kube-api-access-zjql8\") pod \"7d24afeb-7f16-4e65-b159-9e29b3f32c24\" (UID: \"7d24afeb-7f16-4e65-b159-9e29b3f32c24\") "
Oct 30 01:04:48.308826 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:48.308409    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d24afeb-7f16-4e65-b159-9e29b3f32c24-workload-certs" pod "7d24afeb-7f16-4e65-b159-9e29b3f32c24" (UID: "7d24afeb-7f16-4e65-b159-9e29b3f32c24"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:04:48.308926 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:48.308644    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d24afeb-7f16-4e65-b159-9e29b3f32c24-workload-socket" pod "7d24afeb-7f16-4e65-b159-9e29b3f32c24" (UID: "7d24afeb-7f16-4e65-b159-9e29b3f32c24"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:04:48.309132 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:48.308705    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d24afeb-7f16-4e65-b159-9e29b3f32c24-istio-data" pod "7d24afeb-7f16-4e65-b159-9e29b3f32c24" (UID: "7d24afeb-7f16-4e65-b159-9e29b3f32c24"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:04:48.309211 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:48.309156    5002 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d24afeb-7f16-4e65-b159-9e29b3f32c24-istiod-ca-cert\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:04:48.309211 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:48.309177    5002 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d24afeb-7f16-4e65-b159-9e29b3f32c24-workload-certs\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:04:48.309567 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:48.309341    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d24afeb-7f16-4e65-b159-9e29b3f32c24-credential-socket" pod "7d24afeb-7f16-4e65-b159-9e29b3f32c24" (UID: "7d24afeb-7f16-4e65-b159-9e29b3f32c24"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:04:48.311103 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:48.311082    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d24afeb-7f16-4e65-b159-9e29b3f32c24-istio-envoy" pod "7d24afeb-7f16-4e65-b159-9e29b3f32c24" (UID: "7d24afeb-7f16-4e65-b159-9e29b3f32c24"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:04:48.311319 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:48.311300    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d24afeb-7f16-4e65-b159-9e29b3f32c24-istio-token" pod "7d24afeb-7f16-4e65-b159-9e29b3f32c24" (UID: "7d24afeb-7f16-4e65-b159-9e29b3f32c24"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:04:48.312109 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:48.312083    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7d24afeb-7f16-4e65-b159-9e29b3f32c24-istio-podinfo" pod "7d24afeb-7f16-4e65-b159-9e29b3f32c24" (UID: "7d24afeb-7f16-4e65-b159-9e29b3f32c24"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:04:48.312283 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:48.312259    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d24afeb-7f16-4e65-b159-9e29b3f32c24-kube-api-access-zjql8" pod "7d24afeb-7f16-4e65-b159-9e29b3f32c24" (UID: "7d24afeb-7f16-4e65-b159-9e29b3f32c24"). InnerVolumeSpecName "kube-api-access-zjql8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:04:48.410974 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:48.410899    5002 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d24afeb-7f16-4e65-b159-9e29b3f32c24-credential-socket\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:04:48.410974 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:48.410928    5002 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d24afeb-7f16-4e65-b159-9e29b3f32c24-istio-token\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:04:48.410974 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:48.410937    5002 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d24afeb-7f16-4e65-b159-9e29b3f32c24-workload-socket\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:04:48.410974 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:48.410946    5002 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d24afeb-7f16-4e65-b159-9e29b3f32c24-istio-envoy\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:04:48.410974 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:48.410954    5002 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d24afeb-7f16-4e65-b159-9e29b3f32c24-istio-data\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:04:48.410974 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:48.410963    5002 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d24afeb-7f16-4e65-b159-9e29b3f32c24-istio-podinfo\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:04:48.410974 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:48.410971    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zjql8\" (UniqueName: \"kubernetes.io/projected/7d24afeb-7f16-4e65-b159-9e29b3f32c24-kube-api-access-zjql8\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:04:48.945397 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:48.945343    5002 generic.go:356] "Generic (PLEG): container finished" podID="7d24afeb-7f16-4e65-b159-9e29b3f32c24" containerID="a493c7d467e528da883b3ba7767ad707f50543074882fee550e4567563a8bda6" exitCode=0
Oct 30 01:04:48.945820 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:48.945411    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-03d892-openshift-default-6d4cdfbfd9-qs6xc"
Oct 30 01:04:48.945820 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:48.945431    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-03d892-openshift-default-6d4cdfbfd9-qs6xc" event={"ID":"7d24afeb-7f16-4e65-b159-9e29b3f32c24","Type":"ContainerDied","Data":"a493c7d467e528da883b3ba7767ad707f50543074882fee550e4567563a8bda6"}
Oct 30 01:04:48.945820 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:48.945500    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-03d892-openshift-default-6d4cdfbfd9-qs6xc" event={"ID":"7d24afeb-7f16-4e65-b159-9e29b3f32c24","Type":"ContainerDied","Data":"e006575e708dddff37a3ccb7520b472cfcc8c3987a1427e71d62d7bf24b4f5bb"}
Oct 30 01:04:48.945820 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:48.945529    5002 scope.go:122] "RemoveContainer" containerID="a493c7d467e528da883b3ba7767ad707f50543074882fee550e4567563a8bda6"
Oct 30 01:04:48.959988 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:48.959956    5002 scope.go:122] "RemoveContainer" containerID="a493c7d467e528da883b3ba7767ad707f50543074882fee550e4567563a8bda6"
Oct 30 01:04:48.960293 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:04:48.960263    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a493c7d467e528da883b3ba7767ad707f50543074882fee550e4567563a8bda6\": container with ID starting with a493c7d467e528da883b3ba7767ad707f50543074882fee550e4567563a8bda6 not found: ID does not exist" containerID="a493c7d467e528da883b3ba7767ad707f50543074882fee550e4567563a8bda6"
Oct 30 01:04:48.960460 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:48.960305    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a493c7d467e528da883b3ba7767ad707f50543074882fee550e4567563a8bda6"} err="failed to get container status \"a493c7d467e528da883b3ba7767ad707f50543074882fee550e4567563a8bda6\": rpc error: code = NotFound desc = could not find container \"a493c7d467e528da883b3ba7767ad707f50543074882fee550e4567563a8bda6\": container with ID starting with a493c7d467e528da883b3ba7767ad707f50543074882fee550e4567563a8bda6 not found: ID does not exist"
Oct 30 01:04:49.034189 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:49.034157    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-03d892-openshift-default-6d4cdfbfd9-qs6xc"]
Oct 30 01:04:49.067132 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:49.067091    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-03d892-openshift-default-6d4cdfbfd9-qs6xc"]
Oct 30 01:04:50.264300 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:50.264263    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-009e78-54d78f6cf9-sjgcl"]
Oct 30 01:04:50.265175 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:50.265149    5002 cpu_manager.go:414] "containerMap: removing container" podUID="1383e696-ad5a-4b3f-900c-a987399f6bec" containerName="istio-proxy"
Oct 30 01:04:50.265175 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:50.265174    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1383e696-ad5a-4b3f-900c-a987399f6bec" containerName="istio-proxy"
Oct 30 01:04:50.265417 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:50.265196    5002 cpu_manager.go:414] "containerMap: removing container" podUID="711f2f5e-e661-488a-be49-6e85d9673f24" containerName="revoke-keys"
Oct 30 01:04:50.265417 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:50.265203    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="711f2f5e-e661-488a-be49-6e85d9673f24" containerName="revoke-keys"
Oct 30 01:04:50.265417 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:50.265209    5002 cpu_manager.go:414] "containerMap: removing container" podUID="e2a2f913-82ba-406c-9cd8-3fb4d39d0763" containerName="revoke-keys"
Oct 30 01:04:50.265417 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:50.265214    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2a2f913-82ba-406c-9cd8-3fb4d39d0763" containerName="revoke-keys"
Oct 30 01:04:50.265417 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:50.265231    5002 cpu_manager.go:414] "containerMap: removing container" podUID="7d24afeb-7f16-4e65-b159-9e29b3f32c24" containerName="istio-proxy"
Oct 30 01:04:50.265417 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:50.265236    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d24afeb-7f16-4e65-b159-9e29b3f32c24" containerName="istio-proxy"
Oct 30 01:04:50.265417 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:50.265354    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="1383e696-ad5a-4b3f-900c-a987399f6bec" containerName="istio-proxy"
Oct 30 01:04:50.265417 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:50.265365    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="711f2f5e-e661-488a-be49-6e85d9673f24" containerName="revoke-keys"
Oct 30 01:04:50.265417 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:50.265371    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2a2f913-82ba-406c-9cd8-3fb4d39d0763" containerName="revoke-keys"
Oct 30 01:04:50.265417 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:50.265397    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d24afeb-7f16-4e65-b159-9e29b3f32c24" containerName="istio-proxy"
Oct 30 01:04:50.270204 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:50.270179    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-009e78-54d78f6cf9-sjgcl"
Oct 30 01:04:50.287675 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:50.287647    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w6-009e78\""
Oct 30 01:04:50.287837 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:50.287817    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w6-009e78-dockercfg-lcjv9\""
Oct 30 01:04:50.290965 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:50.290939    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-009e78-54d78f6cf9-sjgcl"]
Oct 30 01:04:50.338025 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:50.337972    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rml98\" (UniqueName: \"kubernetes.io/projected/a0a93655-85f4-4cf7-9e80-2ea8ce25bcb5-kube-api-access-rml98\") pod \"payload-pre-processing-e2e-shared-a-w6-009e78-54d78f6cf9-sjgcl\" (UID: \"a0a93655-85f4-4cf7-9e80-2ea8ce25bcb5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-009e78-54d78f6cf9-sjgcl"
Oct 30 01:04:50.338253 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:50.338037    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0a93655-85f4-4cf7-9e80-2ea8ce25bcb5-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w6-009e78-54d78f6cf9-sjgcl\" (UID: \"a0a93655-85f4-4cf7-9e80-2ea8ce25bcb5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-009e78-54d78f6cf9-sjgcl"
Oct 30 01:04:50.342313 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:50.342234    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-009e78-847cbfbf7d-vgf6m"]
Oct 30 01:04:50.361584 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:50.356300    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-009e78-847cbfbf7d-vgf6m"
Oct 30 01:04:50.402414 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:50.365641    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-009e78-847cbfbf7d-vgf6m"]
Oct 30 01:04:50.483710 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:50.471246    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tv8xm\" (UniqueName: \"kubernetes.io/projected/5a0523a2-66bc-4ec0-a5bf-ac5cbafcdbfd-kube-api-access-tv8xm\") pod \"payload-processing-e2e-shared-a-w6-009e78-847cbfbf7d-vgf6m\" (UID: \"5a0523a2-66bc-4ec0-a5bf-ac5cbafcdbfd\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-009e78-847cbfbf7d-vgf6m"
Oct 30 01:04:50.483710 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:50.475600    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a0523a2-66bc-4ec0-a5bf-ac5cbafcdbfd-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w6-009e78-847cbfbf7d-vgf6m\" (UID: \"5a0523a2-66bc-4ec0-a5bf-ac5cbafcdbfd\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-009e78-847cbfbf7d-vgf6m"
Oct 30 01:04:50.483710 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:50.475826    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rml98\" (UniqueName: \"kubernetes.io/projected/a0a93655-85f4-4cf7-9e80-2ea8ce25bcb5-kube-api-access-rml98\") pod \"payload-pre-processing-e2e-shared-a-w6-009e78-54d78f6cf9-sjgcl\" (UID: \"a0a93655-85f4-4cf7-9e80-2ea8ce25bcb5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-009e78-54d78f6cf9-sjgcl"
Oct 30 01:04:50.538463 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:50.518424    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0a93655-85f4-4cf7-9e80-2ea8ce25bcb5-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w6-009e78-54d78f6cf9-sjgcl\" (UID: \"a0a93655-85f4-4cf7-9e80-2ea8ce25bcb5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-009e78-54d78f6cf9-sjgcl"
Oct 30 01:04:50.538463 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:50.519949    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0a93655-85f4-4cf7-9e80-2ea8ce25bcb5-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w6-009e78-54d78f6cf9-sjgcl\" (UID: \"a0a93655-85f4-4cf7-9e80-2ea8ce25bcb5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-009e78-54d78f6cf9-sjgcl"
Oct 30 01:04:50.620495 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:50.609904    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d24afeb-7f16-4e65-b159-9e29b3f32c24" path="/var/lib/kubelet/pods/7d24afeb-7f16-4e65-b159-9e29b3f32c24/volumes"
Oct 30 01:04:50.638742 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:50.635561    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rml98\" (UniqueName: \"kubernetes.io/projected/a0a93655-85f4-4cf7-9e80-2ea8ce25bcb5-kube-api-access-rml98\") pod \"payload-pre-processing-e2e-shared-a-w6-009e78-54d78f6cf9-sjgcl\" (UID: \"a0a93655-85f4-4cf7-9e80-2ea8ce25bcb5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-009e78-54d78f6cf9-sjgcl"
Oct 30 01:04:50.638742 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:50.636559    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tv8xm\" (UniqueName: \"kubernetes.io/projected/5a0523a2-66bc-4ec0-a5bf-ac5cbafcdbfd-kube-api-access-tv8xm\") pod \"payload-processing-e2e-shared-a-w6-009e78-847cbfbf7d-vgf6m\" (UID: \"5a0523a2-66bc-4ec0-a5bf-ac5cbafcdbfd\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-009e78-847cbfbf7d-vgf6m"
Oct 30 01:04:50.638742 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:50.636630    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a0523a2-66bc-4ec0-a5bf-ac5cbafcdbfd-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w6-009e78-847cbfbf7d-vgf6m\" (UID: \"5a0523a2-66bc-4ec0-a5bf-ac5cbafcdbfd\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-009e78-847cbfbf7d-vgf6m"
Oct 30 01:04:50.638742 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:50.637836    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a0523a2-66bc-4ec0-a5bf-ac5cbafcdbfd-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w6-009e78-847cbfbf7d-vgf6m\" (UID: \"5a0523a2-66bc-4ec0-a5bf-ac5cbafcdbfd\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-009e78-847cbfbf7d-vgf6m"
Oct 30 01:04:50.663253 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:50.663217    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tv8xm\" (UniqueName: \"kubernetes.io/projected/5a0523a2-66bc-4ec0-a5bf-ac5cbafcdbfd-kube-api-access-tv8xm\") pod \"payload-processing-e2e-shared-a-w6-009e78-847cbfbf7d-vgf6m\" (UID: \"5a0523a2-66bc-4ec0-a5bf-ac5cbafcdbfd\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-009e78-847cbfbf7d-vgf6m"
Oct 30 01:04:50.890799 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:50.890709    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-009e78-54d78f6cf9-sjgcl"
Oct 30 01:04:50.938743 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:50.938702    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-009e78-847cbfbf7d-vgf6m"
Oct 30 01:04:51.204428 ip-10-0-128-7 kubenswrapper[5002]: W1030 01:04:51.204394    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0a93655_85f4_4cf7_9e80_2ea8ce25bcb5.slice/crio-f605b853ec1bd0e754e74a585d9087e1a5d33a2561199027fae067e8cf8616b3 WatchSource:0}: Error finding container f605b853ec1bd0e754e74a585d9087e1a5d33a2561199027fae067e8cf8616b3: Status 404 returned error can't find the container with id f605b853ec1bd0e754e74a585d9087e1a5d33a2561199027fae067e8cf8616b3
Oct 30 01:04:51.206150 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:51.206043    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-009e78-54d78f6cf9-sjgcl"]
Oct 30 01:04:51.290033 ip-10-0-128-7 kubenswrapper[5002]: W1030 01:04:51.290004    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a0523a2_66bc_4ec0_a5bf_ac5cbafcdbfd.slice/crio-c2020329b9e8c2fde20084b7282dfcb7bd61a461d8990e00a29f0ec01a833442 WatchSource:0}: Error finding container c2020329b9e8c2fde20084b7282dfcb7bd61a461d8990e00a29f0ec01a833442: Status 404 returned error can't find the container with id c2020329b9e8c2fde20084b7282dfcb7bd61a461d8990e00a29f0ec01a833442
Oct 30 01:04:51.291566 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:51.291536    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-009e78-847cbfbf7d-vgf6m"]
Oct 30 01:04:51.816402 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:51.815554    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8d2a70-cxtxf"]
Oct 30 01:04:51.836406 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:51.828905    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8d2a70-cxtxf"
Oct 30 01:04:51.840401 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:51.838367    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8d2a70-cxtxf"]
Oct 30 01:04:51.856424 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:51.856186    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:04:51.858620 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:51.858576    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkbnf\" (UniqueName: \"kubernetes.io/projected/002a3d53-2925-4977-967a-5a23a655a12f-kube-api-access-nkbnf\") pod \"maas-api-revoke-keys-e2e-inf-a-8d2a70-cxtxf\" (UID: \"002a3d53-2925-4977-967a-5a23a655a12f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8d2a70-cxtxf"
Oct 30 01:04:51.858762 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:51.858654    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/002a3d53-2925-4977-967a-5a23a655a12f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-8d2a70-cxtxf\" (UID: \"002a3d53-2925-4977-967a-5a23a655a12f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8d2a70-cxtxf"
Oct 30 01:04:51.858974 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:51.858950    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/002a3d53-2925-4977-967a-5a23a655a12f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-8d2a70-cxtxf\" (UID: \"002a3d53-2925-4977-967a-5a23a655a12f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8d2a70-cxtxf"
Oct 30 01:04:51.873331 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:51.872616    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:04:51.873331 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:51.872681    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5phsh\""
Oct 30 01:04:51.873331 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:51.873273    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:04:51.960038 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:51.959999    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkbnf\" (UniqueName: \"kubernetes.io/projected/002a3d53-2925-4977-967a-5a23a655a12f-kube-api-access-nkbnf\") pod \"maas-api-revoke-keys-e2e-inf-a-8d2a70-cxtxf\" (UID: \"002a3d53-2925-4977-967a-5a23a655a12f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8d2a70-cxtxf"
Oct 30 01:04:51.960256 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:51.960073    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/002a3d53-2925-4977-967a-5a23a655a12f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-8d2a70-cxtxf\" (UID: \"002a3d53-2925-4977-967a-5a23a655a12f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8d2a70-cxtxf"
Oct 30 01:04:51.960346 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:51.960329    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/002a3d53-2925-4977-967a-5a23a655a12f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-8d2a70-cxtxf\" (UID: \"002a3d53-2925-4977-967a-5a23a655a12f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8d2a70-cxtxf"
Oct 30 01:04:51.960818 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:51.960791    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/002a3d53-2925-4977-967a-5a23a655a12f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-8d2a70-cxtxf\" (UID: \"002a3d53-2925-4977-967a-5a23a655a12f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8d2a70-cxtxf"
Oct 30 01:04:51.963089 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:51.963066    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/002a3d53-2925-4977-967a-5a23a655a12f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-8d2a70-cxtxf\" (UID: \"002a3d53-2925-4977-967a-5a23a655a12f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8d2a70-cxtxf"
Oct 30 01:04:51.986002 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:51.985966    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-009e78-847cbfbf7d-vgf6m" event={"ID":"5a0523a2-66bc-4ec0-a5bf-ac5cbafcdbfd","Type":"ContainerStarted","Data":"de20524b1394285bbc1d024f323d1fee8647693e0395af6a079b4befd317d87a"}
Oct 30 01:04:51.986167 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:51.986036    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-009e78-847cbfbf7d-vgf6m" event={"ID":"5a0523a2-66bc-4ec0-a5bf-ac5cbafcdbfd","Type":"ContainerStarted","Data":"c2020329b9e8c2fde20084b7282dfcb7bd61a461d8990e00a29f0ec01a833442"}
Oct 30 01:04:51.986233 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:51.986167    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-009e78-847cbfbf7d-vgf6m"
Oct 30 01:04:51.986847 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:51.986821    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkbnf\" (UniqueName: \"kubernetes.io/projected/002a3d53-2925-4977-967a-5a23a655a12f-kube-api-access-nkbnf\") pod \"maas-api-revoke-keys-e2e-inf-a-8d2a70-cxtxf\" (UID: \"002a3d53-2925-4977-967a-5a23a655a12f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8d2a70-cxtxf"
Oct 30 01:04:51.988074 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:51.988047    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-009e78-54d78f6cf9-sjgcl" event={"ID":"a0a93655-85f4-4cf7-9e80-2ea8ce25bcb5","Type":"ContainerStarted","Data":"70e6127b29dc6420849daa9cd497a262f43584dacb52352a76584bcfbd32c3aa"}
Oct 30 01:04:51.988198 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:51.988092    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-009e78-54d78f6cf9-sjgcl" event={"ID":"a0a93655-85f4-4cf7-9e80-2ea8ce25bcb5","Type":"ContainerStarted","Data":"f605b853ec1bd0e754e74a585d9087e1a5d33a2561199027fae067e8cf8616b3"}
Oct 30 01:04:51.988198 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:51.988124    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-009e78-54d78f6cf9-sjgcl"
Oct 30 01:04:52.033295 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:52.033234    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-009e78-847cbfbf7d-vgf6m" podStartSLOduration=2.033220114 podStartE2EDuration="2.033220114s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:04:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:04:52.031676412 +0000 UTC m=+1987.648109302" watchObservedRunningTime="2026-10-30 01:04:52.033220114 +0000 UTC m=+1987.649652992"
Oct 30 01:04:52.078373 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:52.078258    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-009e78-54d78f6cf9-sjgcl" podStartSLOduration=2.0782445 podStartE2EDuration="2.0782445s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:04:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:04:52.074736494 +0000 UTC m=+1987.691169372" watchObservedRunningTime="2026-10-30 01:04:52.0782445 +0000 UTC m=+1987.694677376"
Oct 30 01:04:52.177613 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:52.177578    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8d2a70-cxtxf"
Oct 30 01:04:52.539801 ip-10-0-128-7 kubenswrapper[5002]: W1030 01:04:52.539769    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod002a3d53_2925_4977_967a_5a23a655a12f.slice/crio-0ca5f6378e2c1fe40da179f5fe3cd6130b48ddbea37471a6eaef8eccfa09ec1a WatchSource:0}: Error finding container 0ca5f6378e2c1fe40da179f5fe3cd6130b48ddbea37471a6eaef8eccfa09ec1a: Status 404 returned error can't find the container with id 0ca5f6378e2c1fe40da179f5fe3cd6130b48ddbea37471a6eaef8eccfa09ec1a
Oct 30 01:04:52.540968 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:52.540940    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8d2a70-cxtxf"]
Oct 30 01:04:52.997850 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:52.997800    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8d2a70-cxtxf" event={"ID":"002a3d53-2925-4977-967a-5a23a655a12f","Type":"ContainerStarted","Data":"91009f3671528fd5344084e5034c38e3d4724d598f75411aae25f069d7bccaea"}
Oct 30 01:04:52.998022 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:52.997872    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8d2a70-cxtxf" event={"ID":"002a3d53-2925-4977-967a-5a23a655a12f","Type":"ContainerStarted","Data":"0ca5f6378e2c1fe40da179f5fe3cd6130b48ddbea37471a6eaef8eccfa09ec1a"}
Oct 30 01:04:53.044085 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:53.044029    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8d2a70-cxtxf" podStartSLOduration=2.04401287 podStartE2EDuration="2.04401287s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:04:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:04:53.040795219 +0000 UTC m=+1988.657228098" watchObservedRunningTime="2026-10-30 01:04:53.04401287 +0000 UTC m=+1988.660445748"
Oct 30 01:04:54.007892 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:54.007853    5002 generic.go:356] "Generic (PLEG): container finished" podID="002a3d53-2925-4977-967a-5a23a655a12f" containerID="91009f3671528fd5344084e5034c38e3d4724d598f75411aae25f069d7bccaea" exitCode=0
Oct 30 01:04:54.008353 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:54.007913    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8d2a70-cxtxf" event={"ID":"002a3d53-2925-4977-967a-5a23a655a12f","Type":"ContainerDied","Data":"91009f3671528fd5344084e5034c38e3d4724d598f75411aae25f069d7bccaea"}
Oct 30 01:04:55.204525 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:55.204489    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8d2a70-cxtxf"
Oct 30 01:04:55.318114 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:55.318034    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/002a3d53-2925-4977-967a-5a23a655a12f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/002a3d53-2925-4977-967a-5a23a655a12f-maas-api-openshift-ca\") pod \"002a3d53-2925-4977-967a-5a23a655a12f\" (UID: \"002a3d53-2925-4977-967a-5a23a655a12f\") "
Oct 30 01:04:55.318299 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:55.318127    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/002a3d53-2925-4977-967a-5a23a655a12f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/002a3d53-2925-4977-967a-5a23a655a12f-maas-api-xks-ca\") pod \"002a3d53-2925-4977-967a-5a23a655a12f\" (UID: \"002a3d53-2925-4977-967a-5a23a655a12f\") "
Oct 30 01:04:55.318299 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:55.318219    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/002a3d53-2925-4977-967a-5a23a655a12f-kube-api-access-nkbnf\" (UniqueName: \"kubernetes.io/projected/002a3d53-2925-4977-967a-5a23a655a12f-kube-api-access-nkbnf\") pod \"002a3d53-2925-4977-967a-5a23a655a12f\" (UID: \"002a3d53-2925-4977-967a-5a23a655a12f\") "
Oct 30 01:04:55.318601 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:55.318565    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/002a3d53-2925-4977-967a-5a23a655a12f-maas-api-openshift-ca" pod "002a3d53-2925-4977-967a-5a23a655a12f" (UID: "002a3d53-2925-4977-967a-5a23a655a12f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:04:55.318997 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:55.318973    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/002a3d53-2925-4977-967a-5a23a655a12f-maas-api-openshift-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:04:55.320482 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:55.320456    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/002a3d53-2925-4977-967a-5a23a655a12f-maas-api-xks-ca" pod "002a3d53-2925-4977-967a-5a23a655a12f" (UID: "002a3d53-2925-4977-967a-5a23a655a12f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:04:55.320985 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:55.320959    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/002a3d53-2925-4977-967a-5a23a655a12f-kube-api-access-nkbnf" pod "002a3d53-2925-4977-967a-5a23a655a12f" (UID: "002a3d53-2925-4977-967a-5a23a655a12f"). InnerVolumeSpecName "kube-api-access-nkbnf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:04:55.421084 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:55.421050    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/002a3d53-2925-4977-967a-5a23a655a12f-maas-api-xks-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:04:55.421084 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:55.421081    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nkbnf\" (UniqueName: \"kubernetes.io/projected/002a3d53-2925-4977-967a-5a23a655a12f-kube-api-access-nkbnf\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:04:56.027644 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:56.027614    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8d2a70-cxtxf"
Oct 30 01:04:56.027810 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:56.027645    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8d2a70-cxtxf" event={"ID":"002a3d53-2925-4977-967a-5a23a655a12f","Type":"ContainerDied","Data":"0ca5f6378e2c1fe40da179f5fe3cd6130b48ddbea37471a6eaef8eccfa09ec1a"}
Oct 30 01:04:56.027810 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:56.027701    5002 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0ca5f6378e2c1fe40da179f5fe3cd6130b48ddbea37471a6eaef8eccfa09ec1a"
Oct 30 01:04:56.330053 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:56.329972    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-549b3758-gs27m"]
Oct 30 01:04:56.330811 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:56.330790    5002 cpu_manager.go:414] "containerMap: removing container" podUID="002a3d53-2925-4977-967a-5a23a655a12f" containerName="revoke-keys"
Oct 30 01:04:56.330811 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:56.330810    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="002a3d53-2925-4977-967a-5a23a655a12f" containerName="revoke-keys"
Oct 30 01:04:56.330976 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:56.330917    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="002a3d53-2925-4977-967a-5a23a655a12f" containerName="revoke-keys"
Oct 30 01:04:56.334825 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:56.334806    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-549b3758-gs27m"
Oct 30 01:04:56.353741 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:56.353710    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-549b3758-gs27m"]
Oct 30 01:04:56.361912 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:56.361884    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:04:56.362051 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:56.361938    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:04:56.362645 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:56.362619    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5phsh\""
Oct 30 01:04:56.362765 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:56.362680    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:04:56.437930 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:56.437895    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc015bf9-5c24-4af9-b931-69203486dd49-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-549b3758-gs27m\" (UID: \"cc015bf9-5c24-4af9-b931-69203486dd49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-549b3758-gs27m"
Oct 30 01:04:56.438073 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:56.438053    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttld4\" (UniqueName: \"kubernetes.io/projected/cc015bf9-5c24-4af9-b931-69203486dd49-kube-api-access-ttld4\") pod \"maas-api-revoke-sub-549b3758-gs27m\" (UID: \"cc015bf9-5c24-4af9-b931-69203486dd49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-549b3758-gs27m"
Oct 30 01:04:56.438129 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:56.438115    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc015bf9-5c24-4af9-b931-69203486dd49-maas-api-xks-ca\") pod \"maas-api-revoke-sub-549b3758-gs27m\" (UID: \"cc015bf9-5c24-4af9-b931-69203486dd49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-549b3758-gs27m"
Oct 30 01:04:56.539532 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:56.539497    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc015bf9-5c24-4af9-b931-69203486dd49-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-549b3758-gs27m\" (UID: \"cc015bf9-5c24-4af9-b931-69203486dd49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-549b3758-gs27m"
Oct 30 01:04:56.539717 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:56.539602    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttld4\" (UniqueName: \"kubernetes.io/projected/cc015bf9-5c24-4af9-b931-69203486dd49-kube-api-access-ttld4\") pod \"maas-api-revoke-sub-549b3758-gs27m\" (UID: \"cc015bf9-5c24-4af9-b931-69203486dd49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-549b3758-gs27m"
Oct 30 01:04:56.539717 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:56.539647    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc015bf9-5c24-4af9-b931-69203486dd49-maas-api-xks-ca\") pod \"maas-api-revoke-sub-549b3758-gs27m\" (UID: \"cc015bf9-5c24-4af9-b931-69203486dd49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-549b3758-gs27m"
Oct 30 01:04:56.540144 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:56.540114    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc015bf9-5c24-4af9-b931-69203486dd49-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-549b3758-gs27m\" (UID: \"cc015bf9-5c24-4af9-b931-69203486dd49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-549b3758-gs27m"
Oct 30 01:04:56.542083 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:56.542058    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc015bf9-5c24-4af9-b931-69203486dd49-maas-api-xks-ca\") pod \"maas-api-revoke-sub-549b3758-gs27m\" (UID: \"cc015bf9-5c24-4af9-b931-69203486dd49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-549b3758-gs27m"
Oct 30 01:04:56.564273 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:56.564238    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttld4\" (UniqueName: \"kubernetes.io/projected/cc015bf9-5c24-4af9-b931-69203486dd49-kube-api-access-ttld4\") pod \"maas-api-revoke-sub-549b3758-gs27m\" (UID: \"cc015bf9-5c24-4af9-b931-69203486dd49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-549b3758-gs27m"
Oct 30 01:04:56.654440 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:56.654328    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-549b3758-gs27m"
Oct 30 01:04:56.965281 ip-10-0-128-7 kubenswrapper[5002]: W1030 01:04:56.965243    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc015bf9_5c24_4af9_b931_69203486dd49.slice/crio-af7e8c35feba2d17753e3ea9b19d19771b7e9fa918eaa7d275ac8f3746178ebe WatchSource:0}: Error finding container af7e8c35feba2d17753e3ea9b19d19771b7e9fa918eaa7d275ac8f3746178ebe: Status 404 returned error can't find the container with id af7e8c35feba2d17753e3ea9b19d19771b7e9fa918eaa7d275ac8f3746178ebe
Oct 30 01:04:56.966759 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:56.966730    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-549b3758-gs27m"]
Oct 30 01:04:57.040491 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:57.040451    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-549b3758-gs27m" event={"ID":"cc015bf9-5c24-4af9-b931-69203486dd49","Type":"ContainerStarted","Data":"af7e8c35feba2d17753e3ea9b19d19771b7e9fa918eaa7d275ac8f3746178ebe"}
Oct 30 01:04:58.049424 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:58.049367    5002 generic.go:356] "Generic (PLEG): container finished" podID="cc015bf9-5c24-4af9-b931-69203486dd49" containerID="822f817f9f3ed6f587314779dde3b475e21ce478e78104238649d77d56da2eb7" exitCode=0
Oct 30 01:04:58.049424 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:58.049409    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-549b3758-gs27m" event={"ID":"cc015bf9-5c24-4af9-b931-69203486dd49","Type":"ContainerDied","Data":"822f817f9f3ed6f587314779dde3b475e21ce478e78104238649d77d56da2eb7"}
Oct 30 01:04:58.800922 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:58.800885    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-w4gt5"]
Oct 30 01:04:58.805539 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:58.805515    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-w4gt5"
Oct 30 01:04:58.848151 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:58.848112    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-w4gt5"]
Oct 30 01:04:58.876449 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:58.876410    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e64a3a3-48e4-4e47-bc24-e24fad22c38e-istio-token\") pod \"e2e-shared-b-w1-cce513-openshift-default-84cc9879-w4gt5\" (UID: \"5e64a3a3-48e4-4e47-bc24-e24fad22c38e\") " pod="openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-w4gt5"
Oct 30 01:04:58.876624 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:58.876525    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e64a3a3-48e4-4e47-bc24-e24fad22c38e-credential-socket\") pod \"e2e-shared-b-w1-cce513-openshift-default-84cc9879-w4gt5\" (UID: \"5e64a3a3-48e4-4e47-bc24-e24fad22c38e\") " pod="openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-w4gt5"
Oct 30 01:04:58.876624 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:58.876614    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e64a3a3-48e4-4e47-bc24-e24fad22c38e-istio-podinfo\") pod \"e2e-shared-b-w1-cce513-openshift-default-84cc9879-w4gt5\" (UID: \"5e64a3a3-48e4-4e47-bc24-e24fad22c38e\") " pod="openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-w4gt5"
Oct 30 01:04:58.876726 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:58.876713    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e64a3a3-48e4-4e47-bc24-e24fad22c38e-istio-envoy\") pod \"e2e-shared-b-w1-cce513-openshift-default-84cc9879-w4gt5\" (UID: \"5e64a3a3-48e4-4e47-bc24-e24fad22c38e\") " pod="openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-w4gt5"
Oct 30 01:04:58.876764 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:58.876733    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e64a3a3-48e4-4e47-bc24-e24fad22c38e-istio-data\") pod \"e2e-shared-b-w1-cce513-openshift-default-84cc9879-w4gt5\" (UID: \"5e64a3a3-48e4-4e47-bc24-e24fad22c38e\") " pod="openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-w4gt5"
Oct 30 01:04:58.876806 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:58.876775    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q956l\" (UniqueName: \"kubernetes.io/projected/5e64a3a3-48e4-4e47-bc24-e24fad22c38e-kube-api-access-q956l\") pod \"e2e-shared-b-w1-cce513-openshift-default-84cc9879-w4gt5\" (UID: \"5e64a3a3-48e4-4e47-bc24-e24fad22c38e\") " pod="openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-w4gt5"
Oct 30 01:04:58.876848 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:58.876831    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e64a3a3-48e4-4e47-bc24-e24fad22c38e-workload-certs\") pod \"e2e-shared-b-w1-cce513-openshift-default-84cc9879-w4gt5\" (UID: \"5e64a3a3-48e4-4e47-bc24-e24fad22c38e\") " pod="openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-w4gt5"
Oct 30 01:04:58.876938 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:58.876925    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e64a3a3-48e4-4e47-bc24-e24fad22c38e-workload-socket\") pod \"e2e-shared-b-w1-cce513-openshift-default-84cc9879-w4gt5\" (UID: \"5e64a3a3-48e4-4e47-bc24-e24fad22c38e\") " pod="openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-w4gt5"
Oct 30 01:04:58.876973 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:58.876948    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e64a3a3-48e4-4e47-bc24-e24fad22c38e-istiod-ca-cert\") pod \"e2e-shared-b-w1-cce513-openshift-default-84cc9879-w4gt5\" (UID: \"5e64a3a3-48e4-4e47-bc24-e24fad22c38e\") " pod="openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-w4gt5"
Oct 30 01:04:58.978524 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:58.978487    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e64a3a3-48e4-4e47-bc24-e24fad22c38e-workload-certs\") pod \"e2e-shared-b-w1-cce513-openshift-default-84cc9879-w4gt5\" (UID: \"5e64a3a3-48e4-4e47-bc24-e24fad22c38e\") " pod="openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-w4gt5"
Oct 30 01:04:58.978738 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:58.978564    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e64a3a3-48e4-4e47-bc24-e24fad22c38e-workload-socket\") pod \"e2e-shared-b-w1-cce513-openshift-default-84cc9879-w4gt5\" (UID: \"5e64a3a3-48e4-4e47-bc24-e24fad22c38e\") " pod="openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-w4gt5"
Oct 30 01:04:58.978738 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:58.978584    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e64a3a3-48e4-4e47-bc24-e24fad22c38e-istiod-ca-cert\") pod \"e2e-shared-b-w1-cce513-openshift-default-84cc9879-w4gt5\" (UID: \"5e64a3a3-48e4-4e47-bc24-e24fad22c38e\") " pod="openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-w4gt5"
Oct 30 01:04:58.978738 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:58.978616    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e64a3a3-48e4-4e47-bc24-e24fad22c38e-istio-token\") pod \"e2e-shared-b-w1-cce513-openshift-default-84cc9879-w4gt5\" (UID: \"5e64a3a3-48e4-4e47-bc24-e24fad22c38e\") " pod="openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-w4gt5"
Oct 30 01:04:58.978738 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:58.978661    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e64a3a3-48e4-4e47-bc24-e24fad22c38e-credential-socket\") pod \"e2e-shared-b-w1-cce513-openshift-default-84cc9879-w4gt5\" (UID: \"5e64a3a3-48e4-4e47-bc24-e24fad22c38e\") " pod="openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-w4gt5"
Oct 30 01:04:58.978738 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:58.978736    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e64a3a3-48e4-4e47-bc24-e24fad22c38e-istio-podinfo\") pod \"e2e-shared-b-w1-cce513-openshift-default-84cc9879-w4gt5\" (UID: \"5e64a3a3-48e4-4e47-bc24-e24fad22c38e\") " pod="openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-w4gt5"
Oct 30 01:04:58.979072 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:58.978800    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e64a3a3-48e4-4e47-bc24-e24fad22c38e-istio-envoy\") pod \"e2e-shared-b-w1-cce513-openshift-default-84cc9879-w4gt5\" (UID: \"5e64a3a3-48e4-4e47-bc24-e24fad22c38e\") " pod="openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-w4gt5"
Oct 30 01:04:58.979072 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:58.978822    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e64a3a3-48e4-4e47-bc24-e24fad22c38e-istio-data\") pod \"e2e-shared-b-w1-cce513-openshift-default-84cc9879-w4gt5\" (UID: \"5e64a3a3-48e4-4e47-bc24-e24fad22c38e\") " pod="openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-w4gt5"
Oct 30 01:04:58.979072 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:58.978873    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q956l\" (UniqueName: \"kubernetes.io/projected/5e64a3a3-48e4-4e47-bc24-e24fad22c38e-kube-api-access-q956l\") pod \"e2e-shared-b-w1-cce513-openshift-default-84cc9879-w4gt5\" (UID: \"5e64a3a3-48e4-4e47-bc24-e24fad22c38e\") " pod="openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-w4gt5"
Oct 30 01:04:58.979072 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:58.978939    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e64a3a3-48e4-4e47-bc24-e24fad22c38e-workload-certs\") pod \"e2e-shared-b-w1-cce513-openshift-default-84cc9879-w4gt5\" (UID: \"5e64a3a3-48e4-4e47-bc24-e24fad22c38e\") " pod="openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-w4gt5"
Oct 30 01:04:58.979072 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:58.978962    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e64a3a3-48e4-4e47-bc24-e24fad22c38e-workload-socket\") pod \"e2e-shared-b-w1-cce513-openshift-default-84cc9879-w4gt5\" (UID: \"5e64a3a3-48e4-4e47-bc24-e24fad22c38e\") " pod="openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-w4gt5"
Oct 30 01:04:58.979336 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:58.979192    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e64a3a3-48e4-4e47-bc24-e24fad22c38e-credential-socket\") pod \"e2e-shared-b-w1-cce513-openshift-default-84cc9879-w4gt5\" (UID: \"5e64a3a3-48e4-4e47-bc24-e24fad22c38e\") " pod="openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-w4gt5"
Oct 30 01:04:58.979336 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:58.979220    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e64a3a3-48e4-4e47-bc24-e24fad22c38e-istio-data\") pod \"e2e-shared-b-w1-cce513-openshift-default-84cc9879-w4gt5\" (UID: \"5e64a3a3-48e4-4e47-bc24-e24fad22c38e\") " pod="openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-w4gt5"
Oct 30 01:04:58.979488 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:58.979448    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e64a3a3-48e4-4e47-bc24-e24fad22c38e-istiod-ca-cert\") pod \"e2e-shared-b-w1-cce513-openshift-default-84cc9879-w4gt5\" (UID: \"5e64a3a3-48e4-4e47-bc24-e24fad22c38e\") " pod="openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-w4gt5"
Oct 30 01:04:58.981331 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:58.981308    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e64a3a3-48e4-4e47-bc24-e24fad22c38e-istio-envoy\") pod \"e2e-shared-b-w1-cce513-openshift-default-84cc9879-w4gt5\" (UID: \"5e64a3a3-48e4-4e47-bc24-e24fad22c38e\") " pod="openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-w4gt5"
Oct 30 01:04:58.981498 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:58.981457    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e64a3a3-48e4-4e47-bc24-e24fad22c38e-istio-podinfo\") pod \"e2e-shared-b-w1-cce513-openshift-default-84cc9879-w4gt5\" (UID: \"5e64a3a3-48e4-4e47-bc24-e24fad22c38e\") " pod="openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-w4gt5"
Oct 30 01:04:59.003358 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:59.003330    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e64a3a3-48e4-4e47-bc24-e24fad22c38e-istio-token\") pod \"e2e-shared-b-w1-cce513-openshift-default-84cc9879-w4gt5\" (UID: \"5e64a3a3-48e4-4e47-bc24-e24fad22c38e\") " pod="openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-w4gt5"
Oct 30 01:04:59.003613 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:59.003592    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q956l\" (UniqueName: \"kubernetes.io/projected/5e64a3a3-48e4-4e47-bc24-e24fad22c38e-kube-api-access-q956l\") pod \"e2e-shared-b-w1-cce513-openshift-default-84cc9879-w4gt5\" (UID: \"5e64a3a3-48e4-4e47-bc24-e24fad22c38e\") " pod="openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-w4gt5"
Oct 30 01:04:59.126323 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:59.126063    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-w4gt5"
Oct 30 01:04:59.236496 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:59.233502    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-549b3758-gs27m"
Oct 30 01:04:59.292428 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:59.287122    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc015bf9-5c24-4af9-b931-69203486dd49-kube-api-access-ttld4\" (UniqueName: \"kubernetes.io/projected/cc015bf9-5c24-4af9-b931-69203486dd49-kube-api-access-ttld4\") pod \"cc015bf9-5c24-4af9-b931-69203486dd49\" (UID: \"cc015bf9-5c24-4af9-b931-69203486dd49\") "
Oct 30 01:04:59.292428 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:59.287176    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cc015bf9-5c24-4af9-b931-69203486dd49-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc015bf9-5c24-4af9-b931-69203486dd49-maas-api-xks-ca\") pod \"cc015bf9-5c24-4af9-b931-69203486dd49\" (UID: \"cc015bf9-5c24-4af9-b931-69203486dd49\") "
Oct 30 01:04:59.292428 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:59.287545    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc015bf9-5c24-4af9-b931-69203486dd49-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc015bf9-5c24-4af9-b931-69203486dd49-maas-api-openshift-ca\") pod \"cc015bf9-5c24-4af9-b931-69203486dd49\" (UID: \"cc015bf9-5c24-4af9-b931-69203486dd49\") "
Oct 30 01:04:59.292428 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:59.289603    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc015bf9-5c24-4af9-b931-69203486dd49-maas-api-openshift-ca" pod "cc015bf9-5c24-4af9-b931-69203486dd49" (UID: "cc015bf9-5c24-4af9-b931-69203486dd49"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:04:59.292886 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:59.292862    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc015bf9-5c24-4af9-b931-69203486dd49-maas-api-xks-ca" pod "cc015bf9-5c24-4af9-b931-69203486dd49" (UID: "cc015bf9-5c24-4af9-b931-69203486dd49"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:04:59.294175 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:59.294147    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc015bf9-5c24-4af9-b931-69203486dd49-kube-api-access-ttld4" pod "cc015bf9-5c24-4af9-b931-69203486dd49" (UID: "cc015bf9-5c24-4af9-b931-69203486dd49"). InnerVolumeSpecName "kube-api-access-ttld4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:04:59.390140 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:59.390047    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc015bf9-5c24-4af9-b931-69203486dd49-maas-api-openshift-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:04:59.390140 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:59.390083    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ttld4\" (UniqueName: \"kubernetes.io/projected/cc015bf9-5c24-4af9-b931-69203486dd49-kube-api-access-ttld4\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:04:59.390140 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:59.390096    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc015bf9-5c24-4af9-b931-69203486dd49-maas-api-xks-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:04:59.476286 ip-10-0-128-7 kubenswrapper[5002]: W1030 01:04:59.476249    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e64a3a3_48e4_4e47_bc24_e24fad22c38e.slice/crio-05518878beecaaa406aab8185583fadf33923c11c55f8df870f085c1b05729c1 WatchSource:0}: Error finding container 05518878beecaaa406aab8185583fadf33923c11c55f8df870f085c1b05729c1: Status 404 returned error can't find the container with id 05518878beecaaa406aab8185583fadf33923c11c55f8df870f085c1b05729c1
Oct 30 01:04:59.477331 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:59.477279    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-w4gt5"]
Oct 30 01:04:59.479677 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:59.479632    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:04:59.479775 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:59.479710    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:04:59.479775 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:04:59.479740    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:05:00.077088 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:00.077050    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-w4gt5" event={"ID":"5e64a3a3-48e4-4e47-bc24-e24fad22c38e","Type":"ContainerStarted","Data":"44dc2e49c6d87cdcfbb10e60446c1fd1e2ba64f5e1020eee9a93dac6c4a61f79"}
Oct 30 01:05:00.077298 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:00.077122    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-w4gt5" event={"ID":"5e64a3a3-48e4-4e47-bc24-e24fad22c38e","Type":"ContainerStarted","Data":"05518878beecaaa406aab8185583fadf33923c11c55f8df870f085c1b05729c1"}
Oct 30 01:05:00.078929 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:00.078901    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-549b3758-gs27m" event={"ID":"cc015bf9-5c24-4af9-b931-69203486dd49","Type":"ContainerDied","Data":"af7e8c35feba2d17753e3ea9b19d19771b7e9fa918eaa7d275ac8f3746178ebe"}
Oct 30 01:05:00.078929 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:00.078917    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-549b3758-gs27m"
Oct 30 01:05:00.079072 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:00.078949    5002 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="af7e8c35feba2d17753e3ea9b19d19771b7e9fa918eaa7d275ac8f3746178ebe"
Oct 30 01:05:00.127031 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:00.126993    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-w4gt5"
Oct 30 01:05:00.129564 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:00.129527    5002 patch_prober.go:29] interesting pod/e2e-shared-b-w1-cce513-openshift-default-84cc9879-w4gt5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" start-of-body=
Oct 30 01:05:00.129728 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:00.129595    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-w4gt5" podUID="5e64a3a3-48e4-4e47-bc24-e24fad22c38e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused"
Oct 30 01:05:00.149376 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:00.149307    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-w4gt5" podStartSLOduration=2.149287021 podStartE2EDuration="2.149287021s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:04:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:05:00.143975887 +0000 UTC m=+1995.760408778" watchObservedRunningTime="2026-10-30 01:05:00.149287021 +0000 UTC m=+1995.765719901"
Oct 30 01:05:01.127252 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:01.127210    5002 patch_prober.go:29] interesting pod/e2e-shared-b-w1-cce513-openshift-default-84cc9879-w4gt5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" start-of-body=
Oct 30 01:05:01.127702 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:01.127270    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-w4gt5" podUID="5e64a3a3-48e4-4e47-bc24-e24fad22c38e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused"
Oct 30 01:05:02.127622 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:02.127583    5002 patch_prober.go:29] interesting pod/e2e-shared-b-w1-cce513-openshift-default-84cc9879-w4gt5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" start-of-body=
Oct 30 01:05:02.128041 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:02.127649    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-w4gt5" podUID="5e64a3a3-48e4-4e47-bc24-e24fad22c38e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused"
Oct 30 01:05:03.001861 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:03.001829    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-009e78-54d78f6cf9-sjgcl"
Oct 30 01:05:03.002431 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:03.002399    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-009e78-847cbfbf7d-vgf6m"
Oct 30 01:05:03.127866 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:03.127831    5002 patch_prober.go:29] interesting pod/e2e-shared-b-w1-cce513-openshift-default-84cc9879-w4gt5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" start-of-body=
Oct 30 01:05:03.128332 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:03.127884    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-w4gt5" podUID="5e64a3a3-48e4-4e47-bc24-e24fad22c38e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused"
Oct 30 01:05:04.127150 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:04.127114    5002 patch_prober.go:29] interesting pod/e2e-shared-b-w1-cce513-openshift-default-84cc9879-w4gt5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" start-of-body=
Oct 30 01:05:04.127353 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:04.127169    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-w4gt5" podUID="5e64a3a3-48e4-4e47-bc24-e24fad22c38e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused"
Oct 30 01:05:05.127141 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:05.127106    5002 patch_prober.go:29] interesting pod/e2e-shared-b-w1-cce513-openshift-default-84cc9879-w4gt5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" start-of-body=
Oct 30 01:05:05.127644 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:05.127172    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-w4gt5" podUID="5e64a3a3-48e4-4e47-bc24-e24fad22c38e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused"
Oct 30 01:05:05.667791 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:05.667746    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w1-48df96/rate-test-model-48df96-kserve-5b9bb8959c-pd7z9"]
Oct 30 01:05:05.668915 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:05.668858    5002 cpu_manager.go:414] "containerMap: removing container" podUID="cc015bf9-5c24-4af9-b931-69203486dd49" containerName="revoke-keys"
Oct 30 01:05:05.668915 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:05.668885    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc015bf9-5c24-4af9-b931-69203486dd49" containerName="revoke-keys"
Oct 30 01:05:05.669148 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:05.669045    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc015bf9-5c24-4af9-b931-69203486dd49" containerName="revoke-keys"
Oct 30 01:05:05.680616 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:05.680582    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-48df96/rate-test-model-48df96-kserve-5b9bb8959c-pd7z9"
Oct 30 01:05:05.690513 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:05.690471    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-48df96/rate-test-model-48df96-kserve-5b9bb8959c-pd7z9"]
Oct 30 01:05:05.699825 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:05.699793    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-48df96\"/\"kube-root-ca.crt\""
Oct 30 01:05:05.713812 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:05.713782    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-48df96\"/\"default-dockercfg-7f7cd\""
Oct 30 01:05:05.713812 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:05.713781    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-48df96\"/\"rate-test-model-48df96-kserve-self-signed-certs\""
Oct 30 01:05:05.714713 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:05.714689    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-48df96\"/\"openshift-service-ca.crt\""
Oct 30 01:05:05.804834 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:05.804787    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fab2daef-ee8c-44f5-8e11-1d11372976a1-tls-certs\") pod \"rate-test-model-48df96-kserve-5b9bb8959c-pd7z9\" (UID: \"fab2daef-ee8c-44f5-8e11-1d11372976a1\") " pod="ai-tenant-e2e-shared-a-w1-48df96/rate-test-model-48df96-kserve-5b9bb8959c-pd7z9"
Oct 30 01:05:05.805047 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:05.804906    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fab2daef-ee8c-44f5-8e11-1d11372976a1-dshm\") pod \"rate-test-model-48df96-kserve-5b9bb8959c-pd7z9\" (UID: \"fab2daef-ee8c-44f5-8e11-1d11372976a1\") " pod="ai-tenant-e2e-shared-a-w1-48df96/rate-test-model-48df96-kserve-5b9bb8959c-pd7z9"
Oct 30 01:05:05.805047 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:05.804974    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fab2daef-ee8c-44f5-8e11-1d11372976a1-tmp-dir\") pod \"rate-test-model-48df96-kserve-5b9bb8959c-pd7z9\" (UID: \"fab2daef-ee8c-44f5-8e11-1d11372976a1\") " pod="ai-tenant-e2e-shared-a-w1-48df96/rate-test-model-48df96-kserve-5b9bb8959c-pd7z9"
Oct 30 01:05:05.805047 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:05.805037    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fn4qw\" (UniqueName: \"kubernetes.io/projected/fab2daef-ee8c-44f5-8e11-1d11372976a1-kube-api-access-fn4qw\") pod \"rate-test-model-48df96-kserve-5b9bb8959c-pd7z9\" (UID: \"fab2daef-ee8c-44f5-8e11-1d11372976a1\") " pod="ai-tenant-e2e-shared-a-w1-48df96/rate-test-model-48df96-kserve-5b9bb8959c-pd7z9"
Oct 30 01:05:05.805254 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:05.805226    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fab2daef-ee8c-44f5-8e11-1d11372976a1-home\") pod \"rate-test-model-48df96-kserve-5b9bb8959c-pd7z9\" (UID: \"fab2daef-ee8c-44f5-8e11-1d11372976a1\") " pod="ai-tenant-e2e-shared-a-w1-48df96/rate-test-model-48df96-kserve-5b9bb8959c-pd7z9"
Oct 30 01:05:05.805321 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:05.805255    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fab2daef-ee8c-44f5-8e11-1d11372976a1-model-cache\") pod \"rate-test-model-48df96-kserve-5b9bb8959c-pd7z9\" (UID: \"fab2daef-ee8c-44f5-8e11-1d11372976a1\") " pod="ai-tenant-e2e-shared-a-w1-48df96/rate-test-model-48df96-kserve-5b9bb8959c-pd7z9"
Oct 30 01:05:05.906205 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:05.906169    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fab2daef-ee8c-44f5-8e11-1d11372976a1-tmp-dir\") pod \"rate-test-model-48df96-kserve-5b9bb8959c-pd7z9\" (UID: \"fab2daef-ee8c-44f5-8e11-1d11372976a1\") " pod="ai-tenant-e2e-shared-a-w1-48df96/rate-test-model-48df96-kserve-5b9bb8959c-pd7z9"
Oct 30 01:05:05.906538 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:05.906512    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fn4qw\" (UniqueName: \"kubernetes.io/projected/fab2daef-ee8c-44f5-8e11-1d11372976a1-kube-api-access-fn4qw\") pod \"rate-test-model-48df96-kserve-5b9bb8959c-pd7z9\" (UID: \"fab2daef-ee8c-44f5-8e11-1d11372976a1\") " pod="ai-tenant-e2e-shared-a-w1-48df96/rate-test-model-48df96-kserve-5b9bb8959c-pd7z9"
Oct 30 01:05:05.906672 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:05.906644    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fab2daef-ee8c-44f5-8e11-1d11372976a1-tmp-dir\") pod \"rate-test-model-48df96-kserve-5b9bb8959c-pd7z9\" (UID: \"fab2daef-ee8c-44f5-8e11-1d11372976a1\") " pod="ai-tenant-e2e-shared-a-w1-48df96/rate-test-model-48df96-kserve-5b9bb8959c-pd7z9"
Oct 30 01:05:05.906898 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:05.906878    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fab2daef-ee8c-44f5-8e11-1d11372976a1-home\") pod \"rate-test-model-48df96-kserve-5b9bb8959c-pd7z9\" (UID: \"fab2daef-ee8c-44f5-8e11-1d11372976a1\") " pod="ai-tenant-e2e-shared-a-w1-48df96/rate-test-model-48df96-kserve-5b9bb8959c-pd7z9"
Oct 30 01:05:05.907022 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:05.907006    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fab2daef-ee8c-44f5-8e11-1d11372976a1-model-cache\") pod \"rate-test-model-48df96-kserve-5b9bb8959c-pd7z9\" (UID: \"fab2daef-ee8c-44f5-8e11-1d11372976a1\") " pod="ai-tenant-e2e-shared-a-w1-48df96/rate-test-model-48df96-kserve-5b9bb8959c-pd7z9"
Oct 30 01:05:05.907301 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:05.907281    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fab2daef-ee8c-44f5-8e11-1d11372976a1-tls-certs\") pod \"rate-test-model-48df96-kserve-5b9bb8959c-pd7z9\" (UID: \"fab2daef-ee8c-44f5-8e11-1d11372976a1\") " pod="ai-tenant-e2e-shared-a-w1-48df96/rate-test-model-48df96-kserve-5b9bb8959c-pd7z9"
Oct 30 01:05:05.907523 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:05.907504    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fab2daef-ee8c-44f5-8e11-1d11372976a1-dshm\") pod \"rate-test-model-48df96-kserve-5b9bb8959c-pd7z9\" (UID: \"fab2daef-ee8c-44f5-8e11-1d11372976a1\") " pod="ai-tenant-e2e-shared-a-w1-48df96/rate-test-model-48df96-kserve-5b9bb8959c-pd7z9"
Oct 30 01:05:05.908784 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:05.908756    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fab2daef-ee8c-44f5-8e11-1d11372976a1-model-cache\") pod \"rate-test-model-48df96-kserve-5b9bb8959c-pd7z9\" (UID: \"fab2daef-ee8c-44f5-8e11-1d11372976a1\") " pod="ai-tenant-e2e-shared-a-w1-48df96/rate-test-model-48df96-kserve-5b9bb8959c-pd7z9"
Oct 30 01:05:05.949465 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:05.907312    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fab2daef-ee8c-44f5-8e11-1d11372976a1-home\") pod \"rate-test-model-48df96-kserve-5b9bb8959c-pd7z9\" (UID: \"fab2daef-ee8c-44f5-8e11-1d11372976a1\") " pod="ai-tenant-e2e-shared-a-w1-48df96/rate-test-model-48df96-kserve-5b9bb8959c-pd7z9"
Oct 30 01:05:05.963414 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:05.957107    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fn4qw\" (UniqueName: \"kubernetes.io/projected/fab2daef-ee8c-44f5-8e11-1d11372976a1-kube-api-access-fn4qw\") pod \"rate-test-model-48df96-kserve-5b9bb8959c-pd7z9\" (UID: \"fab2daef-ee8c-44f5-8e11-1d11372976a1\") " pod="ai-tenant-e2e-shared-a-w1-48df96/rate-test-model-48df96-kserve-5b9bb8959c-pd7z9"
Oct 30 01:05:05.963914 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:05.963846    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fab2daef-ee8c-44f5-8e11-1d11372976a1-dshm\") pod \"rate-test-model-48df96-kserve-5b9bb8959c-pd7z9\" (UID: \"fab2daef-ee8c-44f5-8e11-1d11372976a1\") " pod="ai-tenant-e2e-shared-a-w1-48df96/rate-test-model-48df96-kserve-5b9bb8959c-pd7z9"
Oct 30 01:05:05.975443 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:05.966092    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fab2daef-ee8c-44f5-8e11-1d11372976a1-tls-certs\") pod \"rate-test-model-48df96-kserve-5b9bb8959c-pd7z9\" (UID: \"fab2daef-ee8c-44f5-8e11-1d11372976a1\") " pod="ai-tenant-e2e-shared-a-w1-48df96/rate-test-model-48df96-kserve-5b9bb8959c-pd7z9"
Oct 30 01:05:06.022404 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:06.004682    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-48df96/rate-test-model-48df96-kserve-5b9bb8959c-pd7z9"
Oct 30 01:05:06.149487 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:06.140829    5002 patch_prober.go:29] interesting pod/e2e-shared-b-w1-cce513-openshift-default-84cc9879-w4gt5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" start-of-body=
Oct 30 01:05:06.149487 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:06.140892    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-w4gt5" podUID="5e64a3a3-48e4-4e47-bc24-e24fad22c38e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused"
Oct 30 01:05:06.466090 ip-10-0-128-7 kubenswrapper[5002]: W1030 01:05:06.466043    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfab2daef_ee8c_44f5_8e11_1d11372976a1.slice/crio-7e444533a3b72ee0763c06d9f4a72e92d5f90955f272991cc8d195611761ccc6 WatchSource:0}: Error finding container 7e444533a3b72ee0763c06d9f4a72e92d5f90955f272991cc8d195611761ccc6: Status 404 returned error can't find the container with id 7e444533a3b72ee0763c06d9f4a72e92d5f90955f272991cc8d195611761ccc6
Oct 30 01:05:06.466276 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:06.466180    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-48df96/rate-test-model-48df96-kserve-5b9bb8959c-pd7z9"]
Oct 30 01:05:07.132752 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:07.132722    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-w4gt5"
Oct 30 01:05:07.133233 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:07.133219    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-w4gt5"
Oct 30 01:05:07.133994 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:07.133968    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-w4gt5"
Oct 30 01:05:07.167339 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:07.167298    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-48df96/rate-test-model-48df96-kserve-5b9bb8959c-pd7z9" event={"ID":"fab2daef-ee8c-44f5-8e11-1d11372976a1","Type":"ContainerStarted","Data":"e2deec5d8ea1cb5c9d0b6a5462bcf50f11d3ec726456e69252ce73f5798efb8e"}
Oct 30 01:05:07.168211 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:07.167356    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-48df96/rate-test-model-48df96-kserve-5b9bb8959c-pd7z9" event={"ID":"fab2daef-ee8c-44f5-8e11-1d11372976a1","Type":"ContainerStarted","Data":"7e444533a3b72ee0763c06d9f4a72e92d5f90955f272991cc8d195611761ccc6"}
Oct 30 01:05:07.241165 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:07.241103    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w1-48df96/rate-test-model-48df96-kserve-5b9bb8959c-pd7z9" podStartSLOduration=2.241086617 podStartE2EDuration="2.241086617s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:05:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:05:07.238052133 +0000 UTC m=+2002.854485015" watchObservedRunningTime="2026-10-30 01:05:07.241086617 +0000 UTC m=+2002.857519500"
Oct 30 01:05:08.005597 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:08.005561    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w1-48df96/rate-test-model-48df96-kserve-5b9bb8959c-pd7z9"
Oct 30 01:05:12.870129 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:05:12.870082    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbjdq" containerName="extractor"
Oct 30 01:05:16.005673 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:16.005631    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w1-48df96/rate-test-model-48df96-kserve-5b9bb8959c-pd7z9"
Oct 30 01:05:16.035487 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:16.035450    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w1-48df96/rate-test-model-48df96-kserve-5b9bb8959c-pd7z9"
Oct 30 01:05:16.271045 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:16.270964    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w1-48df96/rate-test-model-48df96-kserve-5b9bb8959c-pd7z9"
Oct 30 01:05:19.489259 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:19.489223    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-51b95e/test-model-51b95e-kserve-5c4d8899f7-hc4c2"]
Oct 30 01:05:20.065857 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:20.065819    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4672e3a2-sfk88"]
Oct 30 01:05:20.070555 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:20.070534    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4672e3a2-sfk88"
Oct 30 01:05:20.087074 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:20.087047    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:05:20.087208 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:20.087113    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:05:20.089225 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:20.089197    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4672e3a2-sfk88"]
Oct 30 01:05:20.098433 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:20.098302    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5phsh\""
Oct 30 01:05:20.098433 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:20.098401    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:05:20.230280 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:20.230240    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rd2sg\" (UniqueName: \"kubernetes.io/projected/05f79dc7-3c9c-491a-b288-a5ee57061e7a-kube-api-access-rd2sg\") pod \"maas-api-revoke-sub-4672e3a2-sfk88\" (UID: \"05f79dc7-3c9c-491a-b288-a5ee57061e7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4672e3a2-sfk88"
Oct 30 01:05:20.230280 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:20.230284    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05f79dc7-3c9c-491a-b288-a5ee57061e7a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4672e3a2-sfk88\" (UID: \"05f79dc7-3c9c-491a-b288-a5ee57061e7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4672e3a2-sfk88"
Oct 30 01:05:20.230522 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:20.230370    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05f79dc7-3c9c-491a-b288-a5ee57061e7a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4672e3a2-sfk88\" (UID: \"05f79dc7-3c9c-491a-b288-a5ee57061e7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4672e3a2-sfk88"
Oct 30 01:05:20.332069 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:20.331992    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05f79dc7-3c9c-491a-b288-a5ee57061e7a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4672e3a2-sfk88\" (UID: \"05f79dc7-3c9c-491a-b288-a5ee57061e7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4672e3a2-sfk88"
Oct 30 01:05:20.332328 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:20.332305    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rd2sg\" (UniqueName: \"kubernetes.io/projected/05f79dc7-3c9c-491a-b288-a5ee57061e7a-kube-api-access-rd2sg\") pod \"maas-api-revoke-sub-4672e3a2-sfk88\" (UID: \"05f79dc7-3c9c-491a-b288-a5ee57061e7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4672e3a2-sfk88"
Oct 30 01:05:20.332427 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:20.332354    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05f79dc7-3c9c-491a-b288-a5ee57061e7a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4672e3a2-sfk88\" (UID: \"05f79dc7-3c9c-491a-b288-a5ee57061e7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4672e3a2-sfk88"
Oct 30 01:05:20.333044 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:20.333018    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05f79dc7-3c9c-491a-b288-a5ee57061e7a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4672e3a2-sfk88\" (UID: \"05f79dc7-3c9c-491a-b288-a5ee57061e7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4672e3a2-sfk88"
Oct 30 01:05:20.334533 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:20.334513    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05f79dc7-3c9c-491a-b288-a5ee57061e7a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4672e3a2-sfk88\" (UID: \"05f79dc7-3c9c-491a-b288-a5ee57061e7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4672e3a2-sfk88"
Oct 30 01:05:20.357077 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:20.357035    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rd2sg\" (UniqueName: \"kubernetes.io/projected/05f79dc7-3c9c-491a-b288-a5ee57061e7a-kube-api-access-rd2sg\") pod \"maas-api-revoke-sub-4672e3a2-sfk88\" (UID: \"05f79dc7-3c9c-491a-b288-a5ee57061e7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4672e3a2-sfk88"
Oct 30 01:05:20.386472 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:20.386436    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4672e3a2-sfk88"
Oct 30 01:05:20.696243 ip-10-0-128-7 kubenswrapper[5002]: W1030 01:05:20.696208    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05f79dc7_3c9c_491a_b288_a5ee57061e7a.slice/crio-6f895e07fa9883c4c3f64b2d9f52b24de493f9ea034de7652e47800fb8e43e5e WatchSource:0}: Error finding container 6f895e07fa9883c4c3f64b2d9f52b24de493f9ea034de7652e47800fb8e43e5e: Status 404 returned error can't find the container with id 6f895e07fa9883c4c3f64b2d9f52b24de493f9ea034de7652e47800fb8e43e5e
Oct 30 01:05:20.697799 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:20.697768    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4672e3a2-sfk88"]
Oct 30 01:05:21.379546 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:21.379452    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4672e3a2-sfk88" event={"ID":"05f79dc7-3c9c-491a-b288-a5ee57061e7a","Type":"ContainerStarted","Data":"8e15f8d3e362db3d9581ee7215dde9279ce0b379e21ffcc218dac3072895be88"}
Oct 30 01:05:21.379546 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:21.379514    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4672e3a2-sfk88" event={"ID":"05f79dc7-3c9c-491a-b288-a5ee57061e7a","Type":"ContainerStarted","Data":"6f895e07fa9883c4c3f64b2d9f52b24de493f9ea034de7652e47800fb8e43e5e"}
Oct 30 01:05:21.428727 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:21.428663    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4672e3a2-sfk88" podStartSLOduration=1.428644855 podStartE2EDuration="1.428644855s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:05:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:05:21.42242088 +0000 UTC m=+2017.038853755" watchObservedRunningTime="2026-10-30 01:05:21.428644855 +0000 UTC m=+2017.045077736"
Oct 30 01:05:22.390113 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:22.390060    5002 generic.go:356] "Generic (PLEG): container finished" podID="05f79dc7-3c9c-491a-b288-a5ee57061e7a" containerID="8e15f8d3e362db3d9581ee7215dde9279ce0b379e21ffcc218dac3072895be88" exitCode=0
Oct 30 01:05:22.390574 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:22.390201    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4672e3a2-sfk88" event={"ID":"05f79dc7-3c9c-491a-b288-a5ee57061e7a","Type":"ContainerDied","Data":"8e15f8d3e362db3d9581ee7215dde9279ce0b379e21ffcc218dac3072895be88"}
Oct 30 01:05:23.926142 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:23.926107    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-84b89f-dknxv"]
Oct 30 01:05:23.961402 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:23.956773    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-84b89f-dknxv"]
Oct 30 01:05:23.961402 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:23.956983    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4672e3a2-sfk88"
Oct 30 01:05:24.004132 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:24.004099    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05f79dc7-3c9c-491a-b288-a5ee57061e7a-kube-api-access-rd2sg\" (UniqueName: \"kubernetes.io/projected/05f79dc7-3c9c-491a-b288-a5ee57061e7a-kube-api-access-rd2sg\") pod \"05f79dc7-3c9c-491a-b288-a5ee57061e7a\" (UID: \"05f79dc7-3c9c-491a-b288-a5ee57061e7a\") "
Oct 30 01:05:24.004320 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:24.004259    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05f79dc7-3c9c-491a-b288-a5ee57061e7a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05f79dc7-3c9c-491a-b288-a5ee57061e7a-maas-api-openshift-ca\") pod \"05f79dc7-3c9c-491a-b288-a5ee57061e7a\" (UID: \"05f79dc7-3c9c-491a-b288-a5ee57061e7a\") "
Oct 30 01:05:24.004441 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:24.004365    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/05f79dc7-3c9c-491a-b288-a5ee57061e7a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05f79dc7-3c9c-491a-b288-a5ee57061e7a-maas-api-xks-ca\") pod \"05f79dc7-3c9c-491a-b288-a5ee57061e7a\" (UID: \"05f79dc7-3c9c-491a-b288-a5ee57061e7a\") "
Oct 30 01:05:24.004841 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:24.004668    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05f79dc7-3c9c-491a-b288-a5ee57061e7a-maas-api-openshift-ca" pod "05f79dc7-3c9c-491a-b288-a5ee57061e7a" (UID: "05f79dc7-3c9c-491a-b288-a5ee57061e7a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:05:24.005140 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:24.005117    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05f79dc7-3c9c-491a-b288-a5ee57061e7a-maas-api-openshift-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:05:24.007228 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:24.007205    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05f79dc7-3c9c-491a-b288-a5ee57061e7a-kube-api-access-rd2sg" pod "05f79dc7-3c9c-491a-b288-a5ee57061e7a" (UID: "05f79dc7-3c9c-491a-b288-a5ee57061e7a"). InnerVolumeSpecName "kube-api-access-rd2sg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:05:24.007688 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:24.007665    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05f79dc7-3c9c-491a-b288-a5ee57061e7a-maas-api-xks-ca" pod "05f79dc7-3c9c-491a-b288-a5ee57061e7a" (UID: "05f79dc7-3c9c-491a-b288-a5ee57061e7a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:05:24.114462 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:24.113640    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rd2sg\" (UniqueName: \"kubernetes.io/projected/05f79dc7-3c9c-491a-b288-a5ee57061e7a-kube-api-access-rd2sg\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:05:24.114462 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:24.113685    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05f79dc7-3c9c-491a-b288-a5ee57061e7a-maas-api-xks-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:05:24.435874 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:24.435848    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4672e3a2-sfk88"
Oct 30 01:05:24.436036 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:24.435891    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4672e3a2-sfk88" event={"ID":"05f79dc7-3c9c-491a-b288-a5ee57061e7a","Type":"ContainerDied","Data":"6f895e07fa9883c4c3f64b2d9f52b24de493f9ea034de7652e47800fb8e43e5e"}
Oct 30 01:05:24.436036 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:24.435936    5002 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6f895e07fa9883c4c3f64b2d9f52b24de493f9ea034de7652e47800fb8e43e5e"
Oct 30 01:05:24.496253 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:24.496213    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="711f2f5e-e661-488a-be49-6e85d9673f24" path="/var/lib/kubelet/pods/711f2f5e-e661-488a-be49-6e85d9673f24/volumes"
Oct 30 01:05:31.036835 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:31.036572    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c72c8308-xgd5x"]
Oct 30 01:05:31.048673 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:31.048634    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c72c8308-xgd5x"]
Oct 30 01:05:31.406361 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:31.406267    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-84b89f-openshift-default-f7d86d855-mddrw"]
Oct 30 01:05:31.406583 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:31.406531    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-84b89f-openshift-default-f7d86d855-mddrw" podUID="b88fa1c1-bbb8-4f74-bc2a-3cff7b647082" containerName="istio-proxy" containerID="cri-o://eaf8d2f564d2b486137ba14383111a62ab8163997c338185e309959ec11bba82" gracePeriod=30
Oct 30 01:05:32.498079 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:32.498035    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d76781c5-3cb1-48c5-81d0-8c4d09c26a82" path="/var/lib/kubelet/pods/d76781c5-3cb1-48c5-81d0-8c4d09c26a82/volumes"
Oct 30 01:05:33.736563 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:33.736524    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8d2a70-cxtxf"]
Oct 30 01:05:33.740346 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:33.740315    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8d2a70-cxtxf"]
Oct 30 01:05:34.497623 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:34.497585    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="002a3d53-2925-4977-967a-5a23a655a12f" path="/var/lib/kubelet/pods/002a3d53-2925-4977-967a-5a23a655a12f/volumes"
Oct 30 01:05:34.520756 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:34.520707    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-51b95e/test-model-51b95e-kserve-5c4d8899f7-hc4c2" podUID="87ebc816-fbe5-4a3e-b650-39f2a72e9037" containerName="main" containerID="cri-o://fff95db193bc2b29a8faf558fbbdf9e1e9b14c6e7b149d8a10051bba847bf629" gracePeriod=45
Oct 30 01:05:34.838371 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:34.838342    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-51b95e/test-model-51b95e-kserve-5c4d8899f7-hc4c2"
Oct 30 01:05:34.866271 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:34.866239    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87ebc816-fbe5-4a3e-b650-39f2a72e9037-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87ebc816-fbe5-4a3e-b650-39f2a72e9037-tmp-dir\") pod \"87ebc816-fbe5-4a3e-b650-39f2a72e9037\" (UID: \"87ebc816-fbe5-4a3e-b650-39f2a72e9037\") "
Oct 30 01:05:34.866549 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:34.866522    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87ebc816-fbe5-4a3e-b650-39f2a72e9037-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87ebc816-fbe5-4a3e-b650-39f2a72e9037-model-cache\") pod \"87ebc816-fbe5-4a3e-b650-39f2a72e9037\" (UID: \"87ebc816-fbe5-4a3e-b650-39f2a72e9037\") "
Oct 30 01:05:34.866549 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:34.866542    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87ebc816-fbe5-4a3e-b650-39f2a72e9037-tmp-dir" pod "87ebc816-fbe5-4a3e-b650-39f2a72e9037" (UID: "87ebc816-fbe5-4a3e-b650-39f2a72e9037"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:05:34.866708 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:34.866574    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/87ebc816-fbe5-4a3e-b650-39f2a72e9037-tls-certs\" (UniqueName: \"kubernetes.io/secret/87ebc816-fbe5-4a3e-b650-39f2a72e9037-tls-certs\") pod \"87ebc816-fbe5-4a3e-b650-39f2a72e9037\" (UID: \"87ebc816-fbe5-4a3e-b650-39f2a72e9037\") "
Oct 30 01:05:34.866708 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:34.866602    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87ebc816-fbe5-4a3e-b650-39f2a72e9037-home\" (UniqueName: \"kubernetes.io/empty-dir/87ebc816-fbe5-4a3e-b650-39f2a72e9037-home\") pod \"87ebc816-fbe5-4a3e-b650-39f2a72e9037\" (UID: \"87ebc816-fbe5-4a3e-b650-39f2a72e9037\") "
Oct 30 01:05:34.866708 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:34.866698    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87ebc816-fbe5-4a3e-b650-39f2a72e9037-kube-api-access-xdxlq\" (UniqueName: \"kubernetes.io/projected/87ebc816-fbe5-4a3e-b650-39f2a72e9037-kube-api-access-xdxlq\") pod \"87ebc816-fbe5-4a3e-b650-39f2a72e9037\" (UID: \"87ebc816-fbe5-4a3e-b650-39f2a72e9037\") "
Oct 30 01:05:34.866884 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:34.866774    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87ebc816-fbe5-4a3e-b650-39f2a72e9037-dshm\" (UniqueName: \"kubernetes.io/empty-dir/87ebc816-fbe5-4a3e-b650-39f2a72e9037-dshm\") pod \"87ebc816-fbe5-4a3e-b650-39f2a72e9037\" (UID: \"87ebc816-fbe5-4a3e-b650-39f2a72e9037\") "
Oct 30 01:05:34.866884 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:34.866788    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87ebc816-fbe5-4a3e-b650-39f2a72e9037-model-cache" pod "87ebc816-fbe5-4a3e-b650-39f2a72e9037" (UID: "87ebc816-fbe5-4a3e-b650-39f2a72e9037"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:05:34.866991 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:34.866955    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87ebc816-fbe5-4a3e-b650-39f2a72e9037-home" pod "87ebc816-fbe5-4a3e-b650-39f2a72e9037" (UID: "87ebc816-fbe5-4a3e-b650-39f2a72e9037"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:05:34.867590 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:34.867550    5002 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87ebc816-fbe5-4a3e-b650-39f2a72e9037-model-cache\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:05:34.867590 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:34.867581    5002 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/87ebc816-fbe5-4a3e-b650-39f2a72e9037-home\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:05:34.867590 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:34.867594    5002 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87ebc816-fbe5-4a3e-b650-39f2a72e9037-tmp-dir\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:05:34.869772 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:34.869740    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87ebc816-fbe5-4a3e-b650-39f2a72e9037-dshm" pod "87ebc816-fbe5-4a3e-b650-39f2a72e9037" (UID: "87ebc816-fbe5-4a3e-b650-39f2a72e9037"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:05:34.870057 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:34.870028    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87ebc816-fbe5-4a3e-b650-39f2a72e9037-kube-api-access-xdxlq" pod "87ebc816-fbe5-4a3e-b650-39f2a72e9037" (UID: "87ebc816-fbe5-4a3e-b650-39f2a72e9037"). InnerVolumeSpecName "kube-api-access-xdxlq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:05:34.870259 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:34.870230    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87ebc816-fbe5-4a3e-b650-39f2a72e9037-tls-certs" pod "87ebc816-fbe5-4a3e-b650-39f2a72e9037" (UID: "87ebc816-fbe5-4a3e-b650-39f2a72e9037"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:05:34.968887 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:34.968848    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xdxlq\" (UniqueName: \"kubernetes.io/projected/87ebc816-fbe5-4a3e-b650-39f2a72e9037-kube-api-access-xdxlq\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:05:34.968887 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:34.968890    5002 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/87ebc816-fbe5-4a3e-b650-39f2a72e9037-dshm\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:05:34.969106 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:34.968905    5002 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87ebc816-fbe5-4a3e-b650-39f2a72e9037-tls-certs\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:05:35.002244 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:35.002211    5002 patch_prober.go:29] interesting pod/e2e-inf-b-84b89f-openshift-default-f7d86d855-mddrw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 01:05:35.002407 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:35.002292    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-84b89f-openshift-default-f7d86d855-mddrw" podUID="b88fa1c1-bbb8-4f74-bc2a-3cff7b647082" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 01:05:35.635297 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:35.635257    5002 generic.go:356] "Generic (PLEG): container finished" podID="87ebc816-fbe5-4a3e-b650-39f2a72e9037" containerID="fff95db193bc2b29a8faf558fbbdf9e1e9b14c6e7b149d8a10051bba847bf629" exitCode=0
Oct 30 01:05:35.635517 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:35.635323    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-51b95e/test-model-51b95e-kserve-5c4d8899f7-hc4c2"
Oct 30 01:05:35.635517 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:35.635325    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-51b95e/test-model-51b95e-kserve-5c4d8899f7-hc4c2" event={"ID":"87ebc816-fbe5-4a3e-b650-39f2a72e9037","Type":"ContainerDied","Data":"fff95db193bc2b29a8faf558fbbdf9e1e9b14c6e7b149d8a10051bba847bf629"}
Oct 30 01:05:35.635517 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:35.635366    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-51b95e/test-model-51b95e-kserve-5c4d8899f7-hc4c2" event={"ID":"87ebc816-fbe5-4a3e-b650-39f2a72e9037","Type":"ContainerDied","Data":"843570c81ee3123d849a9a65337207d0b39140932a41429192e14930d72e36f8"}
Oct 30 01:05:35.635517 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:35.635427    5002 scope.go:122] "RemoveContainer" containerID="fff95db193bc2b29a8faf558fbbdf9e1e9b14c6e7b149d8a10051bba847bf629"
Oct 30 01:05:35.649592 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:35.649567    5002 scope.go:122] "RemoveContainer" containerID="fff95db193bc2b29a8faf558fbbdf9e1e9b14c6e7b149d8a10051bba847bf629"
Oct 30 01:05:35.650215 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:05:35.650141    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fff95db193bc2b29a8faf558fbbdf9e1e9b14c6e7b149d8a10051bba847bf629\": container with ID starting with fff95db193bc2b29a8faf558fbbdf9e1e9b14c6e7b149d8a10051bba847bf629 not found: ID does not exist" containerID="fff95db193bc2b29a8faf558fbbdf9e1e9b14c6e7b149d8a10051bba847bf629"
Oct 30 01:05:35.650215 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:35.650179    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fff95db193bc2b29a8faf558fbbdf9e1e9b14c6e7b149d8a10051bba847bf629"} err="failed to get container status \"fff95db193bc2b29a8faf558fbbdf9e1e9b14c6e7b149d8a10051bba847bf629\": rpc error: code = NotFound desc = could not find container \"fff95db193bc2b29a8faf558fbbdf9e1e9b14c6e7b149d8a10051bba847bf629\": container with ID starting with fff95db193bc2b29a8faf558fbbdf9e1e9b14c6e7b149d8a10051bba847bf629 not found: ID does not exist"
Oct 30 01:05:35.710201 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:35.710164    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-51b95e/test-model-51b95e-kserve-5c4d8899f7-hc4c2"]
Oct 30 01:05:35.725057 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:35.724743    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-51b95e/test-model-51b95e-kserve-5c4d8899f7-hc4c2"]
Oct 30 01:05:36.498326 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:36.498291    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87ebc816-fbe5-4a3e-b650-39f2a72e9037" path="/var/lib/kubelet/pods/87ebc816-fbe5-4a3e-b650-39f2a72e9037/volumes"
Oct 30 01:05:36.646621 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:36.646589    5002 generic.go:356] "Generic (PLEG): container finished" podID="b88fa1c1-bbb8-4f74-bc2a-3cff7b647082" containerID="eaf8d2f564d2b486137ba14383111a62ab8163997c338185e309959ec11bba82" exitCode=0
Oct 30 01:05:36.646937 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:36.646661    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-84b89f-openshift-default-f7d86d855-mddrw" event={"ID":"b88fa1c1-bbb8-4f74-bc2a-3cff7b647082","Type":"ContainerDied","Data":"eaf8d2f564d2b486137ba14383111a62ab8163997c338185e309959ec11bba82"}
Oct 30 01:05:36.646937 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:36.646734    5002 scope.go:122] "RemoveContainer" containerID="b2b29a7f9905299ed60ad98fe7132fd97f68645f1333a030245fae1aa7c2c409"
Oct 30 01:05:36.710144 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:36.710120    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-84b89f-openshift-default-f7d86d855-mddrw"
Oct 30 01:05:36.805006 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:36.804802    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b88fa1c1-bbb8-4f74-bc2a-3cff7b647082-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b88fa1c1-bbb8-4f74-bc2a-3cff7b647082-workload-socket\") pod \"b88fa1c1-bbb8-4f74-bc2a-3cff7b647082\" (UID: \"b88fa1c1-bbb8-4f74-bc2a-3cff7b647082\") "
Oct 30 01:05:36.805006 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:36.804861    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b88fa1c1-bbb8-4f74-bc2a-3cff7b647082-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b88fa1c1-bbb8-4f74-bc2a-3cff7b647082-credential-socket\") pod \"b88fa1c1-bbb8-4f74-bc2a-3cff7b647082\" (UID: \"b88fa1c1-bbb8-4f74-bc2a-3cff7b647082\") "
Oct 30 01:05:36.805006 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:36.804904    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b88fa1c1-bbb8-4f74-bc2a-3cff7b647082-istio-token\" (UniqueName: \"kubernetes.io/projected/b88fa1c1-bbb8-4f74-bc2a-3cff7b647082-istio-token\") pod \"b88fa1c1-bbb8-4f74-bc2a-3cff7b647082\" (UID: \"b88fa1c1-bbb8-4f74-bc2a-3cff7b647082\") "
Oct 30 01:05:36.805006 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:36.804985    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b88fa1c1-bbb8-4f74-bc2a-3cff7b647082-kube-api-access-5pgz4\" (UniqueName: \"kubernetes.io/projected/b88fa1c1-bbb8-4f74-bc2a-3cff7b647082-kube-api-access-5pgz4\") pod \"b88fa1c1-bbb8-4f74-bc2a-3cff7b647082\" (UID: \"b88fa1c1-bbb8-4f74-bc2a-3cff7b647082\") "
Oct 30 01:05:36.805006 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:36.805024    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b88fa1c1-bbb8-4f74-bc2a-3cff7b647082-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b88fa1c1-bbb8-4f74-bc2a-3cff7b647082-istio-podinfo\") pod \"b88fa1c1-bbb8-4f74-bc2a-3cff7b647082\" (UID: \"b88fa1c1-bbb8-4f74-bc2a-3cff7b647082\") "
Oct 30 01:05:36.805482 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:36.805076    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b88fa1c1-bbb8-4f74-bc2a-3cff7b647082-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b88fa1c1-bbb8-4f74-bc2a-3cff7b647082-istio-data\") pod \"b88fa1c1-bbb8-4f74-bc2a-3cff7b647082\" (UID: \"b88fa1c1-bbb8-4f74-bc2a-3cff7b647082\") "
Oct 30 01:05:36.805482 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:36.805113    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b88fa1c1-bbb8-4f74-bc2a-3cff7b647082-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b88fa1c1-bbb8-4f74-bc2a-3cff7b647082-istiod-ca-cert\") pod \"b88fa1c1-bbb8-4f74-bc2a-3cff7b647082\" (UID: \"b88fa1c1-bbb8-4f74-bc2a-3cff7b647082\") "
Oct 30 01:05:36.805482 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:36.805122    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b88fa1c1-bbb8-4f74-bc2a-3cff7b647082-workload-socket" pod "b88fa1c1-bbb8-4f74-bc2a-3cff7b647082" (UID: "b88fa1c1-bbb8-4f74-bc2a-3cff7b647082"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:05:36.805482 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:36.805210    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b88fa1c1-bbb8-4f74-bc2a-3cff7b647082-credential-socket" pod "b88fa1c1-bbb8-4f74-bc2a-3cff7b647082" (UID: "b88fa1c1-bbb8-4f74-bc2a-3cff7b647082"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:05:36.805482 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:36.805245    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b88fa1c1-bbb8-4f74-bc2a-3cff7b647082-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b88fa1c1-bbb8-4f74-bc2a-3cff7b647082-workload-certs\") pod \"b88fa1c1-bbb8-4f74-bc2a-3cff7b647082\" (UID: \"b88fa1c1-bbb8-4f74-bc2a-3cff7b647082\") "
Oct 30 01:05:36.805482 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:36.805286    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b88fa1c1-bbb8-4f74-bc2a-3cff7b647082-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b88fa1c1-bbb8-4f74-bc2a-3cff7b647082-istio-envoy\") pod \"b88fa1c1-bbb8-4f74-bc2a-3cff7b647082\" (UID: \"b88fa1c1-bbb8-4f74-bc2a-3cff7b647082\") "
Oct 30 01:05:36.805794 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:36.805683    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b88fa1c1-bbb8-4f74-bc2a-3cff7b647082-istio-data" pod "b88fa1c1-bbb8-4f74-bc2a-3cff7b647082" (UID: "b88fa1c1-bbb8-4f74-bc2a-3cff7b647082"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:05:36.806172 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:36.806144    5002 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b88fa1c1-bbb8-4f74-bc2a-3cff7b647082-workload-socket\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:05:36.806292 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:36.806175    5002 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b88fa1c1-bbb8-4f74-bc2a-3cff7b647082-credential-socket\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:05:36.806292 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:36.806191    5002 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b88fa1c1-bbb8-4f74-bc2a-3cff7b647082-istio-data\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:05:36.806502 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:36.806480    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b88fa1c1-bbb8-4f74-bc2a-3cff7b647082-istiod-ca-cert" pod "b88fa1c1-bbb8-4f74-bc2a-3cff7b647082" (UID: "b88fa1c1-bbb8-4f74-bc2a-3cff7b647082"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:05:36.806873 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:36.806846    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b88fa1c1-bbb8-4f74-bc2a-3cff7b647082-workload-certs" pod "b88fa1c1-bbb8-4f74-bc2a-3cff7b647082" (UID: "b88fa1c1-bbb8-4f74-bc2a-3cff7b647082"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:05:36.808323 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:36.808034    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b88fa1c1-bbb8-4f74-bc2a-3cff7b647082-istio-token" pod "b88fa1c1-bbb8-4f74-bc2a-3cff7b647082" (UID: "b88fa1c1-bbb8-4f74-bc2a-3cff7b647082"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:05:36.819322 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:36.819233    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b88fa1c1-bbb8-4f74-bc2a-3cff7b647082-istio-envoy" pod "b88fa1c1-bbb8-4f74-bc2a-3cff7b647082" (UID: "b88fa1c1-bbb8-4f74-bc2a-3cff7b647082"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:05:36.819322 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:36.819287    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b88fa1c1-bbb8-4f74-bc2a-3cff7b647082-kube-api-access-5pgz4" pod "b88fa1c1-bbb8-4f74-bc2a-3cff7b647082" (UID: "b88fa1c1-bbb8-4f74-bc2a-3cff7b647082"). InnerVolumeSpecName "kube-api-access-5pgz4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:05:36.820644 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:36.820620    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b88fa1c1-bbb8-4f74-bc2a-3cff7b647082-istio-podinfo" pod "b88fa1c1-bbb8-4f74-bc2a-3cff7b647082" (UID: "b88fa1c1-bbb8-4f74-bc2a-3cff7b647082"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:05:36.908089 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:36.908048    5002 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b88fa1c1-bbb8-4f74-bc2a-3cff7b647082-istio-token\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:05:36.908089 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:36.908084    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5pgz4\" (UniqueName: \"kubernetes.io/projected/b88fa1c1-bbb8-4f74-bc2a-3cff7b647082-kube-api-access-5pgz4\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:05:36.908089 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:36.908093    5002 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b88fa1c1-bbb8-4f74-bc2a-3cff7b647082-istio-podinfo\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:05:36.908317 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:36.908102    5002 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b88fa1c1-bbb8-4f74-bc2a-3cff7b647082-istiod-ca-cert\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:05:36.908317 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:36.908110    5002 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b88fa1c1-bbb8-4f74-bc2a-3cff7b647082-workload-certs\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:05:36.908317 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:36.908121    5002 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b88fa1c1-bbb8-4f74-bc2a-3cff7b647082-istio-envoy\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:05:37.662649 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:37.662613    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-84b89f-openshift-default-f7d86d855-mddrw"
Oct 30 01:05:37.662649 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:37.662638    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-84b89f-openshift-default-f7d86d855-mddrw" event={"ID":"b88fa1c1-bbb8-4f74-bc2a-3cff7b647082","Type":"ContainerDied","Data":"86cf8a2d24ddfadc3c0f8e564c1a0ece855be88972a81cfe40521c7b0164f927"}
Oct 30 01:05:37.663232 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:37.662708    5002 scope.go:122] "RemoveContainer" containerID="eaf8d2f564d2b486137ba14383111a62ab8163997c338185e309959ec11bba82"
Oct 30 01:05:37.736857 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:37.736827    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-84b89f-openshift-default-f7d86d855-mddrw"]
Oct 30 01:05:37.751791 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:37.751745    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-84b89f-openshift-default-f7d86d855-mddrw"]
Oct 30 01:05:38.503190 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:38.503142    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b88fa1c1-bbb8-4f74-bc2a-3cff7b647082" path="/var/lib/kubelet/pods/b88fa1c1-bbb8-4f74-bc2a-3cff7b647082/volumes"
Oct 30 01:05:41.096810 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:41.096774    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-4989x"]
Oct 30 01:05:41.097615 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:41.097578    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-4989x" podUID="bf2f68d2-28a3-4488-b3c5-82d2e278c311" containerName="istio-proxy" containerID="cri-o://fb68e95a1d2cd04cf1310ef73c36ee324858282514e4098464b09faced7c7239" gracePeriod=30
Oct 30 01:05:41.649076 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:41.649041    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5007e1-l9fgc"]
Oct 30 01:05:41.649695 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:41.649679    5002 cpu_manager.go:414] "containerMap: removing container" podUID="b88fa1c1-bbb8-4f74-bc2a-3cff7b647082" containerName="istio-proxy"
Oct 30 01:05:41.649752 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:41.649695    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b88fa1c1-bbb8-4f74-bc2a-3cff7b647082" containerName="istio-proxy"
Oct 30 01:05:41.649752 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:41.649710    5002 cpu_manager.go:414] "containerMap: removing container" podUID="b88fa1c1-bbb8-4f74-bc2a-3cff7b647082" containerName="istio-proxy"
Oct 30 01:05:41.649752 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:41.649716    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b88fa1c1-bbb8-4f74-bc2a-3cff7b647082" containerName="istio-proxy"
Oct 30 01:05:41.649752 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:41.649748    5002 cpu_manager.go:414] "containerMap: removing container" podUID="05f79dc7-3c9c-491a-b288-a5ee57061e7a" containerName="revoke-keys"
Oct 30 01:05:41.649891 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:41.649754    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05f79dc7-3c9c-491a-b288-a5ee57061e7a" containerName="revoke-keys"
Oct 30 01:05:41.649891 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:41.649779    5002 cpu_manager.go:414] "containerMap: removing container" podUID="87ebc816-fbe5-4a3e-b650-39f2a72e9037" containerName="main"
Oct 30 01:05:41.649891 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:41.649784    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87ebc816-fbe5-4a3e-b650-39f2a72e9037" containerName="main"
Oct 30 01:05:41.649891 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:41.649874    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="05f79dc7-3c9c-491a-b288-a5ee57061e7a" containerName="revoke-keys"
Oct 30 01:05:41.649891 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:41.649887    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="87ebc816-fbe5-4a3e-b650-39f2a72e9037" containerName="main"
Oct 30 01:05:41.650049 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:41.649893    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="b88fa1c1-bbb8-4f74-bc2a-3cff7b647082" containerName="istio-proxy"
Oct 30 01:05:41.650049 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:41.649900    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="b88fa1c1-bbb8-4f74-bc2a-3cff7b647082" containerName="istio-proxy"
Oct 30 01:05:41.653778 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:41.653754    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5007e1-l9fgc"
Oct 30 01:05:41.672427 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:41.670313    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:05:41.672427 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:41.670962    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:05:41.673639 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:41.673615    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5007e1-l9fgc"]
Oct 30 01:05:41.742210 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:41.742178    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:05:41.742414 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:41.742374    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5phsh\""
Oct 30 01:05:41.804971 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:41.804937    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftv6v\" (UniqueName: \"kubernetes.io/projected/76f31114-038a-49a6-b835-163b1b593037-kube-api-access-ftv6v\") pod \"maas-api-revoke-keys-e2e-inf-b-5007e1-l9fgc\" (UID: \"76f31114-038a-49a6-b835-163b1b593037\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5007e1-l9fgc"
Oct 30 01:05:41.805145 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:41.805069    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76f31114-038a-49a6-b835-163b1b593037-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-5007e1-l9fgc\" (UID: \"76f31114-038a-49a6-b835-163b1b593037\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5007e1-l9fgc"
Oct 30 01:05:41.805145 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:41.805098    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76f31114-038a-49a6-b835-163b1b593037-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-5007e1-l9fgc\" (UID: \"76f31114-038a-49a6-b835-163b1b593037\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5007e1-l9fgc"
Oct 30 01:05:41.906805 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:41.906706    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftv6v\" (UniqueName: \"kubernetes.io/projected/76f31114-038a-49a6-b835-163b1b593037-kube-api-access-ftv6v\") pod \"maas-api-revoke-keys-e2e-inf-b-5007e1-l9fgc\" (UID: \"76f31114-038a-49a6-b835-163b1b593037\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5007e1-l9fgc"
Oct 30 01:05:41.906990 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:41.906815    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76f31114-038a-49a6-b835-163b1b593037-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-5007e1-l9fgc\" (UID: \"76f31114-038a-49a6-b835-163b1b593037\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5007e1-l9fgc"
Oct 30 01:05:41.906990 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:41.906845    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76f31114-038a-49a6-b835-163b1b593037-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-5007e1-l9fgc\" (UID: \"76f31114-038a-49a6-b835-163b1b593037\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5007e1-l9fgc"
Oct 30 01:05:41.907747 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:41.907706    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76f31114-038a-49a6-b835-163b1b593037-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-5007e1-l9fgc\" (UID: \"76f31114-038a-49a6-b835-163b1b593037\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5007e1-l9fgc"
Oct 30 01:05:41.909632 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:41.909605    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76f31114-038a-49a6-b835-163b1b593037-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-5007e1-l9fgc\" (UID: \"76f31114-038a-49a6-b835-163b1b593037\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5007e1-l9fgc"
Oct 30 01:05:41.931617 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:41.931591    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftv6v\" (UniqueName: \"kubernetes.io/projected/76f31114-038a-49a6-b835-163b1b593037-kube-api-access-ftv6v\") pod \"maas-api-revoke-keys-e2e-inf-b-5007e1-l9fgc\" (UID: \"76f31114-038a-49a6-b835-163b1b593037\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5007e1-l9fgc"
Oct 30 01:05:41.973200 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:41.973153    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5007e1-l9fgc"
Oct 30 01:05:42.282593 ip-10-0-128-7 kubenswrapper[5002]: W1030 01:05:42.282555    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76f31114_038a_49a6_b835_163b1b593037.slice/crio-98e5b8d507e42da301bacdb14aa52f5228314bb54c65dc2c620412c6cbe9abc2 WatchSource:0}: Error finding container 98e5b8d507e42da301bacdb14aa52f5228314bb54c65dc2c620412c6cbe9abc2: Status 404 returned error can't find the container with id 98e5b8d507e42da301bacdb14aa52f5228314bb54c65dc2c620412c6cbe9abc2
Oct 30 01:05:42.285128 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:42.285099    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5007e1-l9fgc"]
Oct 30 01:05:42.408403 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:42.384935    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7d8z4"]
Oct 30 01:05:42.712900 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:42.712859    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5007e1-l9fgc" event={"ID":"76f31114-038a-49a6-b835-163b1b593037","Type":"ContainerStarted","Data":"98e5b8d507e42da301bacdb14aa52f5228314bb54c65dc2c620412c6cbe9abc2"}
Oct 30 01:05:42.868477 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:05:42.868433    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbjdq" containerName="extractor"
Oct 30 01:05:43.791474 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:43.787409    5002 generic.go:356] "Generic (PLEG): container finished" podID="76f31114-038a-49a6-b835-163b1b593037" containerID="e92c3c6ceacc638f96136b0ceb8bb52aee9679f5602773934393242c2093a2fe" exitCode=0
Oct 30 01:05:43.791474 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:43.787540    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5007e1-l9fgc" event={"ID":"76f31114-038a-49a6-b835-163b1b593037","Type":"ContainerDied","Data":"e92c3c6ceacc638f96136b0ceb8bb52aee9679f5602773934393242c2093a2fe"}
Oct 30 01:05:45.086813 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:45.086743    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5007e1-l9fgc"
Oct 30 01:05:45.230585 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:45.230556    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/76f31114-038a-49a6-b835-163b1b593037-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76f31114-038a-49a6-b835-163b1b593037-maas-api-xks-ca\") pod \"76f31114-038a-49a6-b835-163b1b593037\" (UID: \"76f31114-038a-49a6-b835-163b1b593037\") "
Oct 30 01:05:45.230982 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:45.230960    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76f31114-038a-49a6-b835-163b1b593037-kube-api-access-ftv6v\" (UniqueName: \"kubernetes.io/projected/76f31114-038a-49a6-b835-163b1b593037-kube-api-access-ftv6v\") pod \"76f31114-038a-49a6-b835-163b1b593037\" (UID: \"76f31114-038a-49a6-b835-163b1b593037\") "
Oct 30 01:05:45.231307 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:45.231286    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76f31114-038a-49a6-b835-163b1b593037-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76f31114-038a-49a6-b835-163b1b593037-maas-api-openshift-ca\") pod \"76f31114-038a-49a6-b835-163b1b593037\" (UID: \"76f31114-038a-49a6-b835-163b1b593037\") "
Oct 30 01:05:45.232977 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:45.231954    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76f31114-038a-49a6-b835-163b1b593037-maas-api-openshift-ca" pod "76f31114-038a-49a6-b835-163b1b593037" (UID: "76f31114-038a-49a6-b835-163b1b593037"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:05:45.232977 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:45.232601    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76f31114-038a-49a6-b835-163b1b593037-maas-api-openshift-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:05:45.233140 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:45.233112    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76f31114-038a-49a6-b835-163b1b593037-maas-api-xks-ca" pod "76f31114-038a-49a6-b835-163b1b593037" (UID: "76f31114-038a-49a6-b835-163b1b593037"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:05:45.236464 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:45.234569    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76f31114-038a-49a6-b835-163b1b593037-kube-api-access-ftv6v" pod "76f31114-038a-49a6-b835-163b1b593037" (UID: "76f31114-038a-49a6-b835-163b1b593037"). InnerVolumeSpecName "kube-api-access-ftv6v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:05:45.334249 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:45.334207    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ftv6v\" (UniqueName: \"kubernetes.io/projected/76f31114-038a-49a6-b835-163b1b593037-kube-api-access-ftv6v\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:05:45.334249 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:45.334242    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76f31114-038a-49a6-b835-163b1b593037-maas-api-xks-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:05:45.815129 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:45.815100    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5007e1-l9fgc"
Oct 30 01:05:45.815129 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:45.815120    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5007e1-l9fgc" event={"ID":"76f31114-038a-49a6-b835-163b1b593037","Type":"ContainerDied","Data":"98e5b8d507e42da301bacdb14aa52f5228314bb54c65dc2c620412c6cbe9abc2"}
Oct 30 01:05:45.815422 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:45.815163    5002 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="98e5b8d507e42da301bacdb14aa52f5228314bb54c65dc2c620412c6cbe9abc2"
Oct 30 01:05:46.504943 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:46.504895    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-4989x"
Oct 30 01:05:46.686404 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:46.685878    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf2f68d2-28a3-4488-b3c5-82d2e278c311-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf2f68d2-28a3-4488-b3c5-82d2e278c311-workload-certs\") pod \"bf2f68d2-28a3-4488-b3c5-82d2e278c311\" (UID: \"bf2f68d2-28a3-4488-b3c5-82d2e278c311\") "
Oct 30 01:05:46.686404 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:46.685987    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf2f68d2-28a3-4488-b3c5-82d2e278c311-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf2f68d2-28a3-4488-b3c5-82d2e278c311-istio-data\") pod \"bf2f68d2-28a3-4488-b3c5-82d2e278c311\" (UID: \"bf2f68d2-28a3-4488-b3c5-82d2e278c311\") "
Oct 30 01:05:46.686404 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:46.686062    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf2f68d2-28a3-4488-b3c5-82d2e278c311-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf2f68d2-28a3-4488-b3c5-82d2e278c311-istiod-ca-cert\") pod \"bf2f68d2-28a3-4488-b3c5-82d2e278c311\" (UID: \"bf2f68d2-28a3-4488-b3c5-82d2e278c311\") "
Oct 30 01:05:46.686404 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:46.686312    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf2f68d2-28a3-4488-b3c5-82d2e278c311-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf2f68d2-28a3-4488-b3c5-82d2e278c311-credential-socket\") pod \"bf2f68d2-28a3-4488-b3c5-82d2e278c311\" (UID: \"bf2f68d2-28a3-4488-b3c5-82d2e278c311\") "
Oct 30 01:05:46.686404 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:46.686361    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf2f68d2-28a3-4488-b3c5-82d2e278c311-istio-token\" (UniqueName: \"kubernetes.io/projected/bf2f68d2-28a3-4488-b3c5-82d2e278c311-istio-token\") pod \"bf2f68d2-28a3-4488-b3c5-82d2e278c311\" (UID: \"bf2f68d2-28a3-4488-b3c5-82d2e278c311\") "
Oct 30 01:05:46.686854 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:46.686442    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf2f68d2-28a3-4488-b3c5-82d2e278c311-kube-api-access-ncxg4\" (UniqueName: \"kubernetes.io/projected/bf2f68d2-28a3-4488-b3c5-82d2e278c311-kube-api-access-ncxg4\") pod \"bf2f68d2-28a3-4488-b3c5-82d2e278c311\" (UID: \"bf2f68d2-28a3-4488-b3c5-82d2e278c311\") "
Oct 30 01:05:46.686854 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:46.686473    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf2f68d2-28a3-4488-b3c5-82d2e278c311-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf2f68d2-28a3-4488-b3c5-82d2e278c311-workload-socket\") pod \"bf2f68d2-28a3-4488-b3c5-82d2e278c311\" (UID: \"bf2f68d2-28a3-4488-b3c5-82d2e278c311\") "
Oct 30 01:05:46.686854 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:46.686517    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bf2f68d2-28a3-4488-b3c5-82d2e278c311-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf2f68d2-28a3-4488-b3c5-82d2e278c311-istio-podinfo\") pod \"bf2f68d2-28a3-4488-b3c5-82d2e278c311\" (UID: \"bf2f68d2-28a3-4488-b3c5-82d2e278c311\") "
Oct 30 01:05:46.686854 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:46.686563    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf2f68d2-28a3-4488-b3c5-82d2e278c311-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf2f68d2-28a3-4488-b3c5-82d2e278c311-istio-envoy\") pod \"bf2f68d2-28a3-4488-b3c5-82d2e278c311\" (UID: \"bf2f68d2-28a3-4488-b3c5-82d2e278c311\") "
Oct 30 01:05:46.689135 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:46.687912    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf2f68d2-28a3-4488-b3c5-82d2e278c311-credential-socket" pod "bf2f68d2-28a3-4488-b3c5-82d2e278c311" (UID: "bf2f68d2-28a3-4488-b3c5-82d2e278c311"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:05:46.689135 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:46.688243    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf2f68d2-28a3-4488-b3c5-82d2e278c311-workload-certs" pod "bf2f68d2-28a3-4488-b3c5-82d2e278c311" (UID: "bf2f68d2-28a3-4488-b3c5-82d2e278c311"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:05:46.689135 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:46.688296    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf2f68d2-28a3-4488-b3c5-82d2e278c311-workload-socket" pod "bf2f68d2-28a3-4488-b3c5-82d2e278c311" (UID: "bf2f68d2-28a3-4488-b3c5-82d2e278c311"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:05:46.689135 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:46.688505    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf2f68d2-28a3-4488-b3c5-82d2e278c311-istio-data" pod "bf2f68d2-28a3-4488-b3c5-82d2e278c311" (UID: "bf2f68d2-28a3-4488-b3c5-82d2e278c311"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:05:46.689135 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:46.688862    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf2f68d2-28a3-4488-b3c5-82d2e278c311-istiod-ca-cert" pod "bf2f68d2-28a3-4488-b3c5-82d2e278c311" (UID: "bf2f68d2-28a3-4488-b3c5-82d2e278c311"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:05:46.690604 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:46.690579    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf2f68d2-28a3-4488-b3c5-82d2e278c311-istio-envoy" pod "bf2f68d2-28a3-4488-b3c5-82d2e278c311" (UID: "bf2f68d2-28a3-4488-b3c5-82d2e278c311"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:05:46.691984 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:46.691542    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bf2f68d2-28a3-4488-b3c5-82d2e278c311-istio-podinfo" pod "bf2f68d2-28a3-4488-b3c5-82d2e278c311" (UID: "bf2f68d2-28a3-4488-b3c5-82d2e278c311"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:05:46.691984 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:46.691621    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf2f68d2-28a3-4488-b3c5-82d2e278c311-kube-api-access-ncxg4" pod "bf2f68d2-28a3-4488-b3c5-82d2e278c311" (UID: "bf2f68d2-28a3-4488-b3c5-82d2e278c311"). InnerVolumeSpecName "kube-api-access-ncxg4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:05:46.692345 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:46.692277    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf2f68d2-28a3-4488-b3c5-82d2e278c311-istio-token" pod "bf2f68d2-28a3-4488-b3c5-82d2e278c311" (UID: "bf2f68d2-28a3-4488-b3c5-82d2e278c311"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:05:46.789144 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:46.789103    5002 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf2f68d2-28a3-4488-b3c5-82d2e278c311-istio-data\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:05:46.789427 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:46.789407    5002 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf2f68d2-28a3-4488-b3c5-82d2e278c311-istiod-ca-cert\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:05:46.789538 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:46.789526    5002 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf2f68d2-28a3-4488-b3c5-82d2e278c311-credential-socket\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:05:46.789637 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:46.789626    5002 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf2f68d2-28a3-4488-b3c5-82d2e278c311-istio-token\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:05:46.789716 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:46.789705    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ncxg4\" (UniqueName: \"kubernetes.io/projected/bf2f68d2-28a3-4488-b3c5-82d2e278c311-kube-api-access-ncxg4\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:05:46.789791 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:46.789781    5002 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf2f68d2-28a3-4488-b3c5-82d2e278c311-workload-socket\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:05:46.789878 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:46.789868    5002 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf2f68d2-28a3-4488-b3c5-82d2e278c311-istio-podinfo\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:05:46.789958 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:46.789948    5002 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf2f68d2-28a3-4488-b3c5-82d2e278c311-istio-envoy\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:05:46.790040 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:46.790030    5002 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf2f68d2-28a3-4488-b3c5-82d2e278c311-workload-certs\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:05:46.826622 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:46.826583    5002 generic.go:356] "Generic (PLEG): container finished" podID="bf2f68d2-28a3-4488-b3c5-82d2e278c311" containerID="fb68e95a1d2cd04cf1310ef73c36ee324858282514e4098464b09faced7c7239" exitCode=0
Oct 30 01:05:46.826821 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:46.826765    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-4989x" event={"ID":"bf2f68d2-28a3-4488-b3c5-82d2e278c311","Type":"ContainerDied","Data":"fb68e95a1d2cd04cf1310ef73c36ee324858282514e4098464b09faced7c7239"}
Oct 30 01:05:46.826821 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:46.826814    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-4989x" event={"ID":"bf2f68d2-28a3-4488-b3c5-82d2e278c311","Type":"ContainerDied","Data":"ddaeffcdaad623856514cb3f81997d6166dec591ea445808af9e832374dfaee0"}
Oct 30 01:05:46.826955 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:46.826839    5002 scope.go:122] "RemoveContainer" containerID="fb68e95a1d2cd04cf1310ef73c36ee324858282514e4098464b09faced7c7239"
Oct 30 01:05:46.827029 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:46.827012    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-4989x"
Oct 30 01:05:46.842143 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:46.842122    5002 scope.go:122] "RemoveContainer" containerID="fb68e95a1d2cd04cf1310ef73c36ee324858282514e4098464b09faced7c7239"
Oct 30 01:05:46.842481 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:05:46.842449    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb68e95a1d2cd04cf1310ef73c36ee324858282514e4098464b09faced7c7239\": container with ID starting with fb68e95a1d2cd04cf1310ef73c36ee324858282514e4098464b09faced7c7239 not found: ID does not exist" containerID="fb68e95a1d2cd04cf1310ef73c36ee324858282514e4098464b09faced7c7239"
Oct 30 01:05:46.842565 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:46.842494    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb68e95a1d2cd04cf1310ef73c36ee324858282514e4098464b09faced7c7239"} err="failed to get container status \"fb68e95a1d2cd04cf1310ef73c36ee324858282514e4098464b09faced7c7239\": rpc error: code = NotFound desc = could not find container \"fb68e95a1d2cd04cf1310ef73c36ee324858282514e4098464b09faced7c7239\": container with ID starting with fb68e95a1d2cd04cf1310ef73c36ee324858282514e4098464b09faced7c7239 not found: ID does not exist"
Oct 30 01:05:47.031603 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:47.031474    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-4989x"]
Oct 30 01:05:47.064402 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:47.042434    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-4989x"]
Oct 30 01:05:47.333659 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:47.332463    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-51b95e-tts6l"]
Oct 30 01:05:47.333659 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:47.333402    5002 cpu_manager.go:414] "containerMap: removing container" podUID="bf2f68d2-28a3-4488-b3c5-82d2e278c311" containerName="istio-proxy"
Oct 30 01:05:47.333659 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:47.333427    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf2f68d2-28a3-4488-b3c5-82d2e278c311" containerName="istio-proxy"
Oct 30 01:05:47.333659 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:47.333451    5002 cpu_manager.go:414] "containerMap: removing container" podUID="76f31114-038a-49a6-b835-163b1b593037" containerName="revoke-keys"
Oct 30 01:05:47.333659 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:47.333458    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76f31114-038a-49a6-b835-163b1b593037" containerName="revoke-keys"
Oct 30 01:05:47.334040 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:47.333697    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf2f68d2-28a3-4488-b3c5-82d2e278c311" containerName="istio-proxy"
Oct 30 01:05:47.334040 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:47.333710    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="76f31114-038a-49a6-b835-163b1b593037" containerName="revoke-keys"
Oct 30 01:05:47.345128 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:47.343667    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-51b95e-tts6l"]
Oct 30 01:05:47.345128 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:47.343776    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-51b95e-tts6l"
Oct 30 01:05:47.391420 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:47.370862    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:05:47.391420 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:47.371023    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5phsh\""
Oct 30 01:05:47.391420 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:47.371074    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:05:47.391420 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:47.371787    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:05:47.447248 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:47.446999    5002 scope.go:122] "RemoveContainer" containerID="16bd7ee14049867db2b6e6436c1b25286ba975787ec688d92c4b3b67f856edbe"
Oct 30 01:05:47.508059 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:47.508023    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47c257a4-a208-4824-bbac-cb04c325b2f6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-51b95e-tts6l\" (UID: \"47c257a4-a208-4824-bbac-cb04c325b2f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-51b95e-tts6l"
Oct 30 01:05:47.508627 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:47.508092    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvdlw\" (UniqueName: \"kubernetes.io/projected/47c257a4-a208-4824-bbac-cb04c325b2f6-kube-api-access-zvdlw\") pod \"maas-api-revoke-keys-e2e-inf-a-51b95e-tts6l\" (UID: \"47c257a4-a208-4824-bbac-cb04c325b2f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-51b95e-tts6l"
Oct 30 01:05:47.508627 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:47.508265    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47c257a4-a208-4824-bbac-cb04c325b2f6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-51b95e-tts6l\" (UID: \"47c257a4-a208-4824-bbac-cb04c325b2f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-51b95e-tts6l"
Oct 30 01:05:47.609290 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:47.609192    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47c257a4-a208-4824-bbac-cb04c325b2f6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-51b95e-tts6l\" (UID: \"47c257a4-a208-4824-bbac-cb04c325b2f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-51b95e-tts6l"
Oct 30 01:05:47.609290 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:47.609277    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvdlw\" (UniqueName: \"kubernetes.io/projected/47c257a4-a208-4824-bbac-cb04c325b2f6-kube-api-access-zvdlw\") pod \"maas-api-revoke-keys-e2e-inf-a-51b95e-tts6l\" (UID: \"47c257a4-a208-4824-bbac-cb04c325b2f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-51b95e-tts6l"
Oct 30 01:05:47.609619 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:47.609588    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47c257a4-a208-4824-bbac-cb04c325b2f6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-51b95e-tts6l\" (UID: \"47c257a4-a208-4824-bbac-cb04c325b2f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-51b95e-tts6l"
Oct 30 01:05:47.610412 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:47.610352    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47c257a4-a208-4824-bbac-cb04c325b2f6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-51b95e-tts6l\" (UID: \"47c257a4-a208-4824-bbac-cb04c325b2f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-51b95e-tts6l"
Oct 30 01:05:47.612542 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:47.612516    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47c257a4-a208-4824-bbac-cb04c325b2f6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-51b95e-tts6l\" (UID: \"47c257a4-a208-4824-bbac-cb04c325b2f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-51b95e-tts6l"
Oct 30 01:05:47.634778 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:47.634744    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvdlw\" (UniqueName: \"kubernetes.io/projected/47c257a4-a208-4824-bbac-cb04c325b2f6-kube-api-access-zvdlw\") pod \"maas-api-revoke-keys-e2e-inf-a-51b95e-tts6l\" (UID: \"47c257a4-a208-4824-bbac-cb04c325b2f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-51b95e-tts6l"
Oct 30 01:05:47.712880 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:47.712842    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-51b95e-tts6l"
Oct 30 01:05:48.085577 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:48.085356    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-51b95e-tts6l"]
Oct 30 01:05:48.088820 ip-10-0-128-7 kubenswrapper[5002]: W1030 01:05:48.088787    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47c257a4_a208_4824_bbac_cb04c325b2f6.slice/crio-0fa5238140a0b966c9ba2f6fb840021709a31c5174e627f32dd82e81a6267715 WatchSource:0}: Error finding container 0fa5238140a0b966c9ba2f6fb840021709a31c5174e627f32dd82e81a6267715: Status 404 returned error can't find the container with id 0fa5238140a0b966c9ba2f6fb840021709a31c5174e627f32dd82e81a6267715
Oct 30 01:05:48.499196 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:48.499156    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf2f68d2-28a3-4488-b3c5-82d2e278c311" path="/var/lib/kubelet/pods/bf2f68d2-28a3-4488-b3c5-82d2e278c311/volumes"
Oct 30 01:05:48.915223 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:48.915131    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-51b95e-tts6l" event={"ID":"47c257a4-a208-4824-bbac-cb04c325b2f6","Type":"ContainerStarted","Data":"96a9f524441ea093b606df9abbdc0df63ccfb654ebf95c844dc87f348387d3b9"}
Oct 30 01:05:48.915223 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:48.915197    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-51b95e-tts6l" event={"ID":"47c257a4-a208-4824-bbac-cb04c325b2f6","Type":"ContainerStarted","Data":"0fa5238140a0b966c9ba2f6fb840021709a31c5174e627f32dd82e81a6267715"}
Oct 30 01:05:49.025642 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:49.025564    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-51b95e-tts6l" podStartSLOduration=2.025543217 podStartE2EDuration="2.025543217s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:05:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:05:48.98772826 +0000 UTC m=+2044.604161142" watchObservedRunningTime="2026-10-30 01:05:49.025543217 +0000 UTC m=+2044.641976097"
Oct 30 01:05:49.927438 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:49.927361    5002 generic.go:356] "Generic (PLEG): container finished" podID="47c257a4-a208-4824-bbac-cb04c325b2f6" containerID="96a9f524441ea093b606df9abbdc0df63ccfb654ebf95c844dc87f348387d3b9" exitCode=0
Oct 30 01:05:49.927843 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:49.927464    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-51b95e-tts6l" event={"ID":"47c257a4-a208-4824-bbac-cb04c325b2f6","Type":"ContainerDied","Data":"96a9f524441ea093b606df9abbdc0df63ccfb654ebf95c844dc87f348387d3b9"}
Oct 30 01:05:51.152430 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:51.151473    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-51b95e-tts6l"
Oct 30 01:05:51.275467 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:51.274639    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/47c257a4-a208-4824-bbac-cb04c325b2f6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47c257a4-a208-4824-bbac-cb04c325b2f6-maas-api-xks-ca\") pod \"47c257a4-a208-4824-bbac-cb04c325b2f6\" (UID: \"47c257a4-a208-4824-bbac-cb04c325b2f6\") "
Oct 30 01:05:51.275467 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:51.274687    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47c257a4-a208-4824-bbac-cb04c325b2f6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47c257a4-a208-4824-bbac-cb04c325b2f6-maas-api-openshift-ca\") pod \"47c257a4-a208-4824-bbac-cb04c325b2f6\" (UID: \"47c257a4-a208-4824-bbac-cb04c325b2f6\") "
Oct 30 01:05:51.275467 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:51.274767    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47c257a4-a208-4824-bbac-cb04c325b2f6-kube-api-access-zvdlw\" (UniqueName: \"kubernetes.io/projected/47c257a4-a208-4824-bbac-cb04c325b2f6-kube-api-access-zvdlw\") pod \"47c257a4-a208-4824-bbac-cb04c325b2f6\" (UID: \"47c257a4-a208-4824-bbac-cb04c325b2f6\") "
Oct 30 01:05:51.275467 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:51.275348    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47c257a4-a208-4824-bbac-cb04c325b2f6-maas-api-openshift-ca" pod "47c257a4-a208-4824-bbac-cb04c325b2f6" (UID: "47c257a4-a208-4824-bbac-cb04c325b2f6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:05:51.276127 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:51.276101    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47c257a4-a208-4824-bbac-cb04c325b2f6-maas-api-openshift-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:05:51.277838 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:51.277810    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47c257a4-a208-4824-bbac-cb04c325b2f6-kube-api-access-zvdlw" pod "47c257a4-a208-4824-bbac-cb04c325b2f6" (UID: "47c257a4-a208-4824-bbac-cb04c325b2f6"). InnerVolumeSpecName "kube-api-access-zvdlw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:05:51.278538 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:51.278507    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47c257a4-a208-4824-bbac-cb04c325b2f6-maas-api-xks-ca" pod "47c257a4-a208-4824-bbac-cb04c325b2f6" (UID: "47c257a4-a208-4824-bbac-cb04c325b2f6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:05:51.378837 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:51.378212    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47c257a4-a208-4824-bbac-cb04c325b2f6-maas-api-xks-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:05:51.378837 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:51.378251    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvdlw\" (UniqueName: \"kubernetes.io/projected/47c257a4-a208-4824-bbac-cb04c325b2f6-kube-api-access-zvdlw\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:05:51.946155 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:51.946123    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-51b95e-tts6l"
Oct 30 01:05:51.946155 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:51.946136    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-51b95e-tts6l" event={"ID":"47c257a4-a208-4824-bbac-cb04c325b2f6","Type":"ContainerDied","Data":"0fa5238140a0b966c9ba2f6fb840021709a31c5174e627f32dd82e81a6267715"}
Oct 30 01:05:51.946442 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:51.946181    5002 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0fa5238140a0b966c9ba2f6fb840021709a31c5174e627f32dd82e81a6267715"
Oct 30 01:05:52.098166 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:52.098093    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-5007e1-d7bb8bb4c-xbqmg"]
Oct 30 01:05:54.649070 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:54.649034    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-19bc68-openshift-default-7b6485f77f-s2xkw"]
Oct 30 01:05:54.650035 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:54.650001    5002 cpu_manager.go:414] "containerMap: removing container" podUID="47c257a4-a208-4824-bbac-cb04c325b2f6" containerName="revoke-keys"
Oct 30 01:05:54.650189 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:54.650054    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47c257a4-a208-4824-bbac-cb04c325b2f6" containerName="revoke-keys"
Oct 30 01:05:54.650277 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:54.650264    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="47c257a4-a208-4824-bbac-cb04c325b2f6" containerName="revoke-keys"
Oct 30 01:05:54.653460 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:54.653435    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-19bc68-openshift-default-7b6485f77f-s2xkw"
Oct 30 01:05:54.670465 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:54.670403    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-19bc68-openshift-default-dockercfg-vd65x\""
Oct 30 01:05:54.691670 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:54.691121    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-19bc68-openshift-default-7b6485f77f-s2xkw"]
Oct 30 01:05:54.745018 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:54.744974    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ae75f2d-c292-43f7-a59c-b1a09261c412-istiod-ca-cert\") pod \"e2e-inf-b-19bc68-openshift-default-7b6485f77f-s2xkw\" (UID: \"9ae75f2d-c292-43f7-a59c-b1a09261c412\") " pod="openshift-ingress/e2e-inf-b-19bc68-openshift-default-7b6485f77f-s2xkw"
Oct 30 01:05:54.745232 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:54.745033    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ae75f2d-c292-43f7-a59c-b1a09261c412-istio-podinfo\") pod \"e2e-inf-b-19bc68-openshift-default-7b6485f77f-s2xkw\" (UID: \"9ae75f2d-c292-43f7-a59c-b1a09261c412\") " pod="openshift-ingress/e2e-inf-b-19bc68-openshift-default-7b6485f77f-s2xkw"
Oct 30 01:05:54.745232 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:54.745112    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ae75f2d-c292-43f7-a59c-b1a09261c412-istio-token\") pod \"e2e-inf-b-19bc68-openshift-default-7b6485f77f-s2xkw\" (UID: \"9ae75f2d-c292-43f7-a59c-b1a09261c412\") " pod="openshift-ingress/e2e-inf-b-19bc68-openshift-default-7b6485f77f-s2xkw"
Oct 30 01:05:54.745232 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:54.745138    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ae75f2d-c292-43f7-a59c-b1a09261c412-istio-data\") pod \"e2e-inf-b-19bc68-openshift-default-7b6485f77f-s2xkw\" (UID: \"9ae75f2d-c292-43f7-a59c-b1a09261c412\") " pod="openshift-ingress/e2e-inf-b-19bc68-openshift-default-7b6485f77f-s2xkw"
Oct 30 01:05:54.745232 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:54.745182    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ae75f2d-c292-43f7-a59c-b1a09261c412-workload-socket\") pod \"e2e-inf-b-19bc68-openshift-default-7b6485f77f-s2xkw\" (UID: \"9ae75f2d-c292-43f7-a59c-b1a09261c412\") " pod="openshift-ingress/e2e-inf-b-19bc68-openshift-default-7b6485f77f-s2xkw"
Oct 30 01:05:54.745232 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:54.745214    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ae75f2d-c292-43f7-a59c-b1a09261c412-workload-certs\") pod \"e2e-inf-b-19bc68-openshift-default-7b6485f77f-s2xkw\" (UID: \"9ae75f2d-c292-43f7-a59c-b1a09261c412\") " pod="openshift-ingress/e2e-inf-b-19bc68-openshift-default-7b6485f77f-s2xkw"
Oct 30 01:05:54.745572 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:54.745277    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ae75f2d-c292-43f7-a59c-b1a09261c412-istio-envoy\") pod \"e2e-inf-b-19bc68-openshift-default-7b6485f77f-s2xkw\" (UID: \"9ae75f2d-c292-43f7-a59c-b1a09261c412\") " pod="openshift-ingress/e2e-inf-b-19bc68-openshift-default-7b6485f77f-s2xkw"
Oct 30 01:05:54.745572 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:54.745309    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bv8tn\" (UniqueName: \"kubernetes.io/projected/9ae75f2d-c292-43f7-a59c-b1a09261c412-kube-api-access-bv8tn\") pod \"e2e-inf-b-19bc68-openshift-default-7b6485f77f-s2xkw\" (UID: \"9ae75f2d-c292-43f7-a59c-b1a09261c412\") " pod="openshift-ingress/e2e-inf-b-19bc68-openshift-default-7b6485f77f-s2xkw"
Oct 30 01:05:54.745572 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:54.745373    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ae75f2d-c292-43f7-a59c-b1a09261c412-credential-socket\") pod \"e2e-inf-b-19bc68-openshift-default-7b6485f77f-s2xkw\" (UID: \"9ae75f2d-c292-43f7-a59c-b1a09261c412\") " pod="openshift-ingress/e2e-inf-b-19bc68-openshift-default-7b6485f77f-s2xkw"
Oct 30 01:05:54.846970 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:54.846875    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ae75f2d-c292-43f7-a59c-b1a09261c412-istiod-ca-cert\") pod \"e2e-inf-b-19bc68-openshift-default-7b6485f77f-s2xkw\" (UID: \"9ae75f2d-c292-43f7-a59c-b1a09261c412\") " pod="openshift-ingress/e2e-inf-b-19bc68-openshift-default-7b6485f77f-s2xkw"
Oct 30 01:05:54.846970 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:54.846922    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ae75f2d-c292-43f7-a59c-b1a09261c412-istio-podinfo\") pod \"e2e-inf-b-19bc68-openshift-default-7b6485f77f-s2xkw\" (UID: \"9ae75f2d-c292-43f7-a59c-b1a09261c412\") " pod="openshift-ingress/e2e-inf-b-19bc68-openshift-default-7b6485f77f-s2xkw"
Oct 30 01:05:54.847206 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:54.846998    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ae75f2d-c292-43f7-a59c-b1a09261c412-istio-token\") pod \"e2e-inf-b-19bc68-openshift-default-7b6485f77f-s2xkw\" (UID: \"9ae75f2d-c292-43f7-a59c-b1a09261c412\") " pod="openshift-ingress/e2e-inf-b-19bc68-openshift-default-7b6485f77f-s2xkw"
Oct 30 01:05:54.847206 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:54.847030    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ae75f2d-c292-43f7-a59c-b1a09261c412-istio-data\") pod \"e2e-inf-b-19bc68-openshift-default-7b6485f77f-s2xkw\" (UID: \"9ae75f2d-c292-43f7-a59c-b1a09261c412\") " pod="openshift-ingress/e2e-inf-b-19bc68-openshift-default-7b6485f77f-s2xkw"
Oct 30 01:05:54.847206 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:54.847091    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ae75f2d-c292-43f7-a59c-b1a09261c412-workload-socket\") pod \"e2e-inf-b-19bc68-openshift-default-7b6485f77f-s2xkw\" (UID: \"9ae75f2d-c292-43f7-a59c-b1a09261c412\") " pod="openshift-ingress/e2e-inf-b-19bc68-openshift-default-7b6485f77f-s2xkw"
Oct 30 01:05:54.847206 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:54.847119    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ae75f2d-c292-43f7-a59c-b1a09261c412-workload-certs\") pod \"e2e-inf-b-19bc68-openshift-default-7b6485f77f-s2xkw\" (UID: \"9ae75f2d-c292-43f7-a59c-b1a09261c412\") " pod="openshift-ingress/e2e-inf-b-19bc68-openshift-default-7b6485f77f-s2xkw"
Oct 30 01:05:54.847206 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:54.847177    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ae75f2d-c292-43f7-a59c-b1a09261c412-istio-envoy\") pod \"e2e-inf-b-19bc68-openshift-default-7b6485f77f-s2xkw\" (UID: \"9ae75f2d-c292-43f7-a59c-b1a09261c412\") " pod="openshift-ingress/e2e-inf-b-19bc68-openshift-default-7b6485f77f-s2xkw"
Oct 30 01:05:54.847577 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:54.847209    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bv8tn\" (UniqueName: \"kubernetes.io/projected/9ae75f2d-c292-43f7-a59c-b1a09261c412-kube-api-access-bv8tn\") pod \"e2e-inf-b-19bc68-openshift-default-7b6485f77f-s2xkw\" (UID: \"9ae75f2d-c292-43f7-a59c-b1a09261c412\") " pod="openshift-ingress/e2e-inf-b-19bc68-openshift-default-7b6485f77f-s2xkw"
Oct 30 01:05:54.847577 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:54.847276    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ae75f2d-c292-43f7-a59c-b1a09261c412-credential-socket\") pod \"e2e-inf-b-19bc68-openshift-default-7b6485f77f-s2xkw\" (UID: \"9ae75f2d-c292-43f7-a59c-b1a09261c412\") " pod="openshift-ingress/e2e-inf-b-19bc68-openshift-default-7b6485f77f-s2xkw"
Oct 30 01:05:54.847785 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:54.847761    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ae75f2d-c292-43f7-a59c-b1a09261c412-istiod-ca-cert\") pod \"e2e-inf-b-19bc68-openshift-default-7b6485f77f-s2xkw\" (UID: \"9ae75f2d-c292-43f7-a59c-b1a09261c412\") " pod="openshift-ingress/e2e-inf-b-19bc68-openshift-default-7b6485f77f-s2xkw"
Oct 30 01:05:54.847864 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:54.847802    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ae75f2d-c292-43f7-a59c-b1a09261c412-workload-certs\") pod \"e2e-inf-b-19bc68-openshift-default-7b6485f77f-s2xkw\" (UID: \"9ae75f2d-c292-43f7-a59c-b1a09261c412\") " pod="openshift-ingress/e2e-inf-b-19bc68-openshift-default-7b6485f77f-s2xkw"
Oct 30 01:05:54.847864 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:54.847845    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ae75f2d-c292-43f7-a59c-b1a09261c412-workload-socket\") pod \"e2e-inf-b-19bc68-openshift-default-7b6485f77f-s2xkw\" (UID: \"9ae75f2d-c292-43f7-a59c-b1a09261c412\") " pod="openshift-ingress/e2e-inf-b-19bc68-openshift-default-7b6485f77f-s2xkw"
Oct 30 01:05:54.847963 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:54.847906    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ae75f2d-c292-43f7-a59c-b1a09261c412-istio-data\") pod \"e2e-inf-b-19bc68-openshift-default-7b6485f77f-s2xkw\" (UID: \"9ae75f2d-c292-43f7-a59c-b1a09261c412\") " pod="openshift-ingress/e2e-inf-b-19bc68-openshift-default-7b6485f77f-s2xkw"
Oct 30 01:05:54.848076 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:54.848058    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ae75f2d-c292-43f7-a59c-b1a09261c412-credential-socket\") pod \"e2e-inf-b-19bc68-openshift-default-7b6485f77f-s2xkw\" (UID: \"9ae75f2d-c292-43f7-a59c-b1a09261c412\") " pod="openshift-ingress/e2e-inf-b-19bc68-openshift-default-7b6485f77f-s2xkw"
Oct 30 01:05:54.849861 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:54.849829    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ae75f2d-c292-43f7-a59c-b1a09261c412-istio-podinfo\") pod \"e2e-inf-b-19bc68-openshift-default-7b6485f77f-s2xkw\" (UID: \"9ae75f2d-c292-43f7-a59c-b1a09261c412\") " pod="openshift-ingress/e2e-inf-b-19bc68-openshift-default-7b6485f77f-s2xkw"
Oct 30 01:05:54.849979 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:54.849957    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ae75f2d-c292-43f7-a59c-b1a09261c412-istio-envoy\") pod \"e2e-inf-b-19bc68-openshift-default-7b6485f77f-s2xkw\" (UID: \"9ae75f2d-c292-43f7-a59c-b1a09261c412\") " pod="openshift-ingress/e2e-inf-b-19bc68-openshift-default-7b6485f77f-s2xkw"
Oct 30 01:05:54.871866 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:54.871838    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ae75f2d-c292-43f7-a59c-b1a09261c412-istio-token\") pod \"e2e-inf-b-19bc68-openshift-default-7b6485f77f-s2xkw\" (UID: \"9ae75f2d-c292-43f7-a59c-b1a09261c412\") " pod="openshift-ingress/e2e-inf-b-19bc68-openshift-default-7b6485f77f-s2xkw"
Oct 30 01:05:54.872501 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:54.872476    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bv8tn\" (UniqueName: \"kubernetes.io/projected/9ae75f2d-c292-43f7-a59c-b1a09261c412-kube-api-access-bv8tn\") pod \"e2e-inf-b-19bc68-openshift-default-7b6485f77f-s2xkw\" (UID: \"9ae75f2d-c292-43f7-a59c-b1a09261c412\") " pod="openshift-ingress/e2e-inf-b-19bc68-openshift-default-7b6485f77f-s2xkw"
Oct 30 01:05:54.969898 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:54.969862    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-19bc68-openshift-default-7b6485f77f-s2xkw"
Oct 30 01:05:54.980717 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:54.980675    5002 generic.go:356] "Generic (PLEG): container finished" podID="b38dbda1-9a06-461d-8d0a-a9413b76d333" containerID="b245cf8d44445c2ed5ef47ce7669b2c108f4555f746c312e4322c4795e538938" exitCode=0
Oct 30 01:05:54.980998 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:54.980976    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-73435a-openshift-default-fc454477f-x6g2b" event={"ID":"b38dbda1-9a06-461d-8d0a-a9413b76d333","Type":"ContainerDied","Data":"b245cf8d44445c2ed5ef47ce7669b2c108f4555f746c312e4322c4795e538938"}
Oct 30 01:05:54.981586 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:54.981565    5002 scope.go:122] "RemoveContainer" containerID="b245cf8d44445c2ed5ef47ce7669b2c108f4555f746c312e4322c4795e538938"
Oct 30 01:05:54.985007 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:54.984970    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:05:54.985109 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:54.985061    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:05:54.985154 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:54.985112    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:05:55.303698 ip-10-0-128-7 kubenswrapper[5002]: W1030 01:05:55.303662    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ae75f2d_c292_43f7_a59c_b1a09261c412.slice/crio-38d7495c4366b067976057983bb979e5dd0483f4608d7c530a53fde118d0e7f7 WatchSource:0}: Error finding container 38d7495c4366b067976057983bb979e5dd0483f4608d7c530a53fde118d0e7f7: Status 404 returned error can't find the container with id 38d7495c4366b067976057983bb979e5dd0483f4608d7c530a53fde118d0e7f7
Oct 30 01:05:55.305336 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:55.305308    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-19bc68-openshift-default-7b6485f77f-s2xkw"]
Oct 30 01:05:55.306891 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:55.306854    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:05:55.306998 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:55.306918    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:05:55.306998 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:55.306949    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:05:55.998474 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:55.995504    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-19bc68-openshift-default-7b6485f77f-s2xkw" event={"ID":"9ae75f2d-c292-43f7-a59c-b1a09261c412","Type":"ContainerStarted","Data":"08cfeae333f5df907e1e0bc809b425d2345451236767828cf70d27df76f6f3d3"}
Oct 30 01:05:55.998474 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:55.995568    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-19bc68-openshift-default-7b6485f77f-s2xkw" event={"ID":"9ae75f2d-c292-43f7-a59c-b1a09261c412","Type":"ContainerStarted","Data":"38d7495c4366b067976057983bb979e5dd0483f4608d7c530a53fde118d0e7f7"}
Oct 30 01:05:55.999219 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:55.999144    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-73435a-openshift-default-fc454477f-x6g2b" event={"ID":"b38dbda1-9a06-461d-8d0a-a9413b76d333","Type":"ContainerStarted","Data":"83add8a04a5e9f8469035acafb8674f0fbc3bf3e48da05e04540c95c8ba8038d"}
Oct 30 01:05:56.066404 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:56.057098    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-19bc68-openshift-default-7b6485f77f-s2xkw" podStartSLOduration=2.05707519 podStartE2EDuration="2.05707519s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:05:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:05:56.045015063 +0000 UTC m=+2051.661447956" watchObservedRunningTime="2026-10-30 01:05:56.05707519 +0000 UTC m=+2051.673508070"
Oct 30 01:05:56.522428 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:56.522319    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-73435a-openshift-default-fc454477f-x6g2b"
Oct 30 01:05:56.524015 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:56.523965    5002 patch_prober.go:29] interesting pod/e2e-shared-b-w4-73435a-openshift-default-fc454477f-x6g2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body=
Oct 30 01:05:56.524189 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:56.524027    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-73435a-openshift-default-fc454477f-x6g2b" podUID="b38dbda1-9a06-461d-8d0a-a9413b76d333" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused"
Oct 30 01:05:56.984406 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:56.979218    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-19bc68-openshift-default-7b6485f77f-s2xkw"
Oct 30 01:05:57.006473 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:56.997526    5002 patch_prober.go:29] interesting pod/e2e-inf-b-19bc68-openshift-default-7b6485f77f-s2xkw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" start-of-body=
Oct 30 01:05:57.006473 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:56.997600    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-19bc68-openshift-default-7b6485f77f-s2xkw" podUID="9ae75f2d-c292-43f7-a59c-b1a09261c412" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused"
Oct 30 01:05:57.038103 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:57.038056    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-21ccafef-sr46p"]
Oct 30 01:05:57.072277 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:57.072239    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-21ccafef-sr46p"]
Oct 30 01:05:57.103416 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:57.102484    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5007e1-d7bb8bb4c-xbqmg" podUID="fa5f1440-5fa9-40e6-b85f-073f31d24d01" containerName="payload-pre-processing" containerID="cri-o://246310ce3175bbeb48494f66550d34f30d161f99fe59d9b707fb69a6a7e502fe" gracePeriod=25
Oct 30 01:05:57.543431 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:57.542511    5002 patch_prober.go:29] interesting pod/e2e-shared-b-w4-73435a-openshift-default-fc454477f-x6g2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body=
Oct 30 01:05:57.543431 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:57.542571    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-73435a-openshift-default-fc454477f-x6g2b" podUID="b38dbda1-9a06-461d-8d0a-a9413b76d333" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused"
Oct 30 01:05:58.136405 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:58.123156    5002 patch_prober.go:29] interesting pod/e2e-inf-b-19bc68-openshift-default-7b6485f77f-s2xkw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" start-of-body=
Oct 30 01:05:58.136405 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:58.123220    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-19bc68-openshift-default-7b6485f77f-s2xkw" podUID="9ae75f2d-c292-43f7-a59c-b1a09261c412" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused"
Oct 30 01:05:58.528249 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:58.528206    5002 patch_prober.go:29] interesting pod/e2e-shared-b-w4-73435a-openshift-default-fc454477f-x6g2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body=
Oct 30 01:05:58.528477 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:58.528271    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-73435a-openshift-default-fc454477f-x6g2b" podUID="b38dbda1-9a06-461d-8d0a-a9413b76d333" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused"
Oct 30 01:05:58.544052 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:58.543476    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a5c1b79-e06c-4fdc-ae81-694a1368e991" path="/var/lib/kubelet/pods/0a5c1b79-e06c-4fdc-ae81-694a1368e991/volumes"
Oct 30 01:05:58.977809 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:58.977766    5002 patch_prober.go:29] interesting pod/e2e-inf-b-19bc68-openshift-default-7b6485f77f-s2xkw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" start-of-body=
Oct 30 01:05:58.978108 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:58.978073    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-19bc68-openshift-default-7b6485f77f-s2xkw" podUID="9ae75f2d-c292-43f7-a59c-b1a09261c412" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused"
Oct 30 01:05:59.320081 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:59.318452    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-51b95e-tts6l"]
Oct 30 01:05:59.355412 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:59.349616    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-51b95e-tts6l"]
Oct 30 01:05:59.522426 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:59.522362    5002 patch_prober.go:29] interesting pod/e2e-shared-b-w4-73435a-openshift-default-fc454477f-x6g2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body=
Oct 30 01:05:59.522610 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:59.522445    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-73435a-openshift-default-fc454477f-x6g2b" podUID="b38dbda1-9a06-461d-8d0a-a9413b76d333" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused"
Oct 30 01:05:59.983411 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:59.981026    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-73435a-openshift-default-fc454477f-x6g2b"
Oct 30 01:05:59.983411 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:59.982831    5002 patch_prober.go:29] interesting pod/e2e-inf-b-19bc68-openshift-default-7b6485f77f-s2xkw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" start-of-body=
Oct 30 01:05:59.983411 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:05:59.982883    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-19bc68-openshift-default-7b6485f77f-s2xkw" podUID="9ae75f2d-c292-43f7-a59c-b1a09261c412" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused"
Oct 30 01:06:00.522511 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:00.521831    5002 patch_prober.go:29] interesting pod/e2e-shared-b-w4-73435a-openshift-default-fc454477f-x6g2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body=
Oct 30 01:06:00.522511 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:00.521902    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-73435a-openshift-default-fc454477f-x6g2b" podUID="b38dbda1-9a06-461d-8d0a-a9413b76d333" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused"
Oct 30 01:06:00.525715 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:00.525277    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47c257a4-a208-4824-bbac-cb04c325b2f6" path="/var/lib/kubelet/pods/47c257a4-a208-4824-bbac-cb04c325b2f6/volumes"
Oct 30 01:06:00.973406 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:00.973340    5002 patch_prober.go:29] interesting pod/e2e-inf-b-19bc68-openshift-default-7b6485f77f-s2xkw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" start-of-body=
Oct 30 01:06:00.973723 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:00.973686    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-19bc68-openshift-default-7b6485f77f-s2xkw" podUID="9ae75f2d-c292-43f7-a59c-b1a09261c412" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused"
Oct 30 01:06:01.524805 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:01.524767    5002 patch_prober.go:29] interesting pod/e2e-shared-b-w4-73435a-openshift-default-fc454477f-x6g2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body=
Oct 30 01:06:01.525481 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:01.525433    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-73435a-openshift-default-fc454477f-x6g2b" podUID="b38dbda1-9a06-461d-8d0a-a9413b76d333" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused"
Oct 30 01:06:01.778909 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:01.778798    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5007e1-d7bb8bb4c-xbqmg" podUID="fa5f1440-5fa9-40e6-b85f-073f31d24d01" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:06:02.012563 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:02.012514    5002 patch_prober.go:29] interesting pod/e2e-inf-b-19bc68-openshift-default-7b6485f77f-s2xkw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" start-of-body=
Oct 30 01:06:02.012773 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:02.012592    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-19bc68-openshift-default-7b6485f77f-s2xkw" podUID="9ae75f2d-c292-43f7-a59c-b1a09261c412" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused"
Oct 30 01:06:02.393271 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:02.393222    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-wjg5k"]
Oct 30 01:06:02.399990 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:02.399967    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-wjg5k"
Oct 30 01:06:02.434630 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:02.434577    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-wjg5k"]
Oct 30 01:06:02.509562 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:02.509516    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9821719-d280-4bc0-a68b-9ae8def2955f-istio-data\") pod \"e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-wjg5k\" (UID: \"c9821719-d280-4bc0-a68b-9ae8def2955f\") " pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-wjg5k"
Oct 30 01:06:02.509824 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:02.509807    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9821719-d280-4bc0-a68b-9ae8def2955f-istio-envoy\") pod \"e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-wjg5k\" (UID: \"c9821719-d280-4bc0-a68b-9ae8def2955f\") " pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-wjg5k"
Oct 30 01:06:02.509989 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:02.509974    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9821719-d280-4bc0-a68b-9ae8def2955f-istiod-ca-cert\") pod \"e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-wjg5k\" (UID: \"c9821719-d280-4bc0-a68b-9ae8def2955f\") " pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-wjg5k"
Oct 30 01:06:02.510195 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:02.510182    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9821719-d280-4bc0-a68b-9ae8def2955f-credential-socket\") pod \"e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-wjg5k\" (UID: \"c9821719-d280-4bc0-a68b-9ae8def2955f\") " pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-wjg5k"
Oct 30 01:06:02.510302 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:02.510288    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4m7v\" (UniqueName: \"kubernetes.io/projected/c9821719-d280-4bc0-a68b-9ae8def2955f-kube-api-access-f4m7v\") pod \"e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-wjg5k\" (UID: \"c9821719-d280-4bc0-a68b-9ae8def2955f\") " pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-wjg5k"
Oct 30 01:06:02.510460 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:02.510446    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9821719-d280-4bc0-a68b-9ae8def2955f-workload-socket\") pod \"e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-wjg5k\" (UID: \"c9821719-d280-4bc0-a68b-9ae8def2955f\") " pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-wjg5k"
Oct 30 01:06:02.510607 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:02.510594    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9821719-d280-4bc0-a68b-9ae8def2955f-workload-certs\") pod \"e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-wjg5k\" (UID: \"c9821719-d280-4bc0-a68b-9ae8def2955f\") " pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-wjg5k"
Oct 30 01:06:02.510816 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:02.510799    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9821719-d280-4bc0-a68b-9ae8def2955f-istio-token\") pod \"e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-wjg5k\" (UID: \"c9821719-d280-4bc0-a68b-9ae8def2955f\") " pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-wjg5k"
Oct 30 01:06:02.510920 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:02.510906    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9821719-d280-4bc0-a68b-9ae8def2955f-istio-podinfo\") pod \"e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-wjg5k\" (UID: \"c9821719-d280-4bc0-a68b-9ae8def2955f\") " pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-wjg5k"
Oct 30 01:06:02.528559 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:02.528528    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-73435a-openshift-default-fc454477f-x6g2b"
Oct 30 01:06:02.530511 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:02.530483    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-73435a-openshift-default-fc454477f-x6g2b"
Oct 30 01:06:02.611795 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:02.611762    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9821719-d280-4bc0-a68b-9ae8def2955f-istio-envoy\") pod \"e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-wjg5k\" (UID: \"c9821719-d280-4bc0-a68b-9ae8def2955f\") " pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-wjg5k"
Oct 30 01:06:02.611955 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:02.611835    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9821719-d280-4bc0-a68b-9ae8def2955f-istiod-ca-cert\") pod \"e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-wjg5k\" (UID: \"c9821719-d280-4bc0-a68b-9ae8def2955f\") " pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-wjg5k"
Oct 30 01:06:02.611955 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:02.611949    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9821719-d280-4bc0-a68b-9ae8def2955f-credential-socket\") pod \"e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-wjg5k\" (UID: \"c9821719-d280-4bc0-a68b-9ae8def2955f\") " pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-wjg5k"
Oct 30 01:06:02.612085 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:02.611979    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4m7v\" (UniqueName: \"kubernetes.io/projected/c9821719-d280-4bc0-a68b-9ae8def2955f-kube-api-access-f4m7v\") pod \"e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-wjg5k\" (UID: \"c9821719-d280-4bc0-a68b-9ae8def2955f\") " pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-wjg5k"
Oct 30 01:06:02.612085 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:02.612017    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9821719-d280-4bc0-a68b-9ae8def2955f-workload-socket\") pod \"e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-wjg5k\" (UID: \"c9821719-d280-4bc0-a68b-9ae8def2955f\") " pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-wjg5k"
Oct 30 01:06:02.612085 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:02.612056    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9821719-d280-4bc0-a68b-9ae8def2955f-workload-certs\") pod \"e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-wjg5k\" (UID: \"c9821719-d280-4bc0-a68b-9ae8def2955f\") " pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-wjg5k"
Oct 30 01:06:02.612250 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:02.612140    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9821719-d280-4bc0-a68b-9ae8def2955f-istio-token\") pod \"e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-wjg5k\" (UID: \"c9821719-d280-4bc0-a68b-9ae8def2955f\") " pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-wjg5k"
Oct 30 01:06:02.612250 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:02.612162    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9821719-d280-4bc0-a68b-9ae8def2955f-istio-podinfo\") pod \"e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-wjg5k\" (UID: \"c9821719-d280-4bc0-a68b-9ae8def2955f\") " pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-wjg5k"
Oct 30 01:06:02.612250 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:02.612198    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9821719-d280-4bc0-a68b-9ae8def2955f-istio-data\") pod \"e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-wjg5k\" (UID: \"c9821719-d280-4bc0-a68b-9ae8def2955f\") " pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-wjg5k"
Oct 30 01:06:02.612439 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:02.612370    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9821719-d280-4bc0-a68b-9ae8def2955f-credential-socket\") pod \"e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-wjg5k\" (UID: \"c9821719-d280-4bc0-a68b-9ae8def2955f\") " pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-wjg5k"
Oct 30 01:06:02.612720 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:02.612691    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9821719-d280-4bc0-a68b-9ae8def2955f-istiod-ca-cert\") pod \"e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-wjg5k\" (UID: \"c9821719-d280-4bc0-a68b-9ae8def2955f\") " pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-wjg5k"
Oct 30 01:06:02.612809 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:02.612792    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9821719-d280-4bc0-a68b-9ae8def2955f-istio-data\") pod \"e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-wjg5k\" (UID: \"c9821719-d280-4bc0-a68b-9ae8def2955f\") " pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-wjg5k"
Oct 30 01:06:02.613134 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:02.613106    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9821719-d280-4bc0-a68b-9ae8def2955f-workload-certs\") pod \"e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-wjg5k\" (UID: \"c9821719-d280-4bc0-a68b-9ae8def2955f\") " pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-wjg5k"
Oct 30 01:06:02.613134 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:02.613121    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9821719-d280-4bc0-a68b-9ae8def2955f-workload-socket\") pod \"e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-wjg5k\" (UID: \"c9821719-d280-4bc0-a68b-9ae8def2955f\") " pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-wjg5k"
Oct 30 01:06:02.614937 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:02.614914    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9821719-d280-4bc0-a68b-9ae8def2955f-istio-envoy\") pod \"e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-wjg5k\" (UID: \"c9821719-d280-4bc0-a68b-9ae8def2955f\") " pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-wjg5k"
Oct 30 01:06:02.615502 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:02.615480    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9821719-d280-4bc0-a68b-9ae8def2955f-istio-podinfo\") pod \"e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-wjg5k\" (UID: \"c9821719-d280-4bc0-a68b-9ae8def2955f\") " pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-wjg5k"
Oct 30 01:06:02.644055 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:02.643569    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9821719-d280-4bc0-a68b-9ae8def2955f-istio-token\") pod \"e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-wjg5k\" (UID: \"c9821719-d280-4bc0-a68b-9ae8def2955f\") " pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-wjg5k"
Oct 30 01:06:02.645003 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:02.644909    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4m7v\" (UniqueName: \"kubernetes.io/projected/c9821719-d280-4bc0-a68b-9ae8def2955f-kube-api-access-f4m7v\") pod \"e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-wjg5k\" (UID: \"c9821719-d280-4bc0-a68b-9ae8def2955f\") " pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-wjg5k"
Oct 30 01:06:02.750984 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:02.750947    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-wjg5k"
Oct 30 01:06:02.980992 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:02.979797    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-19bc68-openshift-default-7b6485f77f-s2xkw"
Oct 30 01:06:02.981276 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:02.981207    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-19bc68-openshift-default-7b6485f77f-s2xkw"
Oct 30 01:06:02.982301 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:02.982281    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-19bc68-openshift-default-7b6485f77f-s2xkw"
Oct 30 01:06:03.132053 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:03.131787    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-wjg5k"]
Oct 30 01:06:03.142073 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:03.140233    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:06:03.142073 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:03.140325    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:06:03.142073 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:03.140370    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:06:03.337237 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:03.337199    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-wjg5k" event={"ID":"c9821719-d280-4bc0-a68b-9ae8def2955f","Type":"ContainerStarted","Data":"eb53ef389b0116b6e54e58b159963fbf155dbd45f4a24d1e26ffa3374df3001a"}
Oct 30 01:06:04.346009 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:04.345967    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-wjg5k" event={"ID":"c9821719-d280-4bc0-a68b-9ae8def2955f","Type":"ContainerStarted","Data":"415acff5a5892a0fd490333e00beffe6cc9790c80f032f230b1c475d35735f0e"}
Oct 30 01:06:04.430362 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:04.430301    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-wjg5k" podStartSLOduration=2.430285969 podStartE2EDuration="2.430285969s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:06:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:06:04.412311926 +0000 UTC m=+2060.028744806" watchObservedRunningTime="2026-10-30 01:06:04.430285969 +0000 UTC m=+2060.046718854"
Oct 30 01:06:04.766404 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:04.760184    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-wjg5k"
Oct 30 01:06:04.766404 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:04.760321    5002 patch_prober.go:29] interesting pod/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-wjg5k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body=
Oct 30 01:06:04.766404 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:04.760366    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-wjg5k" podUID="c9821719-d280-4bc0-a68b-9ae8def2955f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused"
Oct 30 01:06:05.770402 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:05.767800    5002 patch_prober.go:29] interesting pod/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-wjg5k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body=
Oct 30 01:06:05.770402 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:05.767865    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-wjg5k" podUID="c9821719-d280-4bc0-a68b-9ae8def2955f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused"
Oct 30 01:06:06.121437 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:06.119797    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-51b95e-openshift-default-74bf6d5cb5-swcm7"]
Oct 30 01:06:06.121437 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:06.120044    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-51b95e-openshift-default-74bf6d5cb5-swcm7" podUID="2bb71dc1-0795-464e-9887-a6b5ce483890" containerName="istio-proxy" containerID="cri-o://287ceb9a066879306acc32029703e4c5c72d27e5212a8150a5cbf3e5368df071" gracePeriod=30
Oct 30 01:06:06.631056 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:06.630950    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b1f6ed-682td"]
Oct 30 01:06:06.634935 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:06.634908    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b1f6ed-682td"
Oct 30 01:06:06.677885 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:06.670866    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b1f6ed-682td"]
Oct 30 01:06:06.677885 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:06.674848    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:06:06.677885 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:06.675607    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:06:06.677885 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:06.676604    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/285b4ad9-1165-475d-953e-838e3fe08fad-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-b1f6ed-682td\" (UID: \"285b4ad9-1165-475d-953e-838e3fe08fad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b1f6ed-682td"
Oct 30 01:06:06.677885 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:06.676723    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/285b4ad9-1165-475d-953e-838e3fe08fad-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-b1f6ed-682td\" (UID: \"285b4ad9-1165-475d-953e-838e3fe08fad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b1f6ed-682td"
Oct 30 01:06:06.677885 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:06.676768    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pm2ff\" (UniqueName: \"kubernetes.io/projected/285b4ad9-1165-475d-953e-838e3fe08fad-kube-api-access-pm2ff\") pod \"maas-api-revoke-keys-e2e-inf-b-b1f6ed-682td\" (UID: \"285b4ad9-1165-475d-953e-838e3fe08fad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b1f6ed-682td"
Oct 30 01:06:06.732394 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:06.732342    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:06:06.732558 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:06.732502    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5phsh\""
Oct 30 01:06:06.771415 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:06.755791    5002 patch_prober.go:29] interesting pod/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-wjg5k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body=
Oct 30 01:06:06.771415 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:06.755858    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-wjg5k" podUID="c9821719-d280-4bc0-a68b-9ae8def2955f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused"
Oct 30 01:06:06.787413 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:06.784011    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/285b4ad9-1165-475d-953e-838e3fe08fad-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-b1f6ed-682td\" (UID: \"285b4ad9-1165-475d-953e-838e3fe08fad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b1f6ed-682td"
Oct 30 01:06:06.787413 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:06.784133    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/285b4ad9-1165-475d-953e-838e3fe08fad-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-b1f6ed-682td\" (UID: \"285b4ad9-1165-475d-953e-838e3fe08fad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b1f6ed-682td"
Oct 30 01:06:06.787413 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:06.784177    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pm2ff\" (UniqueName: \"kubernetes.io/projected/285b4ad9-1165-475d-953e-838e3fe08fad-kube-api-access-pm2ff\") pod \"maas-api-revoke-keys-e2e-inf-b-b1f6ed-682td\" (UID: \"285b4ad9-1165-475d-953e-838e3fe08fad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b1f6ed-682td"
Oct 30 01:06:06.787413 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:06.786073    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/285b4ad9-1165-475d-953e-838e3fe08fad-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-b1f6ed-682td\" (UID: \"285b4ad9-1165-475d-953e-838e3fe08fad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b1f6ed-682td"
Oct 30 01:06:06.796015 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:06.795947    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/285b4ad9-1165-475d-953e-838e3fe08fad-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-b1f6ed-682td\" (UID: \"285b4ad9-1165-475d-953e-838e3fe08fad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b1f6ed-682td"
Oct 30 01:06:06.810184 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:06.810155    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pm2ff\" (UniqueName: \"kubernetes.io/projected/285b4ad9-1165-475d-953e-838e3fe08fad-kube-api-access-pm2ff\") pod \"maas-api-revoke-keys-e2e-inf-b-b1f6ed-682td\" (UID: \"285b4ad9-1165-475d-953e-838e3fe08fad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b1f6ed-682td"
Oct 30 01:06:07.001507 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:07.001474    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b1f6ed-682td"
Oct 30 01:06:07.384984 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:07.384901    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b1f6ed-682td"]
Oct 30 01:06:07.391855 ip-10-0-128-7 kubenswrapper[5002]: W1030 01:06:07.391821    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod285b4ad9_1165_475d_953e_838e3fe08fad.slice/crio-2a3f6ff0b5e290dae66ab4d8cb82292de38ad52f1d61e0627175c5b310a8cd07 WatchSource:0}: Error finding container 2a3f6ff0b5e290dae66ab4d8cb82292de38ad52f1d61e0627175c5b310a8cd07: Status 404 returned error can't find the container with id 2a3f6ff0b5e290dae66ab4d8cb82292de38ad52f1d61e0627175c5b310a8cd07
Oct 30 01:06:07.442012 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:07.441952    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b1f6ed-682td" event={"ID":"285b4ad9-1165-475d-953e-838e3fe08fad","Type":"ContainerStarted","Data":"2a3f6ff0b5e290dae66ab4d8cb82292de38ad52f1d61e0627175c5b310a8cd07"}
Oct 30 01:06:07.760427 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:07.755970    5002 patch_prober.go:29] interesting pod/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-wjg5k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body=
Oct 30 01:06:07.760427 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:07.756029    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-wjg5k" podUID="c9821719-d280-4bc0-a68b-9ae8def2955f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused"
Oct 30 01:06:08.469405 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:08.462036    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b1f6ed-682td" event={"ID":"285b4ad9-1165-475d-953e-838e3fe08fad","Type":"ContainerStarted","Data":"6bade2f59c598a7ce37e2ef462b40fcac82048901e72be8273836c88efeaabdf"}
Oct 30 01:06:08.754259 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:08.753748    5002 patch_prober.go:29] interesting pod/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-wjg5k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body=
Oct 30 01:06:08.754259 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:08.753820    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-wjg5k" podUID="c9821719-d280-4bc0-a68b-9ae8def2955f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused"
Oct 30 01:06:09.485351 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:09.485305    5002 generic.go:356] "Generic (PLEG): container finished" podID="285b4ad9-1165-475d-953e-838e3fe08fad" containerID="6bade2f59c598a7ce37e2ef462b40fcac82048901e72be8273836c88efeaabdf" exitCode=0
Oct 30 01:06:09.485956 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:09.485446    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b1f6ed-682td" event={"ID":"285b4ad9-1165-475d-953e-838e3fe08fad","Type":"ContainerDied","Data":"6bade2f59c598a7ce37e2ef462b40fcac82048901e72be8273836c88efeaabdf"}
Oct 30 01:06:09.752143 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:09.752057    5002 patch_prober.go:29] interesting pod/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-wjg5k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body=
Oct 30 01:06:09.752143 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:09.752117    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-wjg5k" podUID="c9821719-d280-4bc0-a68b-9ae8def2955f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused"
Oct 30 01:06:10.681394 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:10.681349    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b1f6ed-682td"
Oct 30 01:06:10.757949 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:10.757840    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/285b4ad9-1165-475d-953e-838e3fe08fad-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/285b4ad9-1165-475d-953e-838e3fe08fad-maas-api-xks-ca\") pod \"285b4ad9-1165-475d-953e-838e3fe08fad\" (UID: \"285b4ad9-1165-475d-953e-838e3fe08fad\") "
Oct 30 01:06:10.758125 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:10.757977    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/285b4ad9-1165-475d-953e-838e3fe08fad-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/285b4ad9-1165-475d-953e-838e3fe08fad-maas-api-openshift-ca\") pod \"285b4ad9-1165-475d-953e-838e3fe08fad\" (UID: \"285b4ad9-1165-475d-953e-838e3fe08fad\") "
Oct 30 01:06:10.758125 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:10.758112    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/285b4ad9-1165-475d-953e-838e3fe08fad-kube-api-access-pm2ff\" (UniqueName: \"kubernetes.io/projected/285b4ad9-1165-475d-953e-838e3fe08fad-kube-api-access-pm2ff\") pod \"285b4ad9-1165-475d-953e-838e3fe08fad\" (UID: \"285b4ad9-1165-475d-953e-838e3fe08fad\") "
Oct 30 01:06:10.758756 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:10.758725    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-wjg5k"
Oct 30 01:06:10.758885 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:10.758763    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/285b4ad9-1165-475d-953e-838e3fe08fad-maas-api-openshift-ca" pod "285b4ad9-1165-475d-953e-838e3fe08fad" (UID: "285b4ad9-1165-475d-953e-838e3fe08fad"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:06:10.759189 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:10.759156    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-wjg5k"
Oct 30 01:06:10.759857 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:10.759839    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-wjg5k"
Oct 30 01:06:10.760609 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:10.760586    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/285b4ad9-1165-475d-953e-838e3fe08fad-kube-api-access-pm2ff" pod "285b4ad9-1165-475d-953e-838e3fe08fad" (UID: "285b4ad9-1165-475d-953e-838e3fe08fad"). InnerVolumeSpecName "kube-api-access-pm2ff". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:06:10.761943 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:10.761925    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/285b4ad9-1165-475d-953e-838e3fe08fad-maas-api-xks-ca" pod "285b4ad9-1165-475d-953e-838e3fe08fad" (UID: "285b4ad9-1165-475d-953e-838e3fe08fad"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:06:10.861638 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:10.861603    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/285b4ad9-1165-475d-953e-838e3fe08fad-maas-api-xks-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:06:10.861638 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:10.861643    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/285b4ad9-1165-475d-953e-838e3fe08fad-maas-api-openshift-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:06:10.861638 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:10.861661    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pm2ff\" (UniqueName: \"kubernetes.io/projected/285b4ad9-1165-475d-953e-838e3fe08fad-kube-api-access-pm2ff\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:06:11.442071 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:11.442042    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-51b95e-openshift-default-74bf6d5cb5-swcm7"
Oct 30 01:06:11.488085 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:11.488046    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2bb71dc1-0795-464e-9887-a6b5ce483890-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bb71dc1-0795-464e-9887-a6b5ce483890-istiod-ca-cert\") pod \"2bb71dc1-0795-464e-9887-a6b5ce483890\" (UID: \"2bb71dc1-0795-464e-9887-a6b5ce483890\") "
Oct 30 01:06:11.488267 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:11.488175    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bb71dc1-0795-464e-9887-a6b5ce483890-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bb71dc1-0795-464e-9887-a6b5ce483890-workload-certs\") pod \"2bb71dc1-0795-464e-9887-a6b5ce483890\" (UID: \"2bb71dc1-0795-464e-9887-a6b5ce483890\") "
Oct 30 01:06:11.488267 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:11.488207    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bb71dc1-0795-464e-9887-a6b5ce483890-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bb71dc1-0795-464e-9887-a6b5ce483890-credential-socket\") pod \"2bb71dc1-0795-464e-9887-a6b5ce483890\" (UID: \"2bb71dc1-0795-464e-9887-a6b5ce483890\") "
Oct 30 01:06:11.488267 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:11.488259    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2bb71dc1-0795-464e-9887-a6b5ce483890-kube-api-access-4vf72\" (UniqueName: \"kubernetes.io/projected/2bb71dc1-0795-464e-9887-a6b5ce483890-kube-api-access-4vf72\") pod \"2bb71dc1-0795-464e-9887-a6b5ce483890\" (UID: \"2bb71dc1-0795-464e-9887-a6b5ce483890\") "
Oct 30 01:06:11.488267 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:11.488286    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bb71dc1-0795-464e-9887-a6b5ce483890-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bb71dc1-0795-464e-9887-a6b5ce483890-workload-socket\") pod \"2bb71dc1-0795-464e-9887-a6b5ce483890\" (UID: \"2bb71dc1-0795-464e-9887-a6b5ce483890\") "
Oct 30 01:06:11.488580 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:11.488332    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bb71dc1-0795-464e-9887-a6b5ce483890-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bb71dc1-0795-464e-9887-a6b5ce483890-istio-envoy\") pod \"2bb71dc1-0795-464e-9887-a6b5ce483890\" (UID: \"2bb71dc1-0795-464e-9887-a6b5ce483890\") "
Oct 30 01:06:11.488860 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:11.488829    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bb71dc1-0795-464e-9887-a6b5ce483890-credential-socket" pod "2bb71dc1-0795-464e-9887-a6b5ce483890" (UID: "2bb71dc1-0795-464e-9887-a6b5ce483890"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:06:11.489012 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:11.488930    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bb71dc1-0795-464e-9887-a6b5ce483890-workload-socket" pod "2bb71dc1-0795-464e-9887-a6b5ce483890" (UID: "2bb71dc1-0795-464e-9887-a6b5ce483890"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:06:11.489089 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:11.489061    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bb71dc1-0795-464e-9887-a6b5ce483890-workload-certs" pod "2bb71dc1-0795-464e-9887-a6b5ce483890" (UID: "2bb71dc1-0795-464e-9887-a6b5ce483890"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:06:11.489149 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:11.489086    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bb71dc1-0795-464e-9887-a6b5ce483890-istiod-ca-cert" pod "2bb71dc1-0795-464e-9887-a6b5ce483890" (UID: "2bb71dc1-0795-464e-9887-a6b5ce483890"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:06:11.489149 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:11.489129    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bb71dc1-0795-464e-9887-a6b5ce483890-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bb71dc1-0795-464e-9887-a6b5ce483890-istio-data\") pod \"2bb71dc1-0795-464e-9887-a6b5ce483890\" (UID: \"2bb71dc1-0795-464e-9887-a6b5ce483890\") "
Oct 30 01:06:11.489280 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:11.489213    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2bb71dc1-0795-464e-9887-a6b5ce483890-istio-token\" (UniqueName: \"kubernetes.io/projected/2bb71dc1-0795-464e-9887-a6b5ce483890-istio-token\") pod \"2bb71dc1-0795-464e-9887-a6b5ce483890\" (UID: \"2bb71dc1-0795-464e-9887-a6b5ce483890\") "
Oct 30 01:06:11.489354 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:11.489280    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2bb71dc1-0795-464e-9887-a6b5ce483890-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bb71dc1-0795-464e-9887-a6b5ce483890-istio-podinfo\") pod \"2bb71dc1-0795-464e-9887-a6b5ce483890\" (UID: \"2bb71dc1-0795-464e-9887-a6b5ce483890\") "
Oct 30 01:06:11.489685 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:11.489642    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bb71dc1-0795-464e-9887-a6b5ce483890-istio-data" pod "2bb71dc1-0795-464e-9887-a6b5ce483890" (UID: "2bb71dc1-0795-464e-9887-a6b5ce483890"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:06:11.489685 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:11.491539    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bb71dc1-0795-464e-9887-a6b5ce483890-kube-api-access-4vf72" pod "2bb71dc1-0795-464e-9887-a6b5ce483890" (UID: "2bb71dc1-0795-464e-9887-a6b5ce483890"). InnerVolumeSpecName "kube-api-access-4vf72". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:06:11.489685 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:11.492762    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bb71dc1-0795-464e-9887-a6b5ce483890-istio-envoy" pod "2bb71dc1-0795-464e-9887-a6b5ce483890" (UID: "2bb71dc1-0795-464e-9887-a6b5ce483890"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:06:11.489685 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:11.504082    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bb71dc1-0795-464e-9887-a6b5ce483890-istio-token" pod "2bb71dc1-0795-464e-9887-a6b5ce483890" (UID: "2bb71dc1-0795-464e-9887-a6b5ce483890"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:06:11.506563 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:11.506537    5002 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bb71dc1-0795-464e-9887-a6b5ce483890-istiod-ca-cert\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:06:11.506721 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:11.506586    5002 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bb71dc1-0795-464e-9887-a6b5ce483890-workload-certs\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:06:11.506721 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:11.506601    5002 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bb71dc1-0795-464e-9887-a6b5ce483890-credential-socket\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:06:11.506721 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:11.506650    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4vf72\" (UniqueName: \"kubernetes.io/projected/2bb71dc1-0795-464e-9887-a6b5ce483890-kube-api-access-4vf72\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:06:11.506721 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:11.506668    5002 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bb71dc1-0795-464e-9887-a6b5ce483890-workload-socket\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:06:11.506721 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:11.506681    5002 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bb71dc1-0795-464e-9887-a6b5ce483890-istio-envoy\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:06:11.506721 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:11.506710    5002 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bb71dc1-0795-464e-9887-a6b5ce483890-istio-data\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:06:11.506721 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:11.506722    5002 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2bb71dc1-0795-464e-9887-a6b5ce483890-istio-token\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:06:11.509088 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:11.509059    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2bb71dc1-0795-464e-9887-a6b5ce483890-istio-podinfo" pod "2bb71dc1-0795-464e-9887-a6b5ce483890" (UID: "2bb71dc1-0795-464e-9887-a6b5ce483890"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:06:11.510431 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:11.510294    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b1f6ed-682td"
Oct 30 01:06:11.510431 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:11.510323    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b1f6ed-682td" event={"ID":"285b4ad9-1165-475d-953e-838e3fe08fad","Type":"ContainerDied","Data":"2a3f6ff0b5e290dae66ab4d8cb82292de38ad52f1d61e0627175c5b310a8cd07"}
Oct 30 01:06:11.510431 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:11.510371    5002 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2a3f6ff0b5e290dae66ab4d8cb82292de38ad52f1d61e0627175c5b310a8cd07"
Oct 30 01:06:11.519043 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:11.519009    5002 generic.go:356] "Generic (PLEG): container finished" podID="2bb71dc1-0795-464e-9887-a6b5ce483890" containerID="287ceb9a066879306acc32029703e4c5c72d27e5212a8150a5cbf3e5368df071" exitCode=0
Oct 30 01:06:11.519330 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:11.519053    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-51b95e-openshift-default-74bf6d5cb5-swcm7" event={"ID":"2bb71dc1-0795-464e-9887-a6b5ce483890","Type":"ContainerDied","Data":"287ceb9a066879306acc32029703e4c5c72d27e5212a8150a5cbf3e5368df071"}
Oct 30 01:06:11.520544 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:11.519374    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-51b95e-openshift-default-74bf6d5cb5-swcm7" event={"ID":"2bb71dc1-0795-464e-9887-a6b5ce483890","Type":"ContainerDied","Data":"517d8e5d7594b3f5f60089dcb0d2c615e7903aacad5141f4b664c8def9364167"}
Oct 30 01:06:11.520544 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:11.519790    5002 scope.go:122] "RemoveContainer" containerID="287ceb9a066879306acc32029703e4c5c72d27e5212a8150a5cbf3e5368df071"
Oct 30 01:06:11.520544 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:11.519086    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-51b95e-openshift-default-74bf6d5cb5-swcm7"
Oct 30 01:06:11.557599 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:11.557558    5002 scope.go:122] "RemoveContainer" containerID="525f9d8e4f3147eacee1dd4e21683a0a0d59733dec2dde54440f3169f819f90a"
Oct 30 01:06:11.572362 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:11.572317    5002 scope.go:122] "RemoveContainer" containerID="287ceb9a066879306acc32029703e4c5c72d27e5212a8150a5cbf3e5368df071"
Oct 30 01:06:11.572777 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:06:11.572751    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"287ceb9a066879306acc32029703e4c5c72d27e5212a8150a5cbf3e5368df071\": container with ID starting with 287ceb9a066879306acc32029703e4c5c72d27e5212a8150a5cbf3e5368df071 not found: ID does not exist" containerID="287ceb9a066879306acc32029703e4c5c72d27e5212a8150a5cbf3e5368df071"
Oct 30 01:06:11.572896 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:11.572788    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"287ceb9a066879306acc32029703e4c5c72d27e5212a8150a5cbf3e5368df071"} err="failed to get container status \"287ceb9a066879306acc32029703e4c5c72d27e5212a8150a5cbf3e5368df071\": rpc error: code = NotFound desc = could not find container \"287ceb9a066879306acc32029703e4c5c72d27e5212a8150a5cbf3e5368df071\": container with ID starting with 287ceb9a066879306acc32029703e4c5c72d27e5212a8150a5cbf3e5368df071 not found: ID does not exist"
Oct 30 01:06:11.572896 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:11.572812    5002 scope.go:122] "RemoveContainer" containerID="525f9d8e4f3147eacee1dd4e21683a0a0d59733dec2dde54440f3169f819f90a"
Oct 30 01:06:11.573134 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:06:11.573116    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"525f9d8e4f3147eacee1dd4e21683a0a0d59733dec2dde54440f3169f819f90a\": container with ID starting with 525f9d8e4f3147eacee1dd4e21683a0a0d59733dec2dde54440f3169f819f90a not found: ID does not exist" containerID="525f9d8e4f3147eacee1dd4e21683a0a0d59733dec2dde54440f3169f819f90a"
Oct 30 01:06:11.573195 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:11.573143    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"525f9d8e4f3147eacee1dd4e21683a0a0d59733dec2dde54440f3169f819f90a"} err="failed to get container status \"525f9d8e4f3147eacee1dd4e21683a0a0d59733dec2dde54440f3169f819f90a\": rpc error: code = NotFound desc = could not find container \"525f9d8e4f3147eacee1dd4e21683a0a0d59733dec2dde54440f3169f819f90a\": container with ID starting with 525f9d8e4f3147eacee1dd4e21683a0a0d59733dec2dde54440f3169f819f90a not found: ID does not exist"
Oct 30 01:06:11.582130 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:11.582101    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-51b95e-openshift-default-74bf6d5cb5-swcm7"]
Oct 30 01:06:11.586202 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:11.586173    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-51b95e-openshift-default-74bf6d5cb5-swcm7"]
Oct 30 01:06:11.607984 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:11.607936    5002 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bb71dc1-0795-464e-9887-a6b5ce483890-istio-podinfo\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:06:11.765438 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:11.765371    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5007e1-d7bb8bb4c-xbqmg" podUID="fa5f1440-5fa9-40e6-b85f-073f31d24d01" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:06:12.498656 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:12.498623    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2bb71dc1-0795-464e-9887-a6b5ce483890" path="/var/lib/kubelet/pods/2bb71dc1-0795-464e-9887-a6b5ce483890/volumes"
Oct 30 01:06:12.874965 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:06:12.874874    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbjdq" containerName="extractor"
Oct 30 01:06:17.384758 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:17.384660    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5007e1-l9fgc"]
Oct 30 01:06:17.404573 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:17.404127    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5007e1-l9fgc"]
Oct 30 01:06:17.497835 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:17.497801    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5007e1-d7bb8bb4c-xbqmg"
Oct 30 01:06:17.550982 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:17.550071    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa5f1440-5fa9-40e6-b85f-073f31d24d01-kube-api-access-tj54b\" (UniqueName: \"kubernetes.io/projected/fa5f1440-5fa9-40e6-b85f-073f31d24d01-kube-api-access-tj54b\") pod \"fa5f1440-5fa9-40e6-b85f-073f31d24d01\" (UID: \"fa5f1440-5fa9-40e6-b85f-073f31d24d01\") "
Oct 30 01:06:17.551418 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:17.551363    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa5f1440-5fa9-40e6-b85f-073f31d24d01-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa5f1440-5fa9-40e6-b85f-073f31d24d01-plugins-config-volume\") pod \"fa5f1440-5fa9-40e6-b85f-073f31d24d01\" (UID: \"fa5f1440-5fa9-40e6-b85f-073f31d24d01\") "
Oct 30 01:06:17.552610 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:17.552559    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa5f1440-5fa9-40e6-b85f-073f31d24d01-plugins-config-volume" pod "fa5f1440-5fa9-40e6-b85f-073f31d24d01" (UID: "fa5f1440-5fa9-40e6-b85f-073f31d24d01"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:06:17.608456 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:17.592782    5002 generic.go:356] "Generic (PLEG): container finished" podID="fa5f1440-5fa9-40e6-b85f-073f31d24d01" containerID="246310ce3175bbeb48494f66550d34f30d161f99fe59d9b707fb69a6a7e502fe" exitCode=0
Oct 30 01:06:17.608456 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:17.592830    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5007e1-d7bb8bb4c-xbqmg"
Oct 30 01:06:17.608456 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:17.592908    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5007e1-d7bb8bb4c-xbqmg" event={"ID":"fa5f1440-5fa9-40e6-b85f-073f31d24d01","Type":"ContainerDied","Data":"246310ce3175bbeb48494f66550d34f30d161f99fe59d9b707fb69a6a7e502fe"}
Oct 30 01:06:17.608456 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:17.592948    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5007e1-d7bb8bb4c-xbqmg" event={"ID":"fa5f1440-5fa9-40e6-b85f-073f31d24d01","Type":"ContainerDied","Data":"9e0af7b0e8ab4b936b6c4a09db18d8d3369655b1c2fee911f736036936095360"}
Oct 30 01:06:17.608456 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:17.592973    5002 scope.go:122] "RemoveContainer" containerID="246310ce3175bbeb48494f66550d34f30d161f99fe59d9b707fb69a6a7e502fe"
Oct 30 01:06:17.608456 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:17.602362    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa5f1440-5fa9-40e6-b85f-073f31d24d01-kube-api-access-tj54b" pod "fa5f1440-5fa9-40e6-b85f-073f31d24d01" (UID: "fa5f1440-5fa9-40e6-b85f-073f31d24d01"). InnerVolumeSpecName "kube-api-access-tj54b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:06:17.647620 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:17.647585    5002 scope.go:122] "RemoveContainer" containerID="246310ce3175bbeb48494f66550d34f30d161f99fe59d9b707fb69a6a7e502fe"
Oct 30 01:06:17.648151 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:06:17.648114    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"246310ce3175bbeb48494f66550d34f30d161f99fe59d9b707fb69a6a7e502fe\": container with ID starting with 246310ce3175bbeb48494f66550d34f30d161f99fe59d9b707fb69a6a7e502fe not found: ID does not exist" containerID="246310ce3175bbeb48494f66550d34f30d161f99fe59d9b707fb69a6a7e502fe"
Oct 30 01:06:17.648267 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:17.648164    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"246310ce3175bbeb48494f66550d34f30d161f99fe59d9b707fb69a6a7e502fe"} err="failed to get container status \"246310ce3175bbeb48494f66550d34f30d161f99fe59d9b707fb69a6a7e502fe\": rpc error: code = NotFound desc = could not find container \"246310ce3175bbeb48494f66550d34f30d161f99fe59d9b707fb69a6a7e502fe\": container with ID starting with 246310ce3175bbeb48494f66550d34f30d161f99fe59d9b707fb69a6a7e502fe not found: ID does not exist"
Oct 30 01:06:17.653641 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:17.653601    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tj54b\" (UniqueName: \"kubernetes.io/projected/fa5f1440-5fa9-40e6-b85f-073f31d24d01-kube-api-access-tj54b\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:06:17.653759 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:17.653639    5002 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa5f1440-5fa9-40e6-b85f-073f31d24d01-plugins-config-volume\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:06:17.976767 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:17.976689    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-5007e1-d7bb8bb4c-xbqmg"]
Oct 30 01:06:17.986402 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:17.982803    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-5007e1-d7bb8bb4c-xbqmg"]
Oct 30 01:06:18.496916 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:18.496881    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76f31114-038a-49a6-b835-163b1b593037" path="/var/lib/kubelet/pods/76f31114-038a-49a6-b835-163b1b593037/volumes"
Oct 30 01:06:18.497355 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:18.497340    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa5f1440-5fa9-40e6-b85f-073f31d24d01" path="/var/lib/kubelet/pods/fa5f1440-5fa9-40e6-b85f-073f31d24d01/volumes"
Oct 30 01:06:25.336478 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:25.334422    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-5007e1-openshift-default-64c644d8cc-wrs68"]
Oct 30 01:06:25.336478 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:25.335052    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-5007e1-openshift-default-64c644d8cc-wrs68" podUID="f864da04-4d22-42be-8334-d559fc8cdbe6" containerName="istio-proxy" containerID="cri-o://0cfd0e885b2e6e16bdecc6f9cb22da07156f27ae8a8ab8684e4257e298ea736b" gracePeriod=30
Oct 30 01:06:28.116595 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:28.116549    5002 patch_prober.go:29] interesting pod/e2e-inf-b-5007e1-openshift-default-64c644d8cc-wrs68 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 01:06:28.116962 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:28.116615    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-5007e1-openshift-default-64c644d8cc-wrs68" podUID="f864da04-4d22-42be-8334-d559fc8cdbe6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 01:06:30.663068 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:30.663039    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-5007e1-openshift-default-64c644d8cc-wrs68"
Oct 30 01:06:30.709495 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:30.709454    5002 generic.go:356] "Generic (PLEG): container finished" podID="f864da04-4d22-42be-8334-d559fc8cdbe6" containerID="0cfd0e885b2e6e16bdecc6f9cb22da07156f27ae8a8ab8684e4257e298ea736b" exitCode=0
Oct 30 01:06:30.709688 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:30.709498    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-5007e1-openshift-default-64c644d8cc-wrs68"
Oct 30 01:06:30.709688 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:30.709535    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5007e1-openshift-default-64c644d8cc-wrs68" event={"ID":"f864da04-4d22-42be-8334-d559fc8cdbe6","Type":"ContainerDied","Data":"0cfd0e885b2e6e16bdecc6f9cb22da07156f27ae8a8ab8684e4257e298ea736b"}
Oct 30 01:06:30.709688 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:30.709593    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5007e1-openshift-default-64c644d8cc-wrs68" event={"ID":"f864da04-4d22-42be-8334-d559fc8cdbe6","Type":"ContainerDied","Data":"b35087288c809634a370301688bf5606d59e524e2999d46b267ef706037a1fdd"}
Oct 30 01:06:30.709688 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:30.709626    5002 scope.go:122] "RemoveContainer" containerID="0cfd0e885b2e6e16bdecc6f9cb22da07156f27ae8a8ab8684e4257e298ea736b"
Oct 30 01:06:30.721300 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:30.721276    5002 scope.go:122] "RemoveContainer" containerID="0cfd0e885b2e6e16bdecc6f9cb22da07156f27ae8a8ab8684e4257e298ea736b"
Oct 30 01:06:30.721638 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:06:30.721612    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0cfd0e885b2e6e16bdecc6f9cb22da07156f27ae8a8ab8684e4257e298ea736b\": container with ID starting with 0cfd0e885b2e6e16bdecc6f9cb22da07156f27ae8a8ab8684e4257e298ea736b not found: ID does not exist" containerID="0cfd0e885b2e6e16bdecc6f9cb22da07156f27ae8a8ab8684e4257e298ea736b"
Oct 30 01:06:30.721759 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:30.721646    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0cfd0e885b2e6e16bdecc6f9cb22da07156f27ae8a8ab8684e4257e298ea736b"} err="failed to get container status \"0cfd0e885b2e6e16bdecc6f9cb22da07156f27ae8a8ab8684e4257e298ea736b\": rpc error: code = NotFound desc = could not find container \"0cfd0e885b2e6e16bdecc6f9cb22da07156f27ae8a8ab8684e4257e298ea736b\": container with ID starting with 0cfd0e885b2e6e16bdecc6f9cb22da07156f27ae8a8ab8684e4257e298ea736b not found: ID does not exist"
Oct 30 01:06:30.748182 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:30.748140    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f864da04-4d22-42be-8334-d559fc8cdbe6-kube-api-access-dlrs9\" (UniqueName: \"kubernetes.io/projected/f864da04-4d22-42be-8334-d559fc8cdbe6-kube-api-access-dlrs9\") pod \"f864da04-4d22-42be-8334-d559fc8cdbe6\" (UID: \"f864da04-4d22-42be-8334-d559fc8cdbe6\") "
Oct 30 01:06:30.748376 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:30.748303    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f864da04-4d22-42be-8334-d559fc8cdbe6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f864da04-4d22-42be-8334-d559fc8cdbe6-workload-socket\") pod \"f864da04-4d22-42be-8334-d559fc8cdbe6\" (UID: \"f864da04-4d22-42be-8334-d559fc8cdbe6\") "
Oct 30 01:06:30.748376 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:30.748350    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f864da04-4d22-42be-8334-d559fc8cdbe6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f864da04-4d22-42be-8334-d559fc8cdbe6-istio-envoy\") pod \"f864da04-4d22-42be-8334-d559fc8cdbe6\" (UID: \"f864da04-4d22-42be-8334-d559fc8cdbe6\") "
Oct 30 01:06:30.748578 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:30.748523    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f864da04-4d22-42be-8334-d559fc8cdbe6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f864da04-4d22-42be-8334-d559fc8cdbe6-workload-certs\") pod \"f864da04-4d22-42be-8334-d559fc8cdbe6\" (UID: \"f864da04-4d22-42be-8334-d559fc8cdbe6\") "
Oct 30 01:06:30.748578 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:30.748572    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f864da04-4d22-42be-8334-d559fc8cdbe6-istio-token\" (UniqueName: \"kubernetes.io/projected/f864da04-4d22-42be-8334-d559fc8cdbe6-istio-token\") pod \"f864da04-4d22-42be-8334-d559fc8cdbe6\" (UID: \"f864da04-4d22-42be-8334-d559fc8cdbe6\") "
Oct 30 01:06:30.748693 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:30.748626    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f864da04-4d22-42be-8334-d559fc8cdbe6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f864da04-4d22-42be-8334-d559fc8cdbe6-istio-podinfo\") pod \"f864da04-4d22-42be-8334-d559fc8cdbe6\" (UID: \"f864da04-4d22-42be-8334-d559fc8cdbe6\") "
Oct 30 01:06:30.748752 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:30.748698    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f864da04-4d22-42be-8334-d559fc8cdbe6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f864da04-4d22-42be-8334-d559fc8cdbe6-istio-data\") pod \"f864da04-4d22-42be-8334-d559fc8cdbe6\" (UID: \"f864da04-4d22-42be-8334-d559fc8cdbe6\") "
Oct 30 01:06:30.748804 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:30.748789    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f864da04-4d22-42be-8334-d559fc8cdbe6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f864da04-4d22-42be-8334-d559fc8cdbe6-credential-socket\") pod \"f864da04-4d22-42be-8334-d559fc8cdbe6\" (UID: \"f864da04-4d22-42be-8334-d559fc8cdbe6\") "
Oct 30 01:06:30.748862 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:30.748805    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f864da04-4d22-42be-8334-d559fc8cdbe6-workload-socket" pod "f864da04-4d22-42be-8334-d559fc8cdbe6" (UID: "f864da04-4d22-42be-8334-d559fc8cdbe6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:06:30.748862 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:30.748844    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f864da04-4d22-42be-8334-d559fc8cdbe6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f864da04-4d22-42be-8334-d559fc8cdbe6-istiod-ca-cert\") pod \"f864da04-4d22-42be-8334-d559fc8cdbe6\" (UID: \"f864da04-4d22-42be-8334-d559fc8cdbe6\") "
Oct 30 01:06:30.749545 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:30.749516    5002 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f864da04-4d22-42be-8334-d559fc8cdbe6-workload-socket\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:06:30.751044 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:30.751014    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f864da04-4d22-42be-8334-d559fc8cdbe6-istio-data" pod "f864da04-4d22-42be-8334-d559fc8cdbe6" (UID: "f864da04-4d22-42be-8334-d559fc8cdbe6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:06:30.751842 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:30.751810    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f864da04-4d22-42be-8334-d559fc8cdbe6-credential-socket" pod "f864da04-4d22-42be-8334-d559fc8cdbe6" (UID: "f864da04-4d22-42be-8334-d559fc8cdbe6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:06:30.751935 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:30.751883    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f864da04-4d22-42be-8334-d559fc8cdbe6-workload-certs" pod "f864da04-4d22-42be-8334-d559fc8cdbe6" (UID: "f864da04-4d22-42be-8334-d559fc8cdbe6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:06:30.759224 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:30.759182    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f864da04-4d22-42be-8334-d559fc8cdbe6-istio-token" pod "f864da04-4d22-42be-8334-d559fc8cdbe6" (UID: "f864da04-4d22-42be-8334-d559fc8cdbe6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:06:30.763062 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:30.762680    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f864da04-4d22-42be-8334-d559fc8cdbe6-istio-podinfo" pod "f864da04-4d22-42be-8334-d559fc8cdbe6" (UID: "f864da04-4d22-42be-8334-d559fc8cdbe6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:06:30.766994 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:30.766961    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f864da04-4d22-42be-8334-d559fc8cdbe6-istio-envoy" pod "f864da04-4d22-42be-8334-d559fc8cdbe6" (UID: "f864da04-4d22-42be-8334-d559fc8cdbe6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:06:30.777553 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:30.777525    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f864da04-4d22-42be-8334-d559fc8cdbe6-istiod-ca-cert" pod "f864da04-4d22-42be-8334-d559fc8cdbe6" (UID: "f864da04-4d22-42be-8334-d559fc8cdbe6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:06:30.778426 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:30.778348    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f864da04-4d22-42be-8334-d559fc8cdbe6-kube-api-access-dlrs9" pod "f864da04-4d22-42be-8334-d559fc8cdbe6" (UID: "f864da04-4d22-42be-8334-d559fc8cdbe6"). InnerVolumeSpecName "kube-api-access-dlrs9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:06:30.851407 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:30.850803    5002 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f864da04-4d22-42be-8334-d559fc8cdbe6-istio-envoy\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:06:30.851407 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:30.850838    5002 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f864da04-4d22-42be-8334-d559fc8cdbe6-workload-certs\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:06:30.851407 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:30.850853    5002 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f864da04-4d22-42be-8334-d559fc8cdbe6-istio-token\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:06:30.851407 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:30.850866    5002 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f864da04-4d22-42be-8334-d559fc8cdbe6-istio-podinfo\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:06:30.851407 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:30.850881    5002 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f864da04-4d22-42be-8334-d559fc8cdbe6-istio-data\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:06:30.851407 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:30.850894    5002 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f864da04-4d22-42be-8334-d559fc8cdbe6-credential-socket\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:06:30.851407 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:30.850907    5002 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f864da04-4d22-42be-8334-d559fc8cdbe6-istiod-ca-cert\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:06:30.851407 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:30.850921    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dlrs9\" (UniqueName: \"kubernetes.io/projected/f864da04-4d22-42be-8334-d559fc8cdbe6-kube-api-access-dlrs9\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:06:31.063913 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:31.063867    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-5007e1-openshift-default-64c644d8cc-wrs68"]
Oct 30 01:06:31.069987 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:31.069952    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-5007e1-openshift-default-64c644d8cc-wrs68"]
Oct 30 01:06:32.496831 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:32.496799    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f864da04-4d22-42be-8334-d559fc8cdbe6" path="/var/lib/kubelet/pods/f864da04-4d22-42be-8334-d559fc8cdbe6/volumes"
Oct 30 01:06:34.176529 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:34.168987    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7d8z4"]
Oct 30 01:06:41.634572 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:41.634537    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b1f6ed-682td"]
Oct 30 01:06:41.648342 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:41.648300    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b1f6ed-682td"]
Oct 30 01:06:42.496644 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:42.496607    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="285b4ad9-1165-475d-953e-838e3fe08fad" path="/var/lib/kubelet/pods/285b4ad9-1165-475d-953e-838e3fe08fad/volumes"
Oct 30 01:06:42.868203 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:06:42.868111    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbjdq" containerName="extractor"
Oct 30 01:06:44.919122 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:44.919093    5002 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-g7c2w_5dc56e7e-f3c1-43eb-a21b-8be6bb4706cf/console-operator/1.log"
Oct 30 01:06:44.924109 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:44.924085    5002 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-g7c2w_5dc56e7e-f3c1-43eb-a21b-8be6bb4706cf/console-operator/1.log"
Oct 30 01:06:47.592212 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:47.592171    5002 scope.go:122] "RemoveContainer" containerID="20aa1961d48dcad93eb5f84e2cda6e191ad8e191d5bf163c93253bd5888f44e2"
Oct 30 01:06:48.746138 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:48.743173    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-b1f6ed-openshift-default-767786fdd5-5gvj4"]
Oct 30 01:06:48.747627 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:48.747355    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-b1f6ed-openshift-default-767786fdd5-5gvj4" podUID="4a0578e8-4726-4c93-9c21-d09fac94c229" containerName="istio-proxy" containerID="cri-o://1adb422bfd667770bfc95c1be46853f10ec8122031f8b9cf7ab8ae826c1a740b" gracePeriod=30
Oct 30 01:06:54.059492 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:54.059461    5002 generic.go:356] "Generic (PLEG): container finished" podID="4a0578e8-4726-4c93-9c21-d09fac94c229" containerID="1adb422bfd667770bfc95c1be46853f10ec8122031f8b9cf7ab8ae826c1a740b" exitCode=0
Oct 30 01:06:54.059848 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:54.059511    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b1f6ed-openshift-default-767786fdd5-5gvj4" event={"ID":"4a0578e8-4726-4c93-9c21-d09fac94c229","Type":"ContainerDied","Data":"1adb422bfd667770bfc95c1be46853f10ec8122031f8b9cf7ab8ae826c1a740b"}
Oct 30 01:06:54.059848 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:54.059557    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b1f6ed-openshift-default-767786fdd5-5gvj4" event={"ID":"4a0578e8-4726-4c93-9c21-d09fac94c229","Type":"ContainerDied","Data":"66a100254b44781aee4d9c1537692fcbe49ebdb31d9338f4166016bfb0432e2b"}
Oct 30 01:06:54.059848 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:54.059575    5002 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="66a100254b44781aee4d9c1537692fcbe49ebdb31d9338f4166016bfb0432e2b"
Oct 30 01:06:54.059848 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:54.059590    5002 scope.go:122] "RemoveContainer" containerID="3ffeb3906c89d012b7dd3d63ff485e9872aa3b6d2cbb6723d33e9b567a928133"
Oct 30 01:06:54.061549 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:54.061526    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b1f6ed-openshift-default-767786fdd5-5gvj4"
Oct 30 01:06:54.152436 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:54.148950    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a0578e8-4726-4c93-9c21-d09fac94c229-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a0578e8-4726-4c93-9c21-d09fac94c229-istiod-ca-cert\") pod \"4a0578e8-4726-4c93-9c21-d09fac94c229\" (UID: \"4a0578e8-4726-4c93-9c21-d09fac94c229\") "
Oct 30 01:06:54.152436 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:54.149107    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a0578e8-4726-4c93-9c21-d09fac94c229-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a0578e8-4726-4c93-9c21-d09fac94c229-workload-socket\") pod \"4a0578e8-4726-4c93-9c21-d09fac94c229\" (UID: \"4a0578e8-4726-4c93-9c21-d09fac94c229\") "
Oct 30 01:06:54.152436 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:54.149340    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4a0578e8-4726-4c93-9c21-d09fac94c229-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a0578e8-4726-4c93-9c21-d09fac94c229-istio-podinfo\") pod \"4a0578e8-4726-4c93-9c21-d09fac94c229\" (UID: \"4a0578e8-4726-4c93-9c21-d09fac94c229\") "
Oct 30 01:06:54.152436 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:54.149430    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a0578e8-4726-4c93-9c21-d09fac94c229-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a0578e8-4726-4c93-9c21-d09fac94c229-credential-socket\") pod \"4a0578e8-4726-4c93-9c21-d09fac94c229\" (UID: \"4a0578e8-4726-4c93-9c21-d09fac94c229\") "
Oct 30 01:06:54.152436 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:54.149627    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a0578e8-4726-4c93-9c21-d09fac94c229-istio-token\" (UniqueName: \"kubernetes.io/projected/4a0578e8-4726-4c93-9c21-d09fac94c229-istio-token\") pod \"4a0578e8-4726-4c93-9c21-d09fac94c229\" (UID: \"4a0578e8-4726-4c93-9c21-d09fac94c229\") "
Oct 30 01:06:54.152436 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:54.150799    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a0578e8-4726-4c93-9c21-d09fac94c229-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a0578e8-4726-4c93-9c21-d09fac94c229-istio-envoy\") pod \"4a0578e8-4726-4c93-9c21-d09fac94c229\" (UID: \"4a0578e8-4726-4c93-9c21-d09fac94c229\") "
Oct 30 01:06:54.152436 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:54.150901    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a0578e8-4726-4c93-9c21-d09fac94c229-kube-api-access-9ql2v\" (UniqueName: \"kubernetes.io/projected/4a0578e8-4726-4c93-9c21-d09fac94c229-kube-api-access-9ql2v\") pod \"4a0578e8-4726-4c93-9c21-d09fac94c229\" (UID: \"4a0578e8-4726-4c93-9c21-d09fac94c229\") "
Oct 30 01:06:54.152436 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:54.150984    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a0578e8-4726-4c93-9c21-d09fac94c229-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a0578e8-4726-4c93-9c21-d09fac94c229-workload-certs\") pod \"4a0578e8-4726-4c93-9c21-d09fac94c229\" (UID: \"4a0578e8-4726-4c93-9c21-d09fac94c229\") "
Oct 30 01:06:54.152436 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:54.151070    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a0578e8-4726-4c93-9c21-d09fac94c229-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a0578e8-4726-4c93-9c21-d09fac94c229-istio-data\") pod \"4a0578e8-4726-4c93-9c21-d09fac94c229\" (UID: \"4a0578e8-4726-4c93-9c21-d09fac94c229\") "
Oct 30 01:06:54.152436 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:54.152132    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a0578e8-4726-4c93-9c21-d09fac94c229-istio-data" pod "4a0578e8-4726-4c93-9c21-d09fac94c229" (UID: "4a0578e8-4726-4c93-9c21-d09fac94c229"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:06:54.154341 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:54.153264    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a0578e8-4726-4c93-9c21-d09fac94c229-istiod-ca-cert" pod "4a0578e8-4726-4c93-9c21-d09fac94c229" (UID: "4a0578e8-4726-4c93-9c21-d09fac94c229"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:06:54.154341 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:54.153721    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a0578e8-4726-4c93-9c21-d09fac94c229-workload-socket" pod "4a0578e8-4726-4c93-9c21-d09fac94c229" (UID: "4a0578e8-4726-4c93-9c21-d09fac94c229"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:06:54.156867 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:54.156824    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a0578e8-4726-4c93-9c21-d09fac94c229-workload-certs" pod "4a0578e8-4726-4c93-9c21-d09fac94c229" (UID: "4a0578e8-4726-4c93-9c21-d09fac94c229"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:06:54.157461 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:54.157433    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a0578e8-4726-4c93-9c21-d09fac94c229-credential-socket" pod "4a0578e8-4726-4c93-9c21-d09fac94c229" (UID: "4a0578e8-4726-4c93-9c21-d09fac94c229"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:06:54.159228 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:54.159198    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a0578e8-4726-4c93-9c21-d09fac94c229-kube-api-access-9ql2v" pod "4a0578e8-4726-4c93-9c21-d09fac94c229" (UID: "4a0578e8-4726-4c93-9c21-d09fac94c229"). InnerVolumeSpecName "kube-api-access-9ql2v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:06:54.168353 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:54.168319    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a0578e8-4726-4c93-9c21-d09fac94c229-istio-envoy" pod "4a0578e8-4726-4c93-9c21-d09fac94c229" (UID: "4a0578e8-4726-4c93-9c21-d09fac94c229"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:06:54.168654 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:54.168623    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a0578e8-4726-4c93-9c21-d09fac94c229-istio-token" pod "4a0578e8-4726-4c93-9c21-d09fac94c229" (UID: "4a0578e8-4726-4c93-9c21-d09fac94c229"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:06:54.172166 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:54.172135    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4a0578e8-4726-4c93-9c21-d09fac94c229-istio-podinfo" pod "4a0578e8-4726-4c93-9c21-d09fac94c229" (UID: "4a0578e8-4726-4c93-9c21-d09fac94c229"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:06:54.253289 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:54.253244    5002 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a0578e8-4726-4c93-9c21-d09fac94c229-istiod-ca-cert\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:06:54.253289 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:54.253287    5002 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a0578e8-4726-4c93-9c21-d09fac94c229-workload-socket\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:06:54.253574 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:54.253301    5002 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a0578e8-4726-4c93-9c21-d09fac94c229-istio-podinfo\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:06:54.253574 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:54.253316    5002 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a0578e8-4726-4c93-9c21-d09fac94c229-credential-socket\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:06:54.253574 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:54.253330    5002 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a0578e8-4726-4c93-9c21-d09fac94c229-istio-token\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:06:54.253574 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:54.253343    5002 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a0578e8-4726-4c93-9c21-d09fac94c229-istio-envoy\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:06:54.253574 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:54.253357    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9ql2v\" (UniqueName: \"kubernetes.io/projected/4a0578e8-4726-4c93-9c21-d09fac94c229-kube-api-access-9ql2v\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:06:54.253574 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:54.253372    5002 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a0578e8-4726-4c93-9c21-d09fac94c229-workload-certs\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:06:54.253574 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:54.253401    5002 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a0578e8-4726-4c93-9c21-d09fac94c229-istio-data\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:06:55.070169 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:55.070132    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b1f6ed-openshift-default-767786fdd5-5gvj4"
Oct 30 01:06:55.177342 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:55.177310    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-b1f6ed-openshift-default-767786fdd5-5gvj4"]
Oct 30 01:06:55.181863 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:55.181832    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-b1f6ed-openshift-default-767786fdd5-5gvj4"]
Oct 30 01:06:56.508930 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:56.508843    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a0578e8-4726-4c93-9c21-d09fac94c229" path="/var/lib/kubelet/pods/4a0578e8-4726-4c93-9c21-d09fac94c229/volumes"
Oct 30 01:06:59.984170 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:59.984135    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf59ea86-db8wc"]
Oct 30 01:06:59.984820 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:59.984803    5002 cpu_manager.go:414] "containerMap: removing container" podUID="f864da04-4d22-42be-8334-d559fc8cdbe6" containerName="istio-proxy"
Oct 30 01:06:59.984875 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:59.984821    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f864da04-4d22-42be-8334-d559fc8cdbe6" containerName="istio-proxy"
Oct 30 01:06:59.984875 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:59.984835    5002 cpu_manager.go:414] "containerMap: removing container" podUID="4a0578e8-4726-4c93-9c21-d09fac94c229" containerName="istio-proxy"
Oct 30 01:06:59.984875 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:59.984843    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a0578e8-4726-4c93-9c21-d09fac94c229" containerName="istio-proxy"
Oct 30 01:06:59.984875 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:59.984866    5002 cpu_manager.go:414] "containerMap: removing container" podUID="2bb71dc1-0795-464e-9887-a6b5ce483890" containerName="istio-proxy"
Oct 30 01:06:59.984875 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:59.984873    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2bb71dc1-0795-464e-9887-a6b5ce483890" containerName="istio-proxy"
Oct 30 01:06:59.985031 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:59.984886    5002 cpu_manager.go:414] "containerMap: removing container" podUID="2bb71dc1-0795-464e-9887-a6b5ce483890" containerName="istio-proxy"
Oct 30 01:06:59.985031 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:59.984896    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2bb71dc1-0795-464e-9887-a6b5ce483890" containerName="istio-proxy"
Oct 30 01:06:59.985031 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:59.984913    5002 cpu_manager.go:414] "containerMap: removing container" podUID="4a0578e8-4726-4c93-9c21-d09fac94c229" containerName="istio-proxy"
Oct 30 01:06:59.985031 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:59.984917    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a0578e8-4726-4c93-9c21-d09fac94c229" containerName="istio-proxy"
Oct 30 01:06:59.985031 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:59.984933    5002 cpu_manager.go:414] "containerMap: removing container" podUID="fa5f1440-5fa9-40e6-b85f-073f31d24d01" containerName="payload-pre-processing"
Oct 30 01:06:59.985031 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:59.984937    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa5f1440-5fa9-40e6-b85f-073f31d24d01" containerName="payload-pre-processing"
Oct 30 01:06:59.985031 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:59.984957    5002 cpu_manager.go:414] "containerMap: removing container" podUID="285b4ad9-1165-475d-953e-838e3fe08fad" containerName="revoke-keys"
Oct 30 01:06:59.985031 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:59.984962    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="285b4ad9-1165-475d-953e-838e3fe08fad" containerName="revoke-keys"
Oct 30 01:06:59.985274 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:59.985071    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="2bb71dc1-0795-464e-9887-a6b5ce483890" containerName="istio-proxy"
Oct 30 01:06:59.985274 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:59.985082    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a0578e8-4726-4c93-9c21-d09fac94c229" containerName="istio-proxy"
Oct 30 01:06:59.985274 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:59.985090    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa5f1440-5fa9-40e6-b85f-073f31d24d01" containerName="payload-pre-processing"
Oct 30 01:06:59.985274 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:59.985098    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="285b4ad9-1165-475d-953e-838e3fe08fad" containerName="revoke-keys"
Oct 30 01:06:59.985274 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:59.985106    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="f864da04-4d22-42be-8334-d559fc8cdbe6" containerName="istio-proxy"
Oct 30 01:06:59.985274 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:59.985111    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a0578e8-4726-4c93-9c21-d09fac94c229" containerName="istio-proxy"
Oct 30 01:06:59.990433 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:06:59.990410    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf59ea86-db8wc"
Oct 30 01:07:00.007438 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:00.007409    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf59ea86-db8wc"]
Oct 30 01:07:00.016829 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:00.016797    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:07:00.017728 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:00.017084    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:07:00.017728 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:00.017189    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:07:00.020815 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:00.020789    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5phsh\""
Oct 30 01:07:00.081998 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:00.081961    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zw9bb\" (UniqueName: \"kubernetes.io/projected/12301847-7ec2-4d14-87c9-b0a9743b8e2d-kube-api-access-zw9bb\") pod \"maas-api-revoke-sub-cf59ea86-db8wc\" (UID: \"12301847-7ec2-4d14-87c9-b0a9743b8e2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf59ea86-db8wc"
Oct 30 01:07:00.082182 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:00.082119    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12301847-7ec2-4d14-87c9-b0a9743b8e2d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cf59ea86-db8wc\" (UID: \"12301847-7ec2-4d14-87c9-b0a9743b8e2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf59ea86-db8wc"
Oct 30 01:07:00.082253 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:00.082181    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12301847-7ec2-4d14-87c9-b0a9743b8e2d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cf59ea86-db8wc\" (UID: \"12301847-7ec2-4d14-87c9-b0a9743b8e2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf59ea86-db8wc"
Oct 30 01:07:00.184068 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:00.184027    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zw9bb\" (UniqueName: \"kubernetes.io/projected/12301847-7ec2-4d14-87c9-b0a9743b8e2d-kube-api-access-zw9bb\") pod \"maas-api-revoke-sub-cf59ea86-db8wc\" (UID: \"12301847-7ec2-4d14-87c9-b0a9743b8e2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf59ea86-db8wc"
Oct 30 01:07:00.184274 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:00.184160    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12301847-7ec2-4d14-87c9-b0a9743b8e2d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cf59ea86-db8wc\" (UID: \"12301847-7ec2-4d14-87c9-b0a9743b8e2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf59ea86-db8wc"
Oct 30 01:07:00.184274 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:00.184231    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12301847-7ec2-4d14-87c9-b0a9743b8e2d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cf59ea86-db8wc\" (UID: \"12301847-7ec2-4d14-87c9-b0a9743b8e2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf59ea86-db8wc"
Oct 30 01:07:00.185018 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:00.184995    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12301847-7ec2-4d14-87c9-b0a9743b8e2d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cf59ea86-db8wc\" (UID: \"12301847-7ec2-4d14-87c9-b0a9743b8e2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf59ea86-db8wc"
Oct 30 01:07:00.186633 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:00.186610    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12301847-7ec2-4d14-87c9-b0a9743b8e2d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cf59ea86-db8wc\" (UID: \"12301847-7ec2-4d14-87c9-b0a9743b8e2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf59ea86-db8wc"
Oct 30 01:07:00.209275 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:00.209242    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zw9bb\" (UniqueName: \"kubernetes.io/projected/12301847-7ec2-4d14-87c9-b0a9743b8e2d-kube-api-access-zw9bb\") pod \"maas-api-revoke-sub-cf59ea86-db8wc\" (UID: \"12301847-7ec2-4d14-87c9-b0a9743b8e2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf59ea86-db8wc"
Oct 30 01:07:00.304500 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:00.304413    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf59ea86-db8wc"
Oct 30 01:07:00.626106 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:00.626054    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf59ea86-db8wc"]
Oct 30 01:07:00.626261 ip-10-0-128-7 kubenswrapper[5002]: W1030 01:07:00.626106    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12301847_7ec2_4d14_87c9_b0a9743b8e2d.slice/crio-1d56d937e991a14384e7834c8bf964d765834921432e05b61c4a65e0fd0f7735 WatchSource:0}: Error finding container 1d56d937e991a14384e7834c8bf964d765834921432e05b61c4a65e0fd0f7735: Status 404 returned error can't find the container with id 1d56d937e991a14384e7834c8bf964d765834921432e05b61c4a65e0fd0f7735
Oct 30 01:07:01.161107 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:01.161062    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf59ea86-db8wc" event={"ID":"12301847-7ec2-4d14-87c9-b0a9743b8e2d","Type":"ContainerStarted","Data":"206d5e7265fe257b14ecab6dd2193303682aae5bcda4e1d225a5101eec1250d4"}
Oct 30 01:07:01.161615 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:01.161127    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf59ea86-db8wc" event={"ID":"12301847-7ec2-4d14-87c9-b0a9743b8e2d","Type":"ContainerStarted","Data":"1d56d937e991a14384e7834c8bf964d765834921432e05b61c4a65e0fd0f7735"}
Oct 30 01:07:02.175008 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:02.174912    5002 generic.go:356] "Generic (PLEG): container finished" podID="12301847-7ec2-4d14-87c9-b0a9743b8e2d" containerID="206d5e7265fe257b14ecab6dd2193303682aae5bcda4e1d225a5101eec1250d4" exitCode=0
Oct 30 01:07:02.175008 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:02.174990    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf59ea86-db8wc" event={"ID":"12301847-7ec2-4d14-87c9-b0a9743b8e2d","Type":"ContainerDied","Data":"206d5e7265fe257b14ecab6dd2193303682aae5bcda4e1d225a5101eec1250d4"}
Oct 30 01:07:03.398734 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:03.398700    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf59ea86-db8wc"
Oct 30 01:07:03.443307 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:03.443035    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12301847-7ec2-4d14-87c9-b0a9743b8e2d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12301847-7ec2-4d14-87c9-b0a9743b8e2d-maas-api-openshift-ca\") pod \"12301847-7ec2-4d14-87c9-b0a9743b8e2d\" (UID: \"12301847-7ec2-4d14-87c9-b0a9743b8e2d\") "
Oct 30 01:07:03.443307 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:03.443224    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12301847-7ec2-4d14-87c9-b0a9743b8e2d-kube-api-access-zw9bb\" (UniqueName: \"kubernetes.io/projected/12301847-7ec2-4d14-87c9-b0a9743b8e2d-kube-api-access-zw9bb\") pod \"12301847-7ec2-4d14-87c9-b0a9743b8e2d\" (UID: \"12301847-7ec2-4d14-87c9-b0a9743b8e2d\") "
Oct 30 01:07:03.443307 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:03.443424    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/12301847-7ec2-4d14-87c9-b0a9743b8e2d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12301847-7ec2-4d14-87c9-b0a9743b8e2d-maas-api-xks-ca\") pod \"12301847-7ec2-4d14-87c9-b0a9743b8e2d\" (UID: \"12301847-7ec2-4d14-87c9-b0a9743b8e2d\") "
Oct 30 01:07:03.443833 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:03.443467    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12301847-7ec2-4d14-87c9-b0a9743b8e2d-maas-api-openshift-ca" pod "12301847-7ec2-4d14-87c9-b0a9743b8e2d" (UID: "12301847-7ec2-4d14-87c9-b0a9743b8e2d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:07:03.444080 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:03.444051    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12301847-7ec2-4d14-87c9-b0a9743b8e2d-maas-api-openshift-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:07:03.446190 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:03.446167    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12301847-7ec2-4d14-87c9-b0a9743b8e2d-maas-api-xks-ca" pod "12301847-7ec2-4d14-87c9-b0a9743b8e2d" (UID: "12301847-7ec2-4d14-87c9-b0a9743b8e2d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:07:03.446301 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:03.446204    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12301847-7ec2-4d14-87c9-b0a9743b8e2d-kube-api-access-zw9bb" pod "12301847-7ec2-4d14-87c9-b0a9743b8e2d" (UID: "12301847-7ec2-4d14-87c9-b0a9743b8e2d"). InnerVolumeSpecName "kube-api-access-zw9bb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:07:03.545993 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:03.545912    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zw9bb\" (UniqueName: \"kubernetes.io/projected/12301847-7ec2-4d14-87c9-b0a9743b8e2d-kube-api-access-zw9bb\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:07:03.545993 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:03.545946    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12301847-7ec2-4d14-87c9-b0a9743b8e2d-maas-api-xks-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:07:04.221888 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:04.221851    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf59ea86-db8wc" event={"ID":"12301847-7ec2-4d14-87c9-b0a9743b8e2d","Type":"ContainerDied","Data":"1d56d937e991a14384e7834c8bf964d765834921432e05b61c4a65e0fd0f7735"}
Oct 30 01:07:04.222099 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:04.221909    5002 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1d56d937e991a14384e7834c8bf964d765834921432e05b61c4a65e0fd0f7735"
Oct 30 01:07:04.222099 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:04.221867    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf59ea86-db8wc"
Oct 30 01:07:07.066417 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:07.066348    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-11f521-openshift-default-96689b67c-p9ptg"]
Oct 30 01:07:07.067319 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:07.067295    5002 cpu_manager.go:414] "containerMap: removing container" podUID="12301847-7ec2-4d14-87c9-b0a9743b8e2d" containerName="revoke-keys"
Oct 30 01:07:07.067417 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:07.067324    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12301847-7ec2-4d14-87c9-b0a9743b8e2d" containerName="revoke-keys"
Oct 30 01:07:07.067566 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:07.067552    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="12301847-7ec2-4d14-87c9-b0a9743b8e2d" containerName="revoke-keys"
Oct 30 01:07:07.067604 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:07.067575    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="2bb71dc1-0795-464e-9887-a6b5ce483890" containerName="istio-proxy"
Oct 30 01:07:07.072280 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:07.072254    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-11f521-openshift-default-96689b67c-p9ptg"
Oct 30 01:07:07.095596 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:07.095562    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-11f521-openshift-default-96689b67c-p9ptg"]
Oct 30 01:07:07.098627 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:07.098594    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/412d8cc8-d5e9-4d37-9021-baeda679711a-istiod-ca-cert\") pod \"e2e-shared-a-w3-11f521-openshift-default-96689b67c-p9ptg\" (UID: \"412d8cc8-d5e9-4d37-9021-baeda679711a\") " pod="openshift-ingress/e2e-shared-a-w3-11f521-openshift-default-96689b67c-p9ptg"
Oct 30 01:07:07.098786 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:07.098731    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/412d8cc8-d5e9-4d37-9021-baeda679711a-workload-certs\") pod \"e2e-shared-a-w3-11f521-openshift-default-96689b67c-p9ptg\" (UID: \"412d8cc8-d5e9-4d37-9021-baeda679711a\") " pod="openshift-ingress/e2e-shared-a-w3-11f521-openshift-default-96689b67c-p9ptg"
Oct 30 01:07:07.098924 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:07.098905    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/412d8cc8-d5e9-4d37-9021-baeda679711a-istio-data\") pod \"e2e-shared-a-w3-11f521-openshift-default-96689b67c-p9ptg\" (UID: \"412d8cc8-d5e9-4d37-9021-baeda679711a\") " pod="openshift-ingress/e2e-shared-a-w3-11f521-openshift-default-96689b67c-p9ptg"
Oct 30 01:07:07.099029 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:07.099004    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/412d8cc8-d5e9-4d37-9021-baeda679711a-istio-token\") pod \"e2e-shared-a-w3-11f521-openshift-default-96689b67c-p9ptg\" (UID: \"412d8cc8-d5e9-4d37-9021-baeda679711a\") " pod="openshift-ingress/e2e-shared-a-w3-11f521-openshift-default-96689b67c-p9ptg"
Oct 30 01:07:07.099158 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:07.099069    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfmfx\" (UniqueName: \"kubernetes.io/projected/412d8cc8-d5e9-4d37-9021-baeda679711a-kube-api-access-hfmfx\") pod \"e2e-shared-a-w3-11f521-openshift-default-96689b67c-p9ptg\" (UID: \"412d8cc8-d5e9-4d37-9021-baeda679711a\") " pod="openshift-ingress/e2e-shared-a-w3-11f521-openshift-default-96689b67c-p9ptg"
Oct 30 01:07:07.099158 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:07.099145    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/412d8cc8-d5e9-4d37-9021-baeda679711a-credential-socket\") pod \"e2e-shared-a-w3-11f521-openshift-default-96689b67c-p9ptg\" (UID: \"412d8cc8-d5e9-4d37-9021-baeda679711a\") " pod="openshift-ingress/e2e-shared-a-w3-11f521-openshift-default-96689b67c-p9ptg"
Oct 30 01:07:07.099287 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:07.099201    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/412d8cc8-d5e9-4d37-9021-baeda679711a-workload-socket\") pod \"e2e-shared-a-w3-11f521-openshift-default-96689b67c-p9ptg\" (UID: \"412d8cc8-d5e9-4d37-9021-baeda679711a\") " pod="openshift-ingress/e2e-shared-a-w3-11f521-openshift-default-96689b67c-p9ptg"
Oct 30 01:07:07.099287 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:07.099230    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/412d8cc8-d5e9-4d37-9021-baeda679711a-istio-envoy\") pod \"e2e-shared-a-w3-11f521-openshift-default-96689b67c-p9ptg\" (UID: \"412d8cc8-d5e9-4d37-9021-baeda679711a\") " pod="openshift-ingress/e2e-shared-a-w3-11f521-openshift-default-96689b67c-p9ptg"
Oct 30 01:07:07.099441 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:07.099409    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/412d8cc8-d5e9-4d37-9021-baeda679711a-istio-podinfo\") pod \"e2e-shared-a-w3-11f521-openshift-default-96689b67c-p9ptg\" (UID: \"412d8cc8-d5e9-4d37-9021-baeda679711a\") " pod="openshift-ingress/e2e-shared-a-w3-11f521-openshift-default-96689b67c-p9ptg"
Oct 30 01:07:07.104171 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:07.104147    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-11f521-openshift-default-dockercfg-vkflh\""
Oct 30 01:07:07.200700 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:07.200663    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/412d8cc8-d5e9-4d37-9021-baeda679711a-istio-token\") pod \"e2e-shared-a-w3-11f521-openshift-default-96689b67c-p9ptg\" (UID: \"412d8cc8-d5e9-4d37-9021-baeda679711a\") " pod="openshift-ingress/e2e-shared-a-w3-11f521-openshift-default-96689b67c-p9ptg"
Oct 30 01:07:07.200881 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:07.200718    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfmfx\" (UniqueName: \"kubernetes.io/projected/412d8cc8-d5e9-4d37-9021-baeda679711a-kube-api-access-hfmfx\") pod \"e2e-shared-a-w3-11f521-openshift-default-96689b67c-p9ptg\" (UID: \"412d8cc8-d5e9-4d37-9021-baeda679711a\") " pod="openshift-ingress/e2e-shared-a-w3-11f521-openshift-default-96689b67c-p9ptg"
Oct 30 01:07:07.200881 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:07.200740    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/412d8cc8-d5e9-4d37-9021-baeda679711a-credential-socket\") pod \"e2e-shared-a-w3-11f521-openshift-default-96689b67c-p9ptg\" (UID: \"412d8cc8-d5e9-4d37-9021-baeda679711a\") " pod="openshift-ingress/e2e-shared-a-w3-11f521-openshift-default-96689b67c-p9ptg"
Oct 30 01:07:07.200881 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:07.200769    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/412d8cc8-d5e9-4d37-9021-baeda679711a-workload-socket\") pod \"e2e-shared-a-w3-11f521-openshift-default-96689b67c-p9ptg\" (UID: \"412d8cc8-d5e9-4d37-9021-baeda679711a\") " pod="openshift-ingress/e2e-shared-a-w3-11f521-openshift-default-96689b67c-p9ptg"
Oct 30 01:07:07.200881 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:07.200789    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/412d8cc8-d5e9-4d37-9021-baeda679711a-istio-envoy\") pod \"e2e-shared-a-w3-11f521-openshift-default-96689b67c-p9ptg\" (UID: \"412d8cc8-d5e9-4d37-9021-baeda679711a\") " pod="openshift-ingress/e2e-shared-a-w3-11f521-openshift-default-96689b67c-p9ptg"
Oct 30 01:07:07.200881 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:07.200859    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/412d8cc8-d5e9-4d37-9021-baeda679711a-istio-podinfo\") pod \"e2e-shared-a-w3-11f521-openshift-default-96689b67c-p9ptg\" (UID: \"412d8cc8-d5e9-4d37-9021-baeda679711a\") " pod="openshift-ingress/e2e-shared-a-w3-11f521-openshift-default-96689b67c-p9ptg"
Oct 30 01:07:07.201179 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:07.200902    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/412d8cc8-d5e9-4d37-9021-baeda679711a-istiod-ca-cert\") pod \"e2e-shared-a-w3-11f521-openshift-default-96689b67c-p9ptg\" (UID: \"412d8cc8-d5e9-4d37-9021-baeda679711a\") " pod="openshift-ingress/e2e-shared-a-w3-11f521-openshift-default-96689b67c-p9ptg"
Oct 30 01:07:07.201179 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:07.200961    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/412d8cc8-d5e9-4d37-9021-baeda679711a-workload-certs\") pod \"e2e-shared-a-w3-11f521-openshift-default-96689b67c-p9ptg\" (UID: \"412d8cc8-d5e9-4d37-9021-baeda679711a\") " pod="openshift-ingress/e2e-shared-a-w3-11f521-openshift-default-96689b67c-p9ptg"
Oct 30 01:07:07.201179 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:07.201120    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/412d8cc8-d5e9-4d37-9021-baeda679711a-istio-data\") pod \"e2e-shared-a-w3-11f521-openshift-default-96689b67c-p9ptg\" (UID: \"412d8cc8-d5e9-4d37-9021-baeda679711a\") " pod="openshift-ingress/e2e-shared-a-w3-11f521-openshift-default-96689b67c-p9ptg"
Oct 30 01:07:07.201349 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:07.201199    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/412d8cc8-d5e9-4d37-9021-baeda679711a-credential-socket\") pod \"e2e-shared-a-w3-11f521-openshift-default-96689b67c-p9ptg\" (UID: \"412d8cc8-d5e9-4d37-9021-baeda679711a\") " pod="openshift-ingress/e2e-shared-a-w3-11f521-openshift-default-96689b67c-p9ptg"
Oct 30 01:07:07.201349 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:07.201236    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/412d8cc8-d5e9-4d37-9021-baeda679711a-workload-socket\") pod \"e2e-shared-a-w3-11f521-openshift-default-96689b67c-p9ptg\" (UID: \"412d8cc8-d5e9-4d37-9021-baeda679711a\") " pod="openshift-ingress/e2e-shared-a-w3-11f521-openshift-default-96689b67c-p9ptg"
Oct 30 01:07:07.201541 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:07.201514    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/412d8cc8-d5e9-4d37-9021-baeda679711a-workload-certs\") pod \"e2e-shared-a-w3-11f521-openshift-default-96689b67c-p9ptg\" (UID: \"412d8cc8-d5e9-4d37-9021-baeda679711a\") " pod="openshift-ingress/e2e-shared-a-w3-11f521-openshift-default-96689b67c-p9ptg"
Oct 30 01:07:07.201615 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:07.201553    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/412d8cc8-d5e9-4d37-9021-baeda679711a-istio-data\") pod \"e2e-shared-a-w3-11f521-openshift-default-96689b67c-p9ptg\" (UID: \"412d8cc8-d5e9-4d37-9021-baeda679711a\") " pod="openshift-ingress/e2e-shared-a-w3-11f521-openshift-default-96689b67c-p9ptg"
Oct 30 01:07:07.201682 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:07.201658    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/412d8cc8-d5e9-4d37-9021-baeda679711a-istiod-ca-cert\") pod \"e2e-shared-a-w3-11f521-openshift-default-96689b67c-p9ptg\" (UID: \"412d8cc8-d5e9-4d37-9021-baeda679711a\") " pod="openshift-ingress/e2e-shared-a-w3-11f521-openshift-default-96689b67c-p9ptg"
Oct 30 01:07:07.203362 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:07.203335    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/412d8cc8-d5e9-4d37-9021-baeda679711a-istio-envoy\") pod \"e2e-shared-a-w3-11f521-openshift-default-96689b67c-p9ptg\" (UID: \"412d8cc8-d5e9-4d37-9021-baeda679711a\") " pod="openshift-ingress/e2e-shared-a-w3-11f521-openshift-default-96689b67c-p9ptg"
Oct 30 01:07:07.203640 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:07.203620    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/412d8cc8-d5e9-4d37-9021-baeda679711a-istio-podinfo\") pod \"e2e-shared-a-w3-11f521-openshift-default-96689b67c-p9ptg\" (UID: \"412d8cc8-d5e9-4d37-9021-baeda679711a\") " pod="openshift-ingress/e2e-shared-a-w3-11f521-openshift-default-96689b67c-p9ptg"
Oct 30 01:07:07.225764 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:07.225732    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/412d8cc8-d5e9-4d37-9021-baeda679711a-istio-token\") pod \"e2e-shared-a-w3-11f521-openshift-default-96689b67c-p9ptg\" (UID: \"412d8cc8-d5e9-4d37-9021-baeda679711a\") " pod="openshift-ingress/e2e-shared-a-w3-11f521-openshift-default-96689b67c-p9ptg"
Oct 30 01:07:07.225897 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:07.225861    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfmfx\" (UniqueName: \"kubernetes.io/projected/412d8cc8-d5e9-4d37-9021-baeda679711a-kube-api-access-hfmfx\") pod \"e2e-shared-a-w3-11f521-openshift-default-96689b67c-p9ptg\" (UID: \"412d8cc8-d5e9-4d37-9021-baeda679711a\") " pod="openshift-ingress/e2e-shared-a-w3-11f521-openshift-default-96689b67c-p9ptg"
Oct 30 01:07:07.391151 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:07.391051    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-11f521-openshift-default-96689b67c-p9ptg"
Oct 30 01:07:07.734432 ip-10-0-128-7 kubenswrapper[5002]: W1030 01:07:07.734401    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod412d8cc8_d5e9_4d37_9021_baeda679711a.slice/crio-477cf24986e054a54c80fa25c9ad2692f18b6380aac2baf80c199f92caf53c0f WatchSource:0}: Error finding container 477cf24986e054a54c80fa25c9ad2692f18b6380aac2baf80c199f92caf53c0f: Status 404 returned error can't find the container with id 477cf24986e054a54c80fa25c9ad2692f18b6380aac2baf80c199f92caf53c0f
Oct 30 01:07:07.735901 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:07.735875    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-11f521-openshift-default-96689b67c-p9ptg"]
Oct 30 01:07:07.737477 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:07.737446    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:07:07.737546 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:07.737511    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:07:07.737546 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:07.737541    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:07:08.256691 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:08.256652    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-11f521-openshift-default-96689b67c-p9ptg" event={"ID":"412d8cc8-d5e9-4d37-9021-baeda679711a","Type":"ContainerStarted","Data":"de66c75782cb4c00455584cea03af158133e50b7e124f6f29231156fb0cc75a9"}
Oct 30 01:07:08.257195 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:08.256707    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-11f521-openshift-default-96689b67c-p9ptg" event={"ID":"412d8cc8-d5e9-4d37-9021-baeda679711a","Type":"ContainerStarted","Data":"477cf24986e054a54c80fa25c9ad2692f18b6380aac2baf80c199f92caf53c0f"}
Oct 30 01:07:08.319222 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:08.319170    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-11f521-openshift-default-96689b67c-p9ptg" podStartSLOduration=1.319157946 podStartE2EDuration="1.319157946s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:07:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:07:08.317970689 +0000 UTC m=+2123.934403586" watchObservedRunningTime="2026-10-30 01:07:08.319157946 +0000 UTC m=+2123.935590892"
Oct 30 01:07:08.414414 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:08.407923    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-11f521-openshift-default-96689b67c-p9ptg"
Oct 30 01:07:08.415565 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:08.415520    5002 patch_prober.go:29] interesting pod/e2e-shared-a-w3-11f521-openshift-default-96689b67c-p9ptg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body=
Oct 30 01:07:08.415765 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:08.415598    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-11f521-openshift-default-96689b67c-p9ptg" podUID="412d8cc8-d5e9-4d37-9021-baeda679711a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused"
Oct 30 01:07:09.391442 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:09.391373    5002 patch_prober.go:29] interesting pod/e2e-shared-a-w3-11f521-openshift-default-96689b67c-p9ptg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body=
Oct 30 01:07:09.391867 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:09.391502    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-11f521-openshift-default-96689b67c-p9ptg" podUID="412d8cc8-d5e9-4d37-9021-baeda679711a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused"
Oct 30 01:07:10.391982 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:10.391939    5002 patch_prober.go:29] interesting pod/e2e-shared-a-w3-11f521-openshift-default-96689b67c-p9ptg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body=
Oct 30 01:07:10.392426 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:10.392009    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-11f521-openshift-default-96689b67c-p9ptg" podUID="412d8cc8-d5e9-4d37-9021-baeda679711a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused"
Oct 30 01:07:11.391998 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:11.391949    5002 patch_prober.go:29] interesting pod/e2e-shared-a-w3-11f521-openshift-default-96689b67c-p9ptg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body=
Oct 30 01:07:11.392431 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:11.392032    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-11f521-openshift-default-96689b67c-p9ptg" podUID="412d8cc8-d5e9-4d37-9021-baeda679711a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused"
Oct 30 01:07:12.403809 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:12.403771    5002 patch_prober.go:29] interesting pod/e2e-shared-a-w3-11f521-openshift-default-96689b67c-p9ptg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body=
Oct 30 01:07:12.404413 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:12.404361    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-11f521-openshift-default-96689b67c-p9ptg" podUID="412d8cc8-d5e9-4d37-9021-baeda679711a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused"
Oct 30 01:07:12.897426 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:07:12.897372    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbjdq" containerName="extractor"
Oct 30 01:07:13.392239 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:13.392196    5002 patch_prober.go:29] interesting pod/e2e-shared-a-w3-11f521-openshift-default-96689b67c-p9ptg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body=
Oct 30 01:07:13.392509 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:13.392258    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-11f521-openshift-default-96689b67c-p9ptg" podUID="412d8cc8-d5e9-4d37-9021-baeda679711a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused"
Oct 30 01:07:14.393666 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:14.393627    5002 patch_prober.go:29] interesting pod/e2e-shared-a-w3-11f521-openshift-default-96689b67c-p9ptg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body=
Oct 30 01:07:14.394258 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:14.394226    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-11f521-openshift-default-96689b67c-p9ptg" podUID="412d8cc8-d5e9-4d37-9021-baeda679711a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused"
Oct 30 01:07:15.397420 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:15.397363    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-11f521-openshift-default-96689b67c-p9ptg"
Oct 30 01:07:15.397884 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:15.397860    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-11f521-openshift-default-96689b67c-p9ptg"
Oct 30 01:07:15.398643 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:15.398618    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-11f521-openshift-default-96689b67c-p9ptg"
Oct 30 01:07:21.763962 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:21.763917    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7d8z4"]
Oct 30 01:07:31.819062 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:31.819022    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-rnd66"]
Oct 30 01:07:31.823951 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:31.823926    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-rnd66"
Oct 30 01:07:31.857637 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:31.857591    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w6-7f92f6-openshift-default-dockercfg-n4bqm\""
Oct 30 01:07:31.865572 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:31.865544    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-rnd66"]
Oct 30 01:07:31.898615 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:31.898570    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4df0b3e0-04ca-440d-9b4c-01635ee412e7-istio-data\") pod \"e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-rnd66\" (UID: \"4df0b3e0-04ca-440d-9b4c-01635ee412e7\") " pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-rnd66"
Oct 30 01:07:31.898805 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:31.898646    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwd2v\" (UniqueName: \"kubernetes.io/projected/4df0b3e0-04ca-440d-9b4c-01635ee412e7-kube-api-access-dwd2v\") pod \"e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-rnd66\" (UID: \"4df0b3e0-04ca-440d-9b4c-01635ee412e7\") " pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-rnd66"
Oct 30 01:07:31.898805 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:31.898773    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4df0b3e0-04ca-440d-9b4c-01635ee412e7-istio-token\") pod \"e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-rnd66\" (UID: \"4df0b3e0-04ca-440d-9b4c-01635ee412e7\") " pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-rnd66"
Oct 30 01:07:31.898923 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:31.898819    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4df0b3e0-04ca-440d-9b4c-01635ee412e7-workload-certs\") pod \"e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-rnd66\" (UID: \"4df0b3e0-04ca-440d-9b4c-01635ee412e7\") " pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-rnd66"
Oct 30 01:07:31.898981 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:31.898945    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4df0b3e0-04ca-440d-9b4c-01635ee412e7-workload-socket\") pod \"e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-rnd66\" (UID: \"4df0b3e0-04ca-440d-9b4c-01635ee412e7\") " pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-rnd66"
Oct 30 01:07:31.899042 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:31.898991    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4df0b3e0-04ca-440d-9b4c-01635ee412e7-credential-socket\") pod \"e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-rnd66\" (UID: \"4df0b3e0-04ca-440d-9b4c-01635ee412e7\") " pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-rnd66"
Oct 30 01:07:31.899121 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:31.899106    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4df0b3e0-04ca-440d-9b4c-01635ee412e7-istio-podinfo\") pod \"e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-rnd66\" (UID: \"4df0b3e0-04ca-440d-9b4c-01635ee412e7\") " pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-rnd66"
Oct 30 01:07:31.899183 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:31.899128    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4df0b3e0-04ca-440d-9b4c-01635ee412e7-istiod-ca-cert\") pod \"e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-rnd66\" (UID: \"4df0b3e0-04ca-440d-9b4c-01635ee412e7\") " pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-rnd66"
Oct 30 01:07:31.899183 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:31.899159    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4df0b3e0-04ca-440d-9b4c-01635ee412e7-istio-envoy\") pod \"e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-rnd66\" (UID: \"4df0b3e0-04ca-440d-9b4c-01635ee412e7\") " pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-rnd66"
Oct 30 01:07:32.000501 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:32.000460    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4df0b3e0-04ca-440d-9b4c-01635ee412e7-istio-podinfo\") pod \"e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-rnd66\" (UID: \"4df0b3e0-04ca-440d-9b4c-01635ee412e7\") " pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-rnd66"
Oct 30 01:07:32.000697 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:32.000509    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4df0b3e0-04ca-440d-9b4c-01635ee412e7-istiod-ca-cert\") pod \"e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-rnd66\" (UID: \"4df0b3e0-04ca-440d-9b4c-01635ee412e7\") " pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-rnd66"
Oct 30 01:07:32.000697 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:32.000562    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4df0b3e0-04ca-440d-9b4c-01635ee412e7-istio-envoy\") pod \"e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-rnd66\" (UID: \"4df0b3e0-04ca-440d-9b4c-01635ee412e7\") " pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-rnd66"
Oct 30 01:07:32.000814 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:32.000703    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4df0b3e0-04ca-440d-9b4c-01635ee412e7-istio-data\") pod \"e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-rnd66\" (UID: \"4df0b3e0-04ca-440d-9b4c-01635ee412e7\") " pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-rnd66"
Oct 30 01:07:32.000814 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:32.000758    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwd2v\" (UniqueName: \"kubernetes.io/projected/4df0b3e0-04ca-440d-9b4c-01635ee412e7-kube-api-access-dwd2v\") pod \"e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-rnd66\" (UID: \"4df0b3e0-04ca-440d-9b4c-01635ee412e7\") " pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-rnd66"
Oct 30 01:07:32.000921 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:32.000848    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4df0b3e0-04ca-440d-9b4c-01635ee412e7-istio-token\") pod \"e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-rnd66\" (UID: \"4df0b3e0-04ca-440d-9b4c-01635ee412e7\") " pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-rnd66"
Oct 30 01:07:32.000921 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:32.000890    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4df0b3e0-04ca-440d-9b4c-01635ee412e7-workload-certs\") pod \"e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-rnd66\" (UID: \"4df0b3e0-04ca-440d-9b4c-01635ee412e7\") " pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-rnd66"
Oct 30 01:07:32.001033 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:32.000946    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4df0b3e0-04ca-440d-9b4c-01635ee412e7-workload-socket\") pod \"e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-rnd66\" (UID: \"4df0b3e0-04ca-440d-9b4c-01635ee412e7\") " pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-rnd66"
Oct 30 01:07:32.001033 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:32.000985    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4df0b3e0-04ca-440d-9b4c-01635ee412e7-credential-socket\") pod \"e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-rnd66\" (UID: \"4df0b3e0-04ca-440d-9b4c-01635ee412e7\") " pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-rnd66"
Oct 30 01:07:32.001156 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:32.001097    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4df0b3e0-04ca-440d-9b4c-01635ee412e7-istio-data\") pod \"e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-rnd66\" (UID: \"4df0b3e0-04ca-440d-9b4c-01635ee412e7\") " pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-rnd66"
Oct 30 01:07:32.001422 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:32.001362    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4df0b3e0-04ca-440d-9b4c-01635ee412e7-credential-socket\") pod \"e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-rnd66\" (UID: \"4df0b3e0-04ca-440d-9b4c-01635ee412e7\") " pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-rnd66"
Oct 30 01:07:32.001615 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:32.001400    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4df0b3e0-04ca-440d-9b4c-01635ee412e7-workload-certs\") pod \"e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-rnd66\" (UID: \"4df0b3e0-04ca-440d-9b4c-01635ee412e7\") " pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-rnd66"
Oct 30 01:07:32.001615 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:32.001456    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4df0b3e0-04ca-440d-9b4c-01635ee412e7-istiod-ca-cert\") pod \"e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-rnd66\" (UID: \"4df0b3e0-04ca-440d-9b4c-01635ee412e7\") " pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-rnd66"
Oct 30 01:07:32.001615 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:32.001615    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4df0b3e0-04ca-440d-9b4c-01635ee412e7-workload-socket\") pod \"e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-rnd66\" (UID: \"4df0b3e0-04ca-440d-9b4c-01635ee412e7\") " pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-rnd66"
Oct 30 01:07:32.003042 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:32.003024    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4df0b3e0-04ca-440d-9b4c-01635ee412e7-istio-envoy\") pod \"e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-rnd66\" (UID: \"4df0b3e0-04ca-440d-9b4c-01635ee412e7\") " pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-rnd66"
Oct 30 01:07:32.003206 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:32.003187    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4df0b3e0-04ca-440d-9b4c-01635ee412e7-istio-podinfo\") pod \"e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-rnd66\" (UID: \"4df0b3e0-04ca-440d-9b4c-01635ee412e7\") " pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-rnd66"
Oct 30 01:07:32.025776 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:32.025750    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4df0b3e0-04ca-440d-9b4c-01635ee412e7-istio-token\") pod \"e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-rnd66\" (UID: \"4df0b3e0-04ca-440d-9b4c-01635ee412e7\") " pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-rnd66"
Oct 30 01:07:32.025981 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:32.025962    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwd2v\" (UniqueName: \"kubernetes.io/projected/4df0b3e0-04ca-440d-9b4c-01635ee412e7-kube-api-access-dwd2v\") pod \"e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-rnd66\" (UID: \"4df0b3e0-04ca-440d-9b4c-01635ee412e7\") " pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-rnd66"
Oct 30 01:07:32.140106 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:32.140007    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-rnd66"
Oct 30 01:07:32.473464 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:32.473069    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-rnd66"]
Oct 30 01:07:32.478517 ip-10-0-128-7 kubenswrapper[5002]: W1030 01:07:32.478456    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4df0b3e0_04ca_440d_9b4c_01635ee412e7.slice/crio-1236175ca7da2a4fb9c7c52c8492f25255f7cdb00ba88e07bbce87a2f5efea00 WatchSource:0}: Error finding container 1236175ca7da2a4fb9c7c52c8492f25255f7cdb00ba88e07bbce87a2f5efea00: Status 404 returned error can't find the container with id 1236175ca7da2a4fb9c7c52c8492f25255f7cdb00ba88e07bbce87a2f5efea00
Oct 30 01:07:32.482338 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:32.481202    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:07:32.482338 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:32.481295    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:07:32.482338 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:32.481337    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:07:32.506865 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:32.506835    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-rnd66" event={"ID":"4df0b3e0-04ca-440d-9b4c-01635ee412e7","Type":"ContainerStarted","Data":"1236175ca7da2a4fb9c7c52c8492f25255f7cdb00ba88e07bbce87a2f5efea00"}
Oct 30 01:07:33.516333 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:33.516288    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-rnd66" event={"ID":"4df0b3e0-04ca-440d-9b4c-01635ee412e7","Type":"ContainerStarted","Data":"26d3b1ec3ce483ad4c1644f36ce78ed7f2d22ecdfa2fbe11d3306267a482b6f7"}
Oct 30 01:07:33.566743 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:33.566673    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-rnd66" podStartSLOduration=2.566652339 podStartE2EDuration="2.566652339s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:07:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:07:33.564905514 +0000 UTC m=+2149.181338394" watchObservedRunningTime="2026-10-30 01:07:33.566652339 +0000 UTC m=+2149.183085219"
Oct 30 01:07:34.140200 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:34.140166    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-rnd66"
Oct 30 01:07:34.141906 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:34.141876    5002 patch_prober.go:29] interesting pod/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-rnd66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body=
Oct 30 01:07:34.142044 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:34.141934    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-rnd66" podUID="4df0b3e0-04ca-440d-9b4c-01635ee412e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused"
Oct 30 01:07:34.343219 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:34.343180    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-ef1084/test-model-ef1084-kserve-6fb474f595-xm568"]
Oct 30 01:07:34.902120 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:34.902079    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d7720eca-fk52z"]
Oct 30 01:07:34.906435 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:34.906415    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7720eca-fk52z"
Oct 30 01:07:34.929883 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:34.929845    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d7720eca-fk52z"]
Oct 30 01:07:34.935000 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:34.934972    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:07:34.935215 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:34.934987    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:07:34.935215 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:34.935142    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:07:34.936333 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:34.936311    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5phsh\""
Oct 30 01:07:35.000526 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:35.000481    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33b7d06d-4835-41b8-aa1f-aa3b68232ff6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d7720eca-fk52z\" (UID: \"33b7d06d-4835-41b8-aa1f-aa3b68232ff6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7720eca-fk52z"
Oct 30 01:07:35.000852 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:35.000836    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33b7d06d-4835-41b8-aa1f-aa3b68232ff6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d7720eca-fk52z\" (UID: \"33b7d06d-4835-41b8-aa1f-aa3b68232ff6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7720eca-fk52z"
Oct 30 01:07:35.001071 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:35.001054    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmp7b\" (UniqueName: \"kubernetes.io/projected/33b7d06d-4835-41b8-aa1f-aa3b68232ff6-kube-api-access-qmp7b\") pod \"maas-api-revoke-sub-d7720eca-fk52z\" (UID: \"33b7d06d-4835-41b8-aa1f-aa3b68232ff6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7720eca-fk52z"
Oct 30 01:07:35.102678 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:35.102636    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33b7d06d-4835-41b8-aa1f-aa3b68232ff6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d7720eca-fk52z\" (UID: \"33b7d06d-4835-41b8-aa1f-aa3b68232ff6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7720eca-fk52z"
Oct 30 01:07:35.102852 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:35.102754    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmp7b\" (UniqueName: \"kubernetes.io/projected/33b7d06d-4835-41b8-aa1f-aa3b68232ff6-kube-api-access-qmp7b\") pod \"maas-api-revoke-sub-d7720eca-fk52z\" (UID: \"33b7d06d-4835-41b8-aa1f-aa3b68232ff6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7720eca-fk52z"
Oct 30 01:07:35.102852 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:35.102839    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33b7d06d-4835-41b8-aa1f-aa3b68232ff6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d7720eca-fk52z\" (UID: \"33b7d06d-4835-41b8-aa1f-aa3b68232ff6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7720eca-fk52z"
Oct 30 01:07:35.103549 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:35.103518    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33b7d06d-4835-41b8-aa1f-aa3b68232ff6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d7720eca-fk52z\" (UID: \"33b7d06d-4835-41b8-aa1f-aa3b68232ff6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7720eca-fk52z"
Oct 30 01:07:35.105161 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:35.105139    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33b7d06d-4835-41b8-aa1f-aa3b68232ff6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d7720eca-fk52z\" (UID: \"33b7d06d-4835-41b8-aa1f-aa3b68232ff6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7720eca-fk52z"
Oct 30 01:07:35.127793 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:35.127763    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmp7b\" (UniqueName: \"kubernetes.io/projected/33b7d06d-4835-41b8-aa1f-aa3b68232ff6-kube-api-access-qmp7b\") pod \"maas-api-revoke-sub-d7720eca-fk52z\" (UID: \"33b7d06d-4835-41b8-aa1f-aa3b68232ff6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7720eca-fk52z"
Oct 30 01:07:35.140733 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:35.140699    5002 patch_prober.go:29] interesting pod/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-rnd66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body=
Oct 30 01:07:35.140871 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:35.140757    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-rnd66" podUID="4df0b3e0-04ca-440d-9b4c-01635ee412e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused"
Oct 30 01:07:35.232438 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:35.232405    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7720eca-fk52z"
Oct 30 01:07:35.648715 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:35.648677    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d7720eca-fk52z"]
Oct 30 01:07:36.141334 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:36.141301    5002 patch_prober.go:29] interesting pod/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-rnd66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body=
Oct 30 01:07:36.141733 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:36.141353    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-rnd66" podUID="4df0b3e0-04ca-440d-9b4c-01635ee412e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused"
Oct 30 01:07:36.637425 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:36.637362    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7720eca-fk52z" event={"ID":"33b7d06d-4835-41b8-aa1f-aa3b68232ff6","Type":"ContainerStarted","Data":"77586d68a777918d046f1f66b986c0495cc11601d5e8e1992d3896bd76dd479f"}
Oct 30 01:07:36.637626 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:36.637455    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7720eca-fk52z" event={"ID":"33b7d06d-4835-41b8-aa1f-aa3b68232ff6","Type":"ContainerStarted","Data":"00f6c5eaf7c322ba4a00da387b05b1313956c501fb4df1b41c0ff2f29dad8e6a"}
Oct 30 01:07:36.704043 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:36.703969    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7720eca-fk52z" podStartSLOduration=2.7039516580000003 podStartE2EDuration="2.703951658s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:07:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:07:36.701970753 +0000 UTC m=+2152.318403633" watchObservedRunningTime="2026-10-30 01:07:36.703951658 +0000 UTC m=+2152.320384536"
Oct 30 01:07:37.140368 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:37.140275    5002 patch_prober.go:29] interesting pod/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-rnd66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body=
Oct 30 01:07:37.140368 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:37.140323    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-rnd66" podUID="4df0b3e0-04ca-440d-9b4c-01635ee412e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused"
Oct 30 01:07:37.646077 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:37.646039    5002 generic.go:356] "Generic (PLEG): container finished" podID="33b7d06d-4835-41b8-aa1f-aa3b68232ff6" containerID="77586d68a777918d046f1f66b986c0495cc11601d5e8e1992d3896bd76dd479f" exitCode=0
Oct 30 01:07:37.646566 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:37.646133    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7720eca-fk52z" event={"ID":"33b7d06d-4835-41b8-aa1f-aa3b68232ff6","Type":"ContainerDied","Data":"77586d68a777918d046f1f66b986c0495cc11601d5e8e1992d3896bd76dd479f"}
Oct 30 01:07:38.140763 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:38.140723    5002 patch_prober.go:29] interesting pod/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-rnd66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body=
Oct 30 01:07:38.140938 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:38.140786    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-rnd66" podUID="4df0b3e0-04ca-440d-9b4c-01635ee412e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused"
Oct 30 01:07:38.954655 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:38.954617    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7720eca-fk52z"
Oct 30 01:07:39.040005 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:39.039968    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-99424893-422qg"]
Oct 30 01:07:39.045540 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:39.045498    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-99424893-422qg"]
Oct 30 01:07:39.141726 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:39.141626    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/33b7d06d-4835-41b8-aa1f-aa3b68232ff6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33b7d06d-4835-41b8-aa1f-aa3b68232ff6-maas-api-xks-ca\") pod \"33b7d06d-4835-41b8-aa1f-aa3b68232ff6\" (UID: \"33b7d06d-4835-41b8-aa1f-aa3b68232ff6\") "
Oct 30 01:07:39.141726 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:39.141719    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33b7d06d-4835-41b8-aa1f-aa3b68232ff6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33b7d06d-4835-41b8-aa1f-aa3b68232ff6-maas-api-openshift-ca\") pod \"33b7d06d-4835-41b8-aa1f-aa3b68232ff6\" (UID: \"33b7d06d-4835-41b8-aa1f-aa3b68232ff6\") "
Oct 30 01:07:39.141993 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:39.141904    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33b7d06d-4835-41b8-aa1f-aa3b68232ff6-kube-api-access-qmp7b\" (UniqueName: \"kubernetes.io/projected/33b7d06d-4835-41b8-aa1f-aa3b68232ff6-kube-api-access-qmp7b\") pod \"33b7d06d-4835-41b8-aa1f-aa3b68232ff6\" (UID: \"33b7d06d-4835-41b8-aa1f-aa3b68232ff6\") "
Oct 30 01:07:39.143071 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:39.143038    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33b7d06d-4835-41b8-aa1f-aa3b68232ff6-maas-api-openshift-ca" pod "33b7d06d-4835-41b8-aa1f-aa3b68232ff6" (UID: "33b7d06d-4835-41b8-aa1f-aa3b68232ff6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:07:39.145100 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:39.145046    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33b7d06d-4835-41b8-aa1f-aa3b68232ff6-maas-api-xks-ca" pod "33b7d06d-4835-41b8-aa1f-aa3b68232ff6" (UID: "33b7d06d-4835-41b8-aa1f-aa3b68232ff6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:07:39.145454 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:39.145419    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33b7d06d-4835-41b8-aa1f-aa3b68232ff6-kube-api-access-qmp7b" pod "33b7d06d-4835-41b8-aa1f-aa3b68232ff6" (UID: "33b7d06d-4835-41b8-aa1f-aa3b68232ff6"). InnerVolumeSpecName "kube-api-access-qmp7b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:07:39.244260 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:39.244213    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qmp7b\" (UniqueName: \"kubernetes.io/projected/33b7d06d-4835-41b8-aa1f-aa3b68232ff6-kube-api-access-qmp7b\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:07:39.244260 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:39.244253    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33b7d06d-4835-41b8-aa1f-aa3b68232ff6-maas-api-xks-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:07:39.244260 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:39.244267    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33b7d06d-4835-41b8-aa1f-aa3b68232ff6-maas-api-openshift-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:07:39.538241 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:39.538149    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-rnd66"
Oct 30 01:07:39.539562 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:39.539525    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-rnd66"
Oct 30 01:07:39.552656 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:39.552629    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-rnd66"
Oct 30 01:07:39.683815 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:39.683786    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7720eca-fk52z"
Oct 30 01:07:39.692797 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:39.684025    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7720eca-fk52z" event={"ID":"33b7d06d-4835-41b8-aa1f-aa3b68232ff6","Type":"ContainerDied","Data":"00f6c5eaf7c322ba4a00da387b05b1313956c501fb4df1b41c0ff2f29dad8e6a"}
Oct 30 01:07:39.692797 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:39.692572    5002 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="00f6c5eaf7c322ba4a00da387b05b1313956c501fb4df1b41c0ff2f29dad8e6a"
Oct 30 01:07:40.056432 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:40.056300    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29a0c0c1-8jlgp"]
Oct 30 01:07:40.059535 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:40.059506    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29a0c0c1-8jlgp"]
Oct 30 01:07:40.496504 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:40.496465    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d6db577-4b91-4d50-8ac5-7b73b24aeb80" path="/var/lib/kubelet/pods/6d6db577-4b91-4d50-8ac5-7b73b24aeb80/volumes"
Oct 30 01:07:40.496978 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:40.496963    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9866a8d4-b855-461b-bcb6-2dbd0bcdef0b" path="/var/lib/kubelet/pods/9866a8d4-b855-461b-bcb6-2dbd0bcdef0b/volumes"
Oct 30 01:07:42.865312 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:07:42.865261    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbjdq" containerName="extractor"
Oct 30 01:07:42.996802 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:42.996765    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-5f387c-openshift-default-5ff776bc59-bdp4r"]
Oct 30 01:07:42.997458 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:42.997439    5002 cpu_manager.go:414] "containerMap: removing container" podUID="33b7d06d-4835-41b8-aa1f-aa3b68232ff6" containerName="revoke-keys"
Oct 30 01:07:42.997520 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:42.997459    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33b7d06d-4835-41b8-aa1f-aa3b68232ff6" containerName="revoke-keys"
Oct 30 01:07:42.997580 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:42.997571    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="33b7d06d-4835-41b8-aa1f-aa3b68232ff6" containerName="revoke-keys"
Oct 30 01:07:43.002672 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:43.002649    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-5f387c-openshift-default-5ff776bc59-bdp4r"
Oct 30 01:07:43.011241 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:43.011212    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bb1708e-878e-461f-9294-265673c31b04-credential-socket\") pod \"e2e-shared-a-w0-5f387c-openshift-default-5ff776bc59-bdp4r\" (UID: \"9bb1708e-878e-461f-9294-265673c31b04\") " pod="openshift-ingress/e2e-shared-a-w0-5f387c-openshift-default-5ff776bc59-bdp4r"
Oct 30 01:07:43.011364 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:43.011338    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9bb1708e-878e-461f-9294-265673c31b04-istio-token\") pod \"e2e-shared-a-w0-5f387c-openshift-default-5ff776bc59-bdp4r\" (UID: \"9bb1708e-878e-461f-9294-265673c31b04\") " pod="openshift-ingress/e2e-shared-a-w0-5f387c-openshift-default-5ff776bc59-bdp4r"
Oct 30 01:07:43.011477 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:43.011447    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9bb1708e-878e-461f-9294-265673c31b04-istiod-ca-cert\") pod \"e2e-shared-a-w0-5f387c-openshift-default-5ff776bc59-bdp4r\" (UID: \"9bb1708e-878e-461f-9294-265673c31b04\") " pod="openshift-ingress/e2e-shared-a-w0-5f387c-openshift-default-5ff776bc59-bdp4r"
Oct 30 01:07:43.011532 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:43.011514    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9bb1708e-878e-461f-9294-265673c31b04-istio-podinfo\") pod \"e2e-shared-a-w0-5f387c-openshift-default-5ff776bc59-bdp4r\" (UID: \"9bb1708e-878e-461f-9294-265673c31b04\") " pod="openshift-ingress/e2e-shared-a-w0-5f387c-openshift-default-5ff776bc59-bdp4r"
Oct 30 01:07:43.011593 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:43.011578    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8fjz\" (UniqueName: \"kubernetes.io/projected/9bb1708e-878e-461f-9294-265673c31b04-kube-api-access-m8fjz\") pod \"e2e-shared-a-w0-5f387c-openshift-default-5ff776bc59-bdp4r\" (UID: \"9bb1708e-878e-461f-9294-265673c31b04\") " pod="openshift-ingress/e2e-shared-a-w0-5f387c-openshift-default-5ff776bc59-bdp4r"
Oct 30 01:07:43.011701 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:43.011676    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9bb1708e-878e-461f-9294-265673c31b04-workload-certs\") pod \"e2e-shared-a-w0-5f387c-openshift-default-5ff776bc59-bdp4r\" (UID: \"9bb1708e-878e-461f-9294-265673c31b04\") " pod="openshift-ingress/e2e-shared-a-w0-5f387c-openshift-default-5ff776bc59-bdp4r"
Oct 30 01:07:43.011804 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:43.011783    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9bb1708e-878e-461f-9294-265673c31b04-istio-data\") pod \"e2e-shared-a-w0-5f387c-openshift-default-5ff776bc59-bdp4r\" (UID: \"9bb1708e-878e-461f-9294-265673c31b04\") " pod="openshift-ingress/e2e-shared-a-w0-5f387c-openshift-default-5ff776bc59-bdp4r"
Oct 30 01:07:43.011873 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:43.011852    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bb1708e-878e-461f-9294-265673c31b04-workload-socket\") pod \"e2e-shared-a-w0-5f387c-openshift-default-5ff776bc59-bdp4r\" (UID: \"9bb1708e-878e-461f-9294-265673c31b04\") " pod="openshift-ingress/e2e-shared-a-w0-5f387c-openshift-default-5ff776bc59-bdp4r"
Oct 30 01:07:43.011928 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:43.011879    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9bb1708e-878e-461f-9294-265673c31b04-istio-envoy\") pod \"e2e-shared-a-w0-5f387c-openshift-default-5ff776bc59-bdp4r\" (UID: \"9bb1708e-878e-461f-9294-265673c31b04\") " pod="openshift-ingress/e2e-shared-a-w0-5f387c-openshift-default-5ff776bc59-bdp4r"
Oct 30 01:07:43.027502 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:43.027465    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-5f387c-openshift-default-5ff776bc59-bdp4r"]
Oct 30 01:07:43.035396 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:43.035342    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-5f387c-openshift-default-dockercfg-7wl24\""
Oct 30 01:07:43.113894 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:43.113626    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9bb1708e-878e-461f-9294-265673c31b04-istio-podinfo\") pod \"e2e-shared-a-w0-5f387c-openshift-default-5ff776bc59-bdp4r\" (UID: \"9bb1708e-878e-461f-9294-265673c31b04\") " pod="openshift-ingress/e2e-shared-a-w0-5f387c-openshift-default-5ff776bc59-bdp4r"
Oct 30 01:07:43.113894 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:43.113723    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8fjz\" (UniqueName: \"kubernetes.io/projected/9bb1708e-878e-461f-9294-265673c31b04-kube-api-access-m8fjz\") pod \"e2e-shared-a-w0-5f387c-openshift-default-5ff776bc59-bdp4r\" (UID: \"9bb1708e-878e-461f-9294-265673c31b04\") " pod="openshift-ingress/e2e-shared-a-w0-5f387c-openshift-default-5ff776bc59-bdp4r"
Oct 30 01:07:43.113894 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:43.113770    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9bb1708e-878e-461f-9294-265673c31b04-workload-certs\") pod \"e2e-shared-a-w0-5f387c-openshift-default-5ff776bc59-bdp4r\" (UID: \"9bb1708e-878e-461f-9294-265673c31b04\") " pod="openshift-ingress/e2e-shared-a-w0-5f387c-openshift-default-5ff776bc59-bdp4r"
Oct 30 01:07:43.113894 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:43.113837    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9bb1708e-878e-461f-9294-265673c31b04-istio-data\") pod \"e2e-shared-a-w0-5f387c-openshift-default-5ff776bc59-bdp4r\" (UID: \"9bb1708e-878e-461f-9294-265673c31b04\") " pod="openshift-ingress/e2e-shared-a-w0-5f387c-openshift-default-5ff776bc59-bdp4r"
Oct 30 01:07:43.114279 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:43.113977    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bb1708e-878e-461f-9294-265673c31b04-workload-socket\") pod \"e2e-shared-a-w0-5f387c-openshift-default-5ff776bc59-bdp4r\" (UID: \"9bb1708e-878e-461f-9294-265673c31b04\") " pod="openshift-ingress/e2e-shared-a-w0-5f387c-openshift-default-5ff776bc59-bdp4r"
Oct 30 01:07:43.114279 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:43.114015    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9bb1708e-878e-461f-9294-265673c31b04-istio-envoy\") pod \"e2e-shared-a-w0-5f387c-openshift-default-5ff776bc59-bdp4r\" (UID: \"9bb1708e-878e-461f-9294-265673c31b04\") " pod="openshift-ingress/e2e-shared-a-w0-5f387c-openshift-default-5ff776bc59-bdp4r"
Oct 30 01:07:43.114279 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:43.114161    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bb1708e-878e-461f-9294-265673c31b04-credential-socket\") pod \"e2e-shared-a-w0-5f387c-openshift-default-5ff776bc59-bdp4r\" (UID: \"9bb1708e-878e-461f-9294-265673c31b04\") " pod="openshift-ingress/e2e-shared-a-w0-5f387c-openshift-default-5ff776bc59-bdp4r"
Oct 30 01:07:43.114279 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:43.114256    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9bb1708e-878e-461f-9294-265673c31b04-istio-token\") pod \"e2e-shared-a-w0-5f387c-openshift-default-5ff776bc59-bdp4r\" (UID: \"9bb1708e-878e-461f-9294-265673c31b04\") " pod="openshift-ingress/e2e-shared-a-w0-5f387c-openshift-default-5ff776bc59-bdp4r"
Oct 30 01:07:43.114567 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:43.114284    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9bb1708e-878e-461f-9294-265673c31b04-istiod-ca-cert\") pod \"e2e-shared-a-w0-5f387c-openshift-default-5ff776bc59-bdp4r\" (UID: \"9bb1708e-878e-461f-9294-265673c31b04\") " pod="openshift-ingress/e2e-shared-a-w0-5f387c-openshift-default-5ff776bc59-bdp4r"
Oct 30 01:07:43.115131 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:43.115064    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9bb1708e-878e-461f-9294-265673c31b04-istio-data\") pod \"e2e-shared-a-w0-5f387c-openshift-default-5ff776bc59-bdp4r\" (UID: \"9bb1708e-878e-461f-9294-265673c31b04\") " pod="openshift-ingress/e2e-shared-a-w0-5f387c-openshift-default-5ff776bc59-bdp4r"
Oct 30 01:07:43.115570 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:43.115487    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9bb1708e-878e-461f-9294-265673c31b04-istiod-ca-cert\") pod \"e2e-shared-a-w0-5f387c-openshift-default-5ff776bc59-bdp4r\" (UID: \"9bb1708e-878e-461f-9294-265673c31b04\") " pod="openshift-ingress/e2e-shared-a-w0-5f387c-openshift-default-5ff776bc59-bdp4r"
Oct 30 01:07:43.115891 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:43.115866    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bb1708e-878e-461f-9294-265673c31b04-workload-socket\") pod \"e2e-shared-a-w0-5f387c-openshift-default-5ff776bc59-bdp4r\" (UID: \"9bb1708e-878e-461f-9294-265673c31b04\") " pod="openshift-ingress/e2e-shared-a-w0-5f387c-openshift-default-5ff776bc59-bdp4r"
Oct 30 01:07:43.118198 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:43.118143    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9bb1708e-878e-461f-9294-265673c31b04-workload-certs\") pod \"e2e-shared-a-w0-5f387c-openshift-default-5ff776bc59-bdp4r\" (UID: \"9bb1708e-878e-461f-9294-265673c31b04\") " pod="openshift-ingress/e2e-shared-a-w0-5f387c-openshift-default-5ff776bc59-bdp4r"
Oct 30 01:07:43.118431 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:43.118406    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bb1708e-878e-461f-9294-265673c31b04-credential-socket\") pod \"e2e-shared-a-w0-5f387c-openshift-default-5ff776bc59-bdp4r\" (UID: \"9bb1708e-878e-461f-9294-265673c31b04\") " pod="openshift-ingress/e2e-shared-a-w0-5f387c-openshift-default-5ff776bc59-bdp4r"
Oct 30 01:07:43.119424 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:43.119351    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9bb1708e-878e-461f-9294-265673c31b04-istio-envoy\") pod \"e2e-shared-a-w0-5f387c-openshift-default-5ff776bc59-bdp4r\" (UID: \"9bb1708e-878e-461f-9294-265673c31b04\") " pod="openshift-ingress/e2e-shared-a-w0-5f387c-openshift-default-5ff776bc59-bdp4r"
Oct 30 01:07:43.120994 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:43.120970    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9bb1708e-878e-461f-9294-265673c31b04-istio-podinfo\") pod \"e2e-shared-a-w0-5f387c-openshift-default-5ff776bc59-bdp4r\" (UID: \"9bb1708e-878e-461f-9294-265673c31b04\") " pod="openshift-ingress/e2e-shared-a-w0-5f387c-openshift-default-5ff776bc59-bdp4r"
Oct 30 01:07:43.143026 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:43.142995    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8fjz\" (UniqueName: \"kubernetes.io/projected/9bb1708e-878e-461f-9294-265673c31b04-kube-api-access-m8fjz\") pod \"e2e-shared-a-w0-5f387c-openshift-default-5ff776bc59-bdp4r\" (UID: \"9bb1708e-878e-461f-9294-265673c31b04\") " pod="openshift-ingress/e2e-shared-a-w0-5f387c-openshift-default-5ff776bc59-bdp4r"
Oct 30 01:07:43.143841 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:43.143067    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9bb1708e-878e-461f-9294-265673c31b04-istio-token\") pod \"e2e-shared-a-w0-5f387c-openshift-default-5ff776bc59-bdp4r\" (UID: \"9bb1708e-878e-461f-9294-265673c31b04\") " pod="openshift-ingress/e2e-shared-a-w0-5f387c-openshift-default-5ff776bc59-bdp4r"
Oct 30 01:07:43.318463 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:43.318165    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-5f387c-openshift-default-5ff776bc59-bdp4r"
Oct 30 01:07:43.646181 ip-10-0-128-7 kubenswrapper[5002]: W1030 01:07:43.646154    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bb1708e_878e_461f_9294_265673c31b04.slice/crio-aa45ab1198e767cd66826f199f7801061dd2b6c6ef93213f5408006cf85aed03 WatchSource:0}: Error finding container aa45ab1198e767cd66826f199f7801061dd2b6c6ef93213f5408006cf85aed03: Status 404 returned error can't find the container with id aa45ab1198e767cd66826f199f7801061dd2b6c6ef93213f5408006cf85aed03
Oct 30 01:07:43.648406 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:43.648366    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-5f387c-openshift-default-5ff776bc59-bdp4r"]
Oct 30 01:07:43.649414 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:43.649344    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:07:43.649494 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:43.649447    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:07:43.649494 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:43.649482    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:07:43.726390 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:43.726355    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-5f387c-openshift-default-5ff776bc59-bdp4r" event={"ID":"9bb1708e-878e-461f-9294-265673c31b04","Type":"ContainerStarted","Data":"aa45ab1198e767cd66826f199f7801061dd2b6c6ef93213f5408006cf85aed03"}
Oct 30 01:07:44.735308 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:44.735271    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-5f387c-openshift-default-5ff776bc59-bdp4r" event={"ID":"9bb1708e-878e-461f-9294-265673c31b04","Type":"ContainerStarted","Data":"8166e672cbf52094e193b459b9fea1ced005ca693a6b82ce4d7f64895bd7d14e"}
Oct 30 01:07:44.807892 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:44.807839    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-5f387c-openshift-default-5ff776bc59-bdp4r" podStartSLOduration=2.807824198 podStartE2EDuration="2.807824198s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:07:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:07:44.805788489 +0000 UTC m=+2160.422221368" watchObservedRunningTime="2026-10-30 01:07:44.807824198 +0000 UTC m=+2160.424257076"
Oct 30 01:07:45.318767 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:45.318732    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-5f387c-openshift-default-5ff776bc59-bdp4r"
Oct 30 01:07:45.320772 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:45.320741    5002 patch_prober.go:29] interesting pod/e2e-shared-a-w0-5f387c-openshift-default-5ff776bc59-bdp4r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body=
Oct 30 01:07:45.320894 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:45.320793    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-5f387c-openshift-default-5ff776bc59-bdp4r" podUID="9bb1708e-878e-461f-9294-265673c31b04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused"
Oct 30 01:07:46.318748 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:46.318704    5002 patch_prober.go:29] interesting pod/e2e-shared-a-w0-5f387c-openshift-default-5ff776bc59-bdp4r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body=
Oct 30 01:07:46.319158 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:46.318768    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-5f387c-openshift-default-5ff776bc59-bdp4r" podUID="9bb1708e-878e-461f-9294-265673c31b04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused"
Oct 30 01:07:47.319112 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:47.319073    5002 patch_prober.go:29] interesting pod/e2e-shared-a-w0-5f387c-openshift-default-5ff776bc59-bdp4r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body=
Oct 30 01:07:47.319630 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:47.319137    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-5f387c-openshift-default-5ff776bc59-bdp4r" podUID="9bb1708e-878e-461f-9294-265673c31b04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused"
Oct 30 01:07:47.637677 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:47.637586    5002 scope.go:122] "RemoveContainer" containerID="217a2ee7d3bfe7dc6022dbe22bcd61bd9a8e48fa2caeb8e534a2513534442545"
Oct 30 01:07:47.648649 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:47.648626    5002 scope.go:122] "RemoveContainer" containerID="43ca09206436b3afbd96407256020597f97989fd0be50f5bbc366cd58e853eee"
Oct 30 01:07:47.682448 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:47.682417    5002 scope.go:122] "RemoveContainer" containerID="1adb422bfd667770bfc95c1be46853f10ec8122031f8b9cf7ab8ae826c1a740b"
Oct 30 01:07:48.319581 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:48.319545    5002 patch_prober.go:29] interesting pod/e2e-shared-a-w0-5f387c-openshift-default-5ff776bc59-bdp4r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body=
Oct 30 01:07:48.320012 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:48.319600    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-5f387c-openshift-default-5ff776bc59-bdp4r" podUID="9bb1708e-878e-461f-9294-265673c31b04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused"
Oct 30 01:07:48.531344 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:48.531308    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-19bc68/test-model-19bc68-kserve-77fd689c7c-qw7lc"]
Oct 30 01:07:48.787662 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:48.773439    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ef1084-f6m5q"]
Oct 30 01:07:48.787662 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:48.780461    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ef1084-f6m5q"
Oct 30 01:07:48.800540 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:48.800217    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:07:48.832664 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:48.813811    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5phsh\""
Oct 30 01:07:48.832664 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:48.814036    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:07:48.832664 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:48.814144    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:07:48.832664 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:48.830574    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ef1084-f6m5q"]
Oct 30 01:07:48.879098 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:48.867035    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23d9111c-b9d3-4691-9545-0e0cf3689359-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-ef1084-f6m5q\" (UID: \"23d9111c-b9d3-4691-9545-0e0cf3689359\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ef1084-f6m5q"
Oct 30 01:07:48.879098 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:48.867146    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23d9111c-b9d3-4691-9545-0e0cf3689359-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-ef1084-f6m5q\" (UID: \"23d9111c-b9d3-4691-9545-0e0cf3689359\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ef1084-f6m5q"
Oct 30 01:07:48.879098 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:48.867199    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcqf5\" (UniqueName: \"kubernetes.io/projected/23d9111c-b9d3-4691-9545-0e0cf3689359-kube-api-access-wcqf5\") pod \"maas-api-revoke-keys-e2e-inf-a-ef1084-f6m5q\" (UID: \"23d9111c-b9d3-4691-9545-0e0cf3689359\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ef1084-f6m5q"
Oct 30 01:07:49.014436 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:49.007269    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23d9111c-b9d3-4691-9545-0e0cf3689359-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-ef1084-f6m5q\" (UID: \"23d9111c-b9d3-4691-9545-0e0cf3689359\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ef1084-f6m5q"
Oct 30 01:07:49.014436 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:49.007434    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23d9111c-b9d3-4691-9545-0e0cf3689359-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-ef1084-f6m5q\" (UID: \"23d9111c-b9d3-4691-9545-0e0cf3689359\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ef1084-f6m5q"
Oct 30 01:07:49.014436 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:49.007513    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcqf5\" (UniqueName: \"kubernetes.io/projected/23d9111c-b9d3-4691-9545-0e0cf3689359-kube-api-access-wcqf5\") pod \"maas-api-revoke-keys-e2e-inf-a-ef1084-f6m5q\" (UID: \"23d9111c-b9d3-4691-9545-0e0cf3689359\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ef1084-f6m5q"
Oct 30 01:07:49.014436 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:49.008557    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23d9111c-b9d3-4691-9545-0e0cf3689359-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-ef1084-f6m5q\" (UID: \"23d9111c-b9d3-4691-9545-0e0cf3689359\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ef1084-f6m5q"
Oct 30 01:07:49.014436 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:49.013404    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23d9111c-b9d3-4691-9545-0e0cf3689359-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-ef1084-f6m5q\" (UID: \"23d9111c-b9d3-4691-9545-0e0cf3689359\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ef1084-f6m5q"
Oct 30 01:07:49.038446 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:49.038348    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcqf5\" (UniqueName: \"kubernetes.io/projected/23d9111c-b9d3-4691-9545-0e0cf3689359-kube-api-access-wcqf5\") pod \"maas-api-revoke-keys-e2e-inf-a-ef1084-f6m5q\" (UID: \"23d9111c-b9d3-4691-9545-0e0cf3689359\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ef1084-f6m5q"
Oct 30 01:07:49.200852 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:49.200809    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ef1084-f6m5q"
Oct 30 01:07:49.320630 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:49.319745    5002 patch_prober.go:29] interesting pod/e2e-shared-a-w0-5f387c-openshift-default-5ff776bc59-bdp4r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body=
Oct 30 01:07:49.320630 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:49.319821    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-5f387c-openshift-default-5ff776bc59-bdp4r" podUID="9bb1708e-878e-461f-9294-265673c31b04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused"
Oct 30 01:07:49.418531 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:49.418434    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-ef1084/test-model-ef1084-kserve-6fb474f595-xm568" podUID="652cd481-aaf4-4f3d-8532-978fe17201c8" containerName="main" containerID="cri-o://856e94692202d759fc4f8ab9d6e93edcafdbf33abdabda95860b2eef0173879e" gracePeriod=45
Oct 30 01:07:49.582405 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:49.580187    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ef1084-f6m5q"]
Oct 30 01:07:49.782072 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:49.782038    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-ef1084/test-model-ef1084-kserve-6fb474f595-xm568"
Oct 30 01:07:49.928675 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:49.928579    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/652cd481-aaf4-4f3d-8532-978fe17201c8-tls-certs\" (UniqueName: \"kubernetes.io/secret/652cd481-aaf4-4f3d-8532-978fe17201c8-tls-certs\") pod \"652cd481-aaf4-4f3d-8532-978fe17201c8\" (UID: \"652cd481-aaf4-4f3d-8532-978fe17201c8\") "
Oct 30 01:07:49.928838 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:49.928701    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/652cd481-aaf4-4f3d-8532-978fe17201c8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/652cd481-aaf4-4f3d-8532-978fe17201c8-model-cache\") pod \"652cd481-aaf4-4f3d-8532-978fe17201c8\" (UID: \"652cd481-aaf4-4f3d-8532-978fe17201c8\") "
Oct 30 01:07:49.928838 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:49.928810    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/652cd481-aaf4-4f3d-8532-978fe17201c8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/652cd481-aaf4-4f3d-8532-978fe17201c8-tmp-dir\") pod \"652cd481-aaf4-4f3d-8532-978fe17201c8\" (UID: \"652cd481-aaf4-4f3d-8532-978fe17201c8\") "
Oct 30 01:07:49.928963 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:49.928870    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/652cd481-aaf4-4f3d-8532-978fe17201c8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/652cd481-aaf4-4f3d-8532-978fe17201c8-dshm\") pod \"652cd481-aaf4-4f3d-8532-978fe17201c8\" (UID: \"652cd481-aaf4-4f3d-8532-978fe17201c8\") "
Oct 30 01:07:49.928963 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:49.928898    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/652cd481-aaf4-4f3d-8532-978fe17201c8-home\" (UniqueName: \"kubernetes.io/empty-dir/652cd481-aaf4-4f3d-8532-978fe17201c8-home\") pod \"652cd481-aaf4-4f3d-8532-978fe17201c8\" (UID: \"652cd481-aaf4-4f3d-8532-978fe17201c8\") "
Oct 30 01:07:49.929079 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:49.928974    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/652cd481-aaf4-4f3d-8532-978fe17201c8-kube-api-access-zh8v5\" (UniqueName: \"kubernetes.io/projected/652cd481-aaf4-4f3d-8532-978fe17201c8-kube-api-access-zh8v5\") pod \"652cd481-aaf4-4f3d-8532-978fe17201c8\" (UID: \"652cd481-aaf4-4f3d-8532-978fe17201c8\") "
Oct 30 01:07:49.929079 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:49.929000    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/652cd481-aaf4-4f3d-8532-978fe17201c8-model-cache" pod "652cd481-aaf4-4f3d-8532-978fe17201c8" (UID: "652cd481-aaf4-4f3d-8532-978fe17201c8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:07:49.929197 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:49.929121    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/652cd481-aaf4-4f3d-8532-978fe17201c8-tmp-dir" pod "652cd481-aaf4-4f3d-8532-978fe17201c8" (UID: "652cd481-aaf4-4f3d-8532-978fe17201c8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:07:49.929255 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:49.929222    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/652cd481-aaf4-4f3d-8532-978fe17201c8-home" pod "652cd481-aaf4-4f3d-8532-978fe17201c8" (UID: "652cd481-aaf4-4f3d-8532-978fe17201c8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:07:49.929755 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:49.929727    5002 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/652cd481-aaf4-4f3d-8532-978fe17201c8-model-cache\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:07:49.929755 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:49.929756    5002 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/652cd481-aaf4-4f3d-8532-978fe17201c8-tmp-dir\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:07:49.929944 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:49.929770    5002 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/652cd481-aaf4-4f3d-8532-978fe17201c8-home\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:07:49.931746 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:49.931719    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/652cd481-aaf4-4f3d-8532-978fe17201c8-dshm" pod "652cd481-aaf4-4f3d-8532-978fe17201c8" (UID: "652cd481-aaf4-4f3d-8532-978fe17201c8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:07:49.931746 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:49.931731    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/652cd481-aaf4-4f3d-8532-978fe17201c8-tls-certs" pod "652cd481-aaf4-4f3d-8532-978fe17201c8" (UID: "652cd481-aaf4-4f3d-8532-978fe17201c8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:07:49.931966 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:49.931770    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/652cd481-aaf4-4f3d-8532-978fe17201c8-kube-api-access-zh8v5" pod "652cd481-aaf4-4f3d-8532-978fe17201c8" (UID: "652cd481-aaf4-4f3d-8532-978fe17201c8"). InnerVolumeSpecName "kube-api-access-zh8v5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:07:50.031597 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:50.031558    5002 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/652cd481-aaf4-4f3d-8532-978fe17201c8-tls-certs\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:07:50.031597 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:50.031591    5002 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/652cd481-aaf4-4f3d-8532-978fe17201c8-dshm\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:07:50.031785 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:50.031604    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zh8v5\" (UniqueName: \"kubernetes.io/projected/652cd481-aaf4-4f3d-8532-978fe17201c8-kube-api-access-zh8v5\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:07:50.032948 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:50.032918    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-efcbc3bd-5q876"]
Oct 30 01:07:50.035921 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:50.034960    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ef1084-f6m5q" event={"ID":"23d9111c-b9d3-4691-9545-0e0cf3689359","Type":"ContainerStarted","Data":"8155c960ca17cd303f695344efec7f795ab6964118a072508b07369862bb29f7"}
Oct 30 01:07:50.035921 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:50.035038    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ef1084-f6m5q" event={"ID":"23d9111c-b9d3-4691-9545-0e0cf3689359","Type":"ContainerStarted","Data":"8c346b02f273df252010991a8b547d12f9f4182d1607e545ba342a229ac70845"}
Oct 30 01:07:50.039185 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:50.039153    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-efcbc3bd-5q876"]
Oct 30 01:07:50.040014 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:50.039987    5002 generic.go:356] "Generic (PLEG): container finished" podID="652cd481-aaf4-4f3d-8532-978fe17201c8" containerID="856e94692202d759fc4f8ab9d6e93edcafdbf33abdabda95860b2eef0173879e" exitCode=0
Oct 30 01:07:50.040136 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:50.040029    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-ef1084/test-model-ef1084-kserve-6fb474f595-xm568"
Oct 30 01:07:50.040136 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:50.040077    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-ef1084/test-model-ef1084-kserve-6fb474f595-xm568" event={"ID":"652cd481-aaf4-4f3d-8532-978fe17201c8","Type":"ContainerDied","Data":"856e94692202d759fc4f8ab9d6e93edcafdbf33abdabda95860b2eef0173879e"}
Oct 30 01:07:50.040136 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:50.040119    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-ef1084/test-model-ef1084-kserve-6fb474f595-xm568" event={"ID":"652cd481-aaf4-4f3d-8532-978fe17201c8","Type":"ContainerDied","Data":"6645ee10e1614835f0ea1814757486afdb65be359e5077ba22b99663a41304b7"}
Oct 30 01:07:50.040330 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:50.040144    5002 scope.go:122] "RemoveContainer" containerID="856e94692202d759fc4f8ab9d6e93edcafdbf33abdabda95860b2eef0173879e"
Oct 30 01:07:50.057277 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:50.057247    5002 scope.go:122] "RemoveContainer" containerID="856e94692202d759fc4f8ab9d6e93edcafdbf33abdabda95860b2eef0173879e"
Oct 30 01:07:50.057704 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:07:50.057675    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"856e94692202d759fc4f8ab9d6e93edcafdbf33abdabda95860b2eef0173879e\": container with ID starting with 856e94692202d759fc4f8ab9d6e93edcafdbf33abdabda95860b2eef0173879e not found: ID does not exist" containerID="856e94692202d759fc4f8ab9d6e93edcafdbf33abdabda95860b2eef0173879e"
Oct 30 01:07:50.057774 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:50.057719    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"856e94692202d759fc4f8ab9d6e93edcafdbf33abdabda95860b2eef0173879e"} err="failed to get container status \"856e94692202d759fc4f8ab9d6e93edcafdbf33abdabda95860b2eef0173879e\": rpc error: code = NotFound desc = could not find container \"856e94692202d759fc4f8ab9d6e93edcafdbf33abdabda95860b2eef0173879e\": container with ID starting with 856e94692202d759fc4f8ab9d6e93edcafdbf33abdabda95860b2eef0173879e not found: ID does not exist"
Oct 30 01:07:50.081427 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:50.081344    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ef1084-f6m5q" podStartSLOduration=2.081330653 podStartE2EDuration="2.081330653s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:07:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:07:50.079169183 +0000 UTC m=+2165.695602061" watchObservedRunningTime="2026-10-30 01:07:50.081330653 +0000 UTC m=+2165.697763532"
Oct 30 01:07:50.123131 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:50.123090    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-ef1084/test-model-ef1084-kserve-6fb474f595-xm568"]
Oct 30 01:07:50.127370 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:50.127339    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-ef1084/test-model-ef1084-kserve-6fb474f595-xm568"]
Oct 30 01:07:50.365956 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:50.365918    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-5f387c-openshift-default-5ff776bc59-bdp4r"
Oct 30 01:07:50.367042 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:50.367006    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-5f387c-openshift-default-5ff776bc59-bdp4r"
Oct 30 01:07:50.369842 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:50.369820    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-5f387c-openshift-default-5ff776bc59-bdp4r"
Oct 30 01:07:50.514853 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:50.514816    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="652cd481-aaf4-4f3d-8532-978fe17201c8" path="/var/lib/kubelet/pods/652cd481-aaf4-4f3d-8532-978fe17201c8/volumes"
Oct 30 01:07:50.515692 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:50.515668    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce5d1af7-5443-41bb-9c6d-dbd173f6b2fd" path="/var/lib/kubelet/pods/ce5d1af7-5443-41bb-9c6d-dbd173f6b2fd/volumes"
Oct 30 01:07:51.050852 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:51.050766    5002 generic.go:356] "Generic (PLEG): container finished" podID="23d9111c-b9d3-4691-9545-0e0cf3689359" containerID="8155c960ca17cd303f695344efec7f795ab6964118a072508b07369862bb29f7" exitCode=0
Oct 30 01:07:51.051009 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:51.050849    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ef1084-f6m5q" event={"ID":"23d9111c-b9d3-4691-9545-0e0cf3689359","Type":"ContainerDied","Data":"8155c960ca17cd303f695344efec7f795ab6964118a072508b07369862bb29f7"}
Oct 30 01:07:52.244191 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:52.244159    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ef1084-f6m5q"
Oct 30 01:07:52.265923 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:52.265889    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23d9111c-b9d3-4691-9545-0e0cf3689359-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23d9111c-b9d3-4691-9545-0e0cf3689359-maas-api-openshift-ca\") pod \"23d9111c-b9d3-4691-9545-0e0cf3689359\" (UID: \"23d9111c-b9d3-4691-9545-0e0cf3689359\") "
Oct 30 01:07:52.266108 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:52.265977    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/23d9111c-b9d3-4691-9545-0e0cf3689359-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23d9111c-b9d3-4691-9545-0e0cf3689359-maas-api-xks-ca\") pod \"23d9111c-b9d3-4691-9545-0e0cf3689359\" (UID: \"23d9111c-b9d3-4691-9545-0e0cf3689359\") "
Oct 30 01:07:52.266108 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:52.266032    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23d9111c-b9d3-4691-9545-0e0cf3689359-kube-api-access-wcqf5\" (UniqueName: \"kubernetes.io/projected/23d9111c-b9d3-4691-9545-0e0cf3689359-kube-api-access-wcqf5\") pod \"23d9111c-b9d3-4691-9545-0e0cf3689359\" (UID: \"23d9111c-b9d3-4691-9545-0e0cf3689359\") "
Oct 30 01:07:52.266368 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:52.266341    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23d9111c-b9d3-4691-9545-0e0cf3689359-maas-api-openshift-ca" pod "23d9111c-b9d3-4691-9545-0e0cf3689359" (UID: "23d9111c-b9d3-4691-9545-0e0cf3689359"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:07:52.266787 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:52.266758    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23d9111c-b9d3-4691-9545-0e0cf3689359-maas-api-openshift-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:07:52.268349 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:52.268325    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23d9111c-b9d3-4691-9545-0e0cf3689359-kube-api-access-wcqf5" pod "23d9111c-b9d3-4691-9545-0e0cf3689359" (UID: "23d9111c-b9d3-4691-9545-0e0cf3689359"). InnerVolumeSpecName "kube-api-access-wcqf5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:07:52.268520 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:52.268510    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23d9111c-b9d3-4691-9545-0e0cf3689359-maas-api-xks-ca" pod "23d9111c-b9d3-4691-9545-0e0cf3689359" (UID: "23d9111c-b9d3-4691-9545-0e0cf3689359"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:07:52.381470 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:52.375656    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23d9111c-b9d3-4691-9545-0e0cf3689359-maas-api-xks-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:07:52.381470 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:52.375701    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wcqf5\" (UniqueName: \"kubernetes.io/projected/23d9111c-b9d3-4691-9545-0e0cf3689359-kube-api-access-wcqf5\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:07:53.071366 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:53.071333    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ef1084-f6m5q"
Oct 30 01:07:53.071611 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:53.071368    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ef1084-f6m5q" event={"ID":"23d9111c-b9d3-4691-9545-0e0cf3689359","Type":"ContainerDied","Data":"8c346b02f273df252010991a8b547d12f9f4182d1607e545ba342a229ac70845"}
Oct 30 01:07:53.071611 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:53.071462    5002 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8c346b02f273df252010991a8b547d12f9f4182d1607e545ba342a229ac70845"
Oct 30 01:07:59.144655 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:59.144613    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ef1084-66c68bf584-58x9z"]
Oct 30 01:07:59.179692 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:07:59.179617    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ef1084-64c6bbc5ff-rrjq8"]
Oct 30 01:08:03.615563 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:03.615519    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-19bc68/test-model-19bc68-kserve-77fd689c7c-qw7lc" podUID="076dfc74-4c57-46c9-946f-f06e3fb77f79" containerName="main" containerID="cri-o://09f085c8afdf1111d2dc08b2cbc5d3cccc79dec9aad9ad2fcec4e30737e4a26d" gracePeriod=45
Oct 30 01:08:03.922825 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:03.922795    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-19bc68/test-model-19bc68-kserve-77fd689c7c-qw7lc"
Oct 30 01:08:03.962342 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:03.962303    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/076dfc74-4c57-46c9-946f-f06e3fb77f79-tls-certs\" (UniqueName: \"kubernetes.io/secret/076dfc74-4c57-46c9-946f-f06e3fb77f79-tls-certs\") pod \"076dfc74-4c57-46c9-946f-f06e3fb77f79\" (UID: \"076dfc74-4c57-46c9-946f-f06e3fb77f79\") "
Oct 30 01:08:03.962528 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:03.962465    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/076dfc74-4c57-46c9-946f-f06e3fb77f79-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/076dfc74-4c57-46c9-946f-f06e3fb77f79-tmp-dir\") pod \"076dfc74-4c57-46c9-946f-f06e3fb77f79\" (UID: \"076dfc74-4c57-46c9-946f-f06e3fb77f79\") "
Oct 30 01:08:03.962968 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:03.962624    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/076dfc74-4c57-46c9-946f-f06e3fb77f79-dshm\" (UniqueName: \"kubernetes.io/empty-dir/076dfc74-4c57-46c9-946f-f06e3fb77f79-dshm\") pod \"076dfc74-4c57-46c9-946f-f06e3fb77f79\" (UID: \"076dfc74-4c57-46c9-946f-f06e3fb77f79\") "
Oct 30 01:08:03.962968 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:03.962727    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/076dfc74-4c57-46c9-946f-f06e3fb77f79-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/076dfc74-4c57-46c9-946f-f06e3fb77f79-model-cache\") pod \"076dfc74-4c57-46c9-946f-f06e3fb77f79\" (UID: \"076dfc74-4c57-46c9-946f-f06e3fb77f79\") "
Oct 30 01:08:03.962968 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:03.962759    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/076dfc74-4c57-46c9-946f-f06e3fb77f79-tmp-dir" pod "076dfc74-4c57-46c9-946f-f06e3fb77f79" (UID: "076dfc74-4c57-46c9-946f-f06e3fb77f79"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:08:03.962968 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:03.962794    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/076dfc74-4c57-46c9-946f-f06e3fb77f79-kube-api-access-28gxr\" (UniqueName: \"kubernetes.io/projected/076dfc74-4c57-46c9-946f-f06e3fb77f79-kube-api-access-28gxr\") pod \"076dfc74-4c57-46c9-946f-f06e3fb77f79\" (UID: \"076dfc74-4c57-46c9-946f-f06e3fb77f79\") "
Oct 30 01:08:03.962968 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:03.962873    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/076dfc74-4c57-46c9-946f-f06e3fb77f79-home\" (UniqueName: \"kubernetes.io/empty-dir/076dfc74-4c57-46c9-946f-f06e3fb77f79-home\") pod \"076dfc74-4c57-46c9-946f-f06e3fb77f79\" (UID: \"076dfc74-4c57-46c9-946f-f06e3fb77f79\") "
Oct 30 01:08:03.962968 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:03.962962    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/076dfc74-4c57-46c9-946f-f06e3fb77f79-model-cache" pod "076dfc74-4c57-46c9-946f-f06e3fb77f79" (UID: "076dfc74-4c57-46c9-946f-f06e3fb77f79"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:08:03.963307 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:03.963165    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/076dfc74-4c57-46c9-946f-f06e3fb77f79-home" pod "076dfc74-4c57-46c9-946f-f06e3fb77f79" (UID: "076dfc74-4c57-46c9-946f-f06e3fb77f79"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:08:03.963541 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:03.963518    5002 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/076dfc74-4c57-46c9-946f-f06e3fb77f79-model-cache\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:08:03.963623 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:03.963545    5002 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/076dfc74-4c57-46c9-946f-f06e3fb77f79-home\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:08:03.963623 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:03.963558    5002 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/076dfc74-4c57-46c9-946f-f06e3fb77f79-tmp-dir\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:08:03.964799 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:03.964773    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/076dfc74-4c57-46c9-946f-f06e3fb77f79-tls-certs" pod "076dfc74-4c57-46c9-946f-f06e3fb77f79" (UID: "076dfc74-4c57-46c9-946f-f06e3fb77f79"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:08:03.965153 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:03.965131    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/076dfc74-4c57-46c9-946f-f06e3fb77f79-dshm" pod "076dfc74-4c57-46c9-946f-f06e3fb77f79" (UID: "076dfc74-4c57-46c9-946f-f06e3fb77f79"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:08:03.965941 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:03.965916    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/076dfc74-4c57-46c9-946f-f06e3fb77f79-kube-api-access-28gxr" pod "076dfc74-4c57-46c9-946f-f06e3fb77f79" (UID: "076dfc74-4c57-46c9-946f-f06e3fb77f79"). InnerVolumeSpecName "kube-api-access-28gxr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:08:04.065017 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:04.064985    5002 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/076dfc74-4c57-46c9-946f-f06e3fb77f79-tls-certs\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:08:04.065017 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:04.065012    5002 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/076dfc74-4c57-46c9-946f-f06e3fb77f79-dshm\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:08:04.065267 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:04.065034    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-28gxr\" (UniqueName: \"kubernetes.io/projected/076dfc74-4c57-46c9-946f-f06e3fb77f79-kube-api-access-28gxr\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:08:04.145071 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:04.144951    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-ef1084-66c68bf584-58x9z" podUID="fbe86602-c69c-4fff-b440-5e1ce9dd8e29" containerName="payload-processing" containerID="cri-o://c9975e41fb734905a892f2391dacea87e78d1d17f742f146a44d1b91646a06b6" gracePeriod=25
Oct 30 01:08:04.180694 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:04.180610    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ef1084-64c6bbc5ff-rrjq8" podUID="d405389a-c3e3-4aa9-a7db-ec4f32166b46" containerName="payload-pre-processing" containerID="cri-o://3674e1f28451e5276b398306dd73863073fdddf1e620e90074a493f0754b2013" gracePeriod=25
Oct 30 01:08:04.209534 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:04.209499    5002 generic.go:356] "Generic (PLEG): container finished" podID="076dfc74-4c57-46c9-946f-f06e3fb77f79" containerID="09f085c8afdf1111d2dc08b2cbc5d3cccc79dec9aad9ad2fcec4e30737e4a26d" exitCode=0
Oct 30 01:08:04.209706 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:04.209544    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-19bc68/test-model-19bc68-kserve-77fd689c7c-qw7lc"
Oct 30 01:08:04.209706 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:04.209585    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-19bc68/test-model-19bc68-kserve-77fd689c7c-qw7lc" event={"ID":"076dfc74-4c57-46c9-946f-f06e3fb77f79","Type":"ContainerDied","Data":"09f085c8afdf1111d2dc08b2cbc5d3cccc79dec9aad9ad2fcec4e30737e4a26d"}
Oct 30 01:08:04.209706 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:04.209629    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-19bc68/test-model-19bc68-kserve-77fd689c7c-qw7lc" event={"ID":"076dfc74-4c57-46c9-946f-f06e3fb77f79","Type":"ContainerDied","Data":"5d74017a1b369ab9c5e04a89fdd382eb3e9c66e5f07531c77d5847a96cd3e54b"}
Oct 30 01:08:04.209706 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:04.209647    5002 scope.go:122] "RemoveContainer" containerID="09f085c8afdf1111d2dc08b2cbc5d3cccc79dec9aad9ad2fcec4e30737e4a26d"
Oct 30 01:08:04.221034 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:04.221011    5002 scope.go:122] "RemoveContainer" containerID="09f085c8afdf1111d2dc08b2cbc5d3cccc79dec9aad9ad2fcec4e30737e4a26d"
Oct 30 01:08:04.221298 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:08:04.221278    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09f085c8afdf1111d2dc08b2cbc5d3cccc79dec9aad9ad2fcec4e30737e4a26d\": container with ID starting with 09f085c8afdf1111d2dc08b2cbc5d3cccc79dec9aad9ad2fcec4e30737e4a26d not found: ID does not exist" containerID="09f085c8afdf1111d2dc08b2cbc5d3cccc79dec9aad9ad2fcec4e30737e4a26d"
Oct 30 01:08:04.221356 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:04.221306    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09f085c8afdf1111d2dc08b2cbc5d3cccc79dec9aad9ad2fcec4e30737e4a26d"} err="failed to get container status \"09f085c8afdf1111d2dc08b2cbc5d3cccc79dec9aad9ad2fcec4e30737e4a26d\": rpc error: code = NotFound desc = could not find container \"09f085c8afdf1111d2dc08b2cbc5d3cccc79dec9aad9ad2fcec4e30737e4a26d\": container with ID starting with 09f085c8afdf1111d2dc08b2cbc5d3cccc79dec9aad9ad2fcec4e30737e4a26d not found: ID does not exist"
Oct 30 01:08:04.268895 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:04.268864    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-19bc68/test-model-19bc68-kserve-77fd689c7c-qw7lc"]
Oct 30 01:08:04.282756 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:04.282724    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-19bc68/test-model-19bc68-kserve-77fd689c7c-qw7lc"]
Oct 30 01:08:04.497873 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:04.497839    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="076dfc74-4c57-46c9-946f-f06e3fb77f79" path="/var/lib/kubelet/pods/076dfc74-4c57-46c9-946f-f06e3fb77f79/volumes"
Oct 30 01:08:04.831134 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:04.831043    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-19bc68/test-model-19bc68-kserve-77fd689c7c-qw7lc" podUID="076dfc74-4c57-46c9-946f-f06e3fb77f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.167:8000/ready\": context deadline exceeded"
Oct 30 01:08:05.226369 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:05.226313    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ef1084-64c6bbc5ff-rrjq8" podUID="d405389a-c3e3-4aa9-a7db-ec4f32166b46" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:08:05.226369 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:05.226360    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-ef1084-66c68bf584-58x9z" podUID="fbe86602-c69c-4fff-b440-5e1ce9dd8e29" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:08:09.080012 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:09.079969    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7d8z4"]
Oct 30 01:08:09.967829 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:09.967787    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w6-009e78/auth-test-model-009e78-kserve-75bf4b99d5-gxw7n"]
Oct 30 01:08:09.968807 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:09.968782    5002 cpu_manager.go:414] "containerMap: removing container" podUID="652cd481-aaf4-4f3d-8532-978fe17201c8" containerName="main"
Oct 30 01:08:09.968909 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:09.968811    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="652cd481-aaf4-4f3d-8532-978fe17201c8" containerName="main"
Oct 30 01:08:09.968909 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:09.968852    5002 cpu_manager.go:414] "containerMap: removing container" podUID="076dfc74-4c57-46c9-946f-f06e3fb77f79" containerName="main"
Oct 30 01:08:09.968909 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:09.968862    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="076dfc74-4c57-46c9-946f-f06e3fb77f79" containerName="main"
Oct 30 01:08:09.968909 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:09.968901    5002 cpu_manager.go:414] "containerMap: removing container" podUID="23d9111c-b9d3-4691-9545-0e0cf3689359" containerName="revoke-keys"
Oct 30 01:08:09.968909 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:09.968909    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23d9111c-b9d3-4691-9545-0e0cf3689359" containerName="revoke-keys"
Oct 30 01:08:09.969157 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:09.969084    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="076dfc74-4c57-46c9-946f-f06e3fb77f79" containerName="main"
Oct 30 01:08:09.969157 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:09.969106    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="23d9111c-b9d3-4691-9545-0e0cf3689359" containerName="revoke-keys"
Oct 30 01:08:09.969157 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:09.969124    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="652cd481-aaf4-4f3d-8532-978fe17201c8" containerName="main"
Oct 30 01:08:09.987730 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:09.987697    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w6-009e78/auth-test-model-009e78-kserve-75bf4b99d5-gxw7n"
Oct 30 01:08:10.003776 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:10.003735    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w6-009e78/auth-test-model-009e78-kserve-75bf4b99d5-gxw7n"]
Oct 30 01:08:10.023485 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:10.023446    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w6-009e78\"/\"openshift-service-ca.crt\""
Oct 30 01:08:10.023698 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:10.023562    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w6-009e78\"/\"default-dockercfg-v6wcl\""
Oct 30 01:08:10.023698 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:10.023602    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w6-009e78\"/\"kube-root-ca.crt\""
Oct 30 01:08:10.024033 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:10.024007    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w6-009e78\"/\"auth-test-model-009e78-kserve-self-signed-certs\""
Oct 30 01:08:10.099452 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:10.098441    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lnfs\" (UniqueName: \"kubernetes.io/projected/0df627c9-5e19-4e18-b250-a57bda637815-kube-api-access-7lnfs\") pod \"auth-test-model-009e78-kserve-75bf4b99d5-gxw7n\" (UID: \"0df627c9-5e19-4e18-b250-a57bda637815\") " pod="ai-tenant-e2e-shared-a-w6-009e78/auth-test-model-009e78-kserve-75bf4b99d5-gxw7n"
Oct 30 01:08:10.099452 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:10.098523    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0df627c9-5e19-4e18-b250-a57bda637815-tls-certs\") pod \"auth-test-model-009e78-kserve-75bf4b99d5-gxw7n\" (UID: \"0df627c9-5e19-4e18-b250-a57bda637815\") " pod="ai-tenant-e2e-shared-a-w6-009e78/auth-test-model-009e78-kserve-75bf4b99d5-gxw7n"
Oct 30 01:08:10.099452 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:10.098617    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0df627c9-5e19-4e18-b250-a57bda637815-model-cache\") pod \"auth-test-model-009e78-kserve-75bf4b99d5-gxw7n\" (UID: \"0df627c9-5e19-4e18-b250-a57bda637815\") " pod="ai-tenant-e2e-shared-a-w6-009e78/auth-test-model-009e78-kserve-75bf4b99d5-gxw7n"
Oct 30 01:08:10.099452 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:10.098740    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0df627c9-5e19-4e18-b250-a57bda637815-dshm\") pod \"auth-test-model-009e78-kserve-75bf4b99d5-gxw7n\" (UID: \"0df627c9-5e19-4e18-b250-a57bda637815\") " pod="ai-tenant-e2e-shared-a-w6-009e78/auth-test-model-009e78-kserve-75bf4b99d5-gxw7n"
Oct 30 01:08:10.099452 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:10.098821    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0df627c9-5e19-4e18-b250-a57bda637815-home\") pod \"auth-test-model-009e78-kserve-75bf4b99d5-gxw7n\" (UID: \"0df627c9-5e19-4e18-b250-a57bda637815\") " pod="ai-tenant-e2e-shared-a-w6-009e78/auth-test-model-009e78-kserve-75bf4b99d5-gxw7n"
Oct 30 01:08:10.099452 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:10.098883    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0df627c9-5e19-4e18-b250-a57bda637815-tmp-dir\") pod \"auth-test-model-009e78-kserve-75bf4b99d5-gxw7n\" (UID: \"0df627c9-5e19-4e18-b250-a57bda637815\") " pod="ai-tenant-e2e-shared-a-w6-009e78/auth-test-model-009e78-kserve-75bf4b99d5-gxw7n"
Oct 30 01:08:10.200610 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:10.200565    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lnfs\" (UniqueName: \"kubernetes.io/projected/0df627c9-5e19-4e18-b250-a57bda637815-kube-api-access-7lnfs\") pod \"auth-test-model-009e78-kserve-75bf4b99d5-gxw7n\" (UID: \"0df627c9-5e19-4e18-b250-a57bda637815\") " pod="ai-tenant-e2e-shared-a-w6-009e78/auth-test-model-009e78-kserve-75bf4b99d5-gxw7n"
Oct 30 01:08:10.200785 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:10.200657    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0df627c9-5e19-4e18-b250-a57bda637815-tls-certs\") pod \"auth-test-model-009e78-kserve-75bf4b99d5-gxw7n\" (UID: \"0df627c9-5e19-4e18-b250-a57bda637815\") " pod="ai-tenant-e2e-shared-a-w6-009e78/auth-test-model-009e78-kserve-75bf4b99d5-gxw7n"
Oct 30 01:08:10.200849 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:10.200767    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0df627c9-5e19-4e18-b250-a57bda637815-model-cache\") pod \"auth-test-model-009e78-kserve-75bf4b99d5-gxw7n\" (UID: \"0df627c9-5e19-4e18-b250-a57bda637815\") " pod="ai-tenant-e2e-shared-a-w6-009e78/auth-test-model-009e78-kserve-75bf4b99d5-gxw7n"
Oct 30 01:08:10.200849 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:10.200829    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0df627c9-5e19-4e18-b250-a57bda637815-dshm\") pod \"auth-test-model-009e78-kserve-75bf4b99d5-gxw7n\" (UID: \"0df627c9-5e19-4e18-b250-a57bda637815\") " pod="ai-tenant-e2e-shared-a-w6-009e78/auth-test-model-009e78-kserve-75bf4b99d5-gxw7n"
Oct 30 01:08:10.200980 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:10.200921    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0df627c9-5e19-4e18-b250-a57bda637815-home\") pod \"auth-test-model-009e78-kserve-75bf4b99d5-gxw7n\" (UID: \"0df627c9-5e19-4e18-b250-a57bda637815\") " pod="ai-tenant-e2e-shared-a-w6-009e78/auth-test-model-009e78-kserve-75bf4b99d5-gxw7n"
Oct 30 01:08:10.200980 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:10.200964    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0df627c9-5e19-4e18-b250-a57bda637815-tmp-dir\") pod \"auth-test-model-009e78-kserve-75bf4b99d5-gxw7n\" (UID: \"0df627c9-5e19-4e18-b250-a57bda637815\") " pod="ai-tenant-e2e-shared-a-w6-009e78/auth-test-model-009e78-kserve-75bf4b99d5-gxw7n"
Oct 30 01:08:10.201651 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:10.201619    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0df627c9-5e19-4e18-b250-a57bda637815-tmp-dir\") pod \"auth-test-model-009e78-kserve-75bf4b99d5-gxw7n\" (UID: \"0df627c9-5e19-4e18-b250-a57bda637815\") " pod="ai-tenant-e2e-shared-a-w6-009e78/auth-test-model-009e78-kserve-75bf4b99d5-gxw7n"
Oct 30 01:08:10.204521 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:10.203016    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0df627c9-5e19-4e18-b250-a57bda637815-home\") pod \"auth-test-model-009e78-kserve-75bf4b99d5-gxw7n\" (UID: \"0df627c9-5e19-4e18-b250-a57bda637815\") " pod="ai-tenant-e2e-shared-a-w6-009e78/auth-test-model-009e78-kserve-75bf4b99d5-gxw7n"
Oct 30 01:08:10.204521 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:10.203271    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0df627c9-5e19-4e18-b250-a57bda637815-model-cache\") pod \"auth-test-model-009e78-kserve-75bf4b99d5-gxw7n\" (UID: \"0df627c9-5e19-4e18-b250-a57bda637815\") " pod="ai-tenant-e2e-shared-a-w6-009e78/auth-test-model-009e78-kserve-75bf4b99d5-gxw7n"
Oct 30 01:08:10.226691 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:10.206106    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0df627c9-5e19-4e18-b250-a57bda637815-tls-certs\") pod \"auth-test-model-009e78-kserve-75bf4b99d5-gxw7n\" (UID: \"0df627c9-5e19-4e18-b250-a57bda637815\") " pod="ai-tenant-e2e-shared-a-w6-009e78/auth-test-model-009e78-kserve-75bf4b99d5-gxw7n"
Oct 30 01:08:10.226691 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:10.223758    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0df627c9-5e19-4e18-b250-a57bda637815-dshm\") pod \"auth-test-model-009e78-kserve-75bf4b99d5-gxw7n\" (UID: \"0df627c9-5e19-4e18-b250-a57bda637815\") " pod="ai-tenant-e2e-shared-a-w6-009e78/auth-test-model-009e78-kserve-75bf4b99d5-gxw7n"
Oct 30 01:08:10.241401 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:10.236087    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lnfs\" (UniqueName: \"kubernetes.io/projected/0df627c9-5e19-4e18-b250-a57bda637815-kube-api-access-7lnfs\") pod \"auth-test-model-009e78-kserve-75bf4b99d5-gxw7n\" (UID: \"0df627c9-5e19-4e18-b250-a57bda637815\") " pod="ai-tenant-e2e-shared-a-w6-009e78/auth-test-model-009e78-kserve-75bf4b99d5-gxw7n"
Oct 30 01:08:10.312884 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:10.312603    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w6-009e78/auth-test-model-009e78-kserve-75bf4b99d5-gxw7n"
Oct 30 01:08:11.003949 ip-10-0-128-7 kubenswrapper[5002]: W1030 01:08:11.003916    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0df627c9_5e19_4e18_b250_a57bda637815.slice/crio-f1f0f28ffc13f35fecd1421acff931ce5602343702948d3a4446dbfd1d2c4c95 WatchSource:0}: Error finding container f1f0f28ffc13f35fecd1421acff931ce5602343702948d3a4446dbfd1d2c4c95: Status 404 returned error can't find the container with id f1f0f28ffc13f35fecd1421acff931ce5602343702948d3a4446dbfd1d2c4c95
Oct 30 01:08:11.006749 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:11.006721    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w6-009e78/auth-test-model-009e78-kserve-75bf4b99d5-gxw7n"]
Oct 30 01:08:11.466834 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:11.466740    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w6-009e78/auth-test-model-009e78-kserve-75bf4b99d5-gxw7n" event={"ID":"0df627c9-5e19-4e18-b250-a57bda637815","Type":"ContainerStarted","Data":"ee11bf850384ae4fe0fa6e6a7ca73d678b6a914fa221b4580fc3a0d7a00aab6c"}
Oct 30 01:08:11.466834 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:11.466798    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w6-009e78/auth-test-model-009e78-kserve-75bf4b99d5-gxw7n" event={"ID":"0df627c9-5e19-4e18-b250-a57bda637815","Type":"ContainerStarted","Data":"f1f0f28ffc13f35fecd1421acff931ce5602343702948d3a4446dbfd1d2c4c95"}
Oct 30 01:08:11.511582 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:11.511530    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w6-009e78/auth-test-model-009e78-kserve-75bf4b99d5-gxw7n" podStartSLOduration=2.51151665 podStartE2EDuration="2.51151665s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:08:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:08:11.511095726 +0000 UTC m=+2187.127528632" watchObservedRunningTime="2026-10-30 01:08:11.51151665 +0000 UTC m=+2187.127949529"
Oct 30 01:08:12.316404 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:12.315458    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w6-009e78/auth-test-model-009e78-kserve-75bf4b99d5-gxw7n"
Oct 30 01:08:12.983915 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:08:12.983833    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbjdq" containerName="extractor"
Oct 30 01:08:15.235339 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:15.235296    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-ef1084-66c68bf584-58x9z" podUID="fbe86602-c69c-4fff-b440-5e1ce9dd8e29" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:08:15.236677 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:15.236645    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ef1084-64c6bbc5ff-rrjq8" podUID="d405389a-c3e3-4aa9-a7db-ec4f32166b46" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:08:16.929741 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:16.929706    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-plcp4"]
Oct 30 01:08:16.934294 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:16.934271    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-plcp4"
Oct 30 01:08:16.974027 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:16.973991    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-plcp4"]
Oct 30 01:08:17.021906 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:17.021867    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b369691-a360-4ec3-ac10-b55af864f470-workload-certs\") pod \"e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-plcp4\" (UID: \"3b369691-a360-4ec3-ac10-b55af864f470\") " pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-plcp4"
Oct 30 01:08:17.022087 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:17.022022    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b369691-a360-4ec3-ac10-b55af864f470-istio-podinfo\") pod \"e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-plcp4\" (UID: \"3b369691-a360-4ec3-ac10-b55af864f470\") " pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-plcp4"
Oct 30 01:08:17.022133 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:17.022121    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b369691-a360-4ec3-ac10-b55af864f470-istio-data\") pod \"e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-plcp4\" (UID: \"3b369691-a360-4ec3-ac10-b55af864f470\") " pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-plcp4"
Oct 30 01:08:17.022206 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:17.022187    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b369691-a360-4ec3-ac10-b55af864f470-istio-envoy\") pod \"e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-plcp4\" (UID: \"3b369691-a360-4ec3-ac10-b55af864f470\") " pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-plcp4"
Oct 30 01:08:17.022253 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:17.022241    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b369691-a360-4ec3-ac10-b55af864f470-istiod-ca-cert\") pod \"e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-plcp4\" (UID: \"3b369691-a360-4ec3-ac10-b55af864f470\") " pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-plcp4"
Oct 30 01:08:17.022336 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:17.022324    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b369691-a360-4ec3-ac10-b55af864f470-credential-socket\") pod \"e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-plcp4\" (UID: \"3b369691-a360-4ec3-ac10-b55af864f470\") " pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-plcp4"
Oct 30 01:08:17.022373 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:17.022359    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b369691-a360-4ec3-ac10-b55af864f470-istio-token\") pod \"e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-plcp4\" (UID: \"3b369691-a360-4ec3-ac10-b55af864f470\") " pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-plcp4"
Oct 30 01:08:17.022445 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:17.022434    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdh74\" (UniqueName: \"kubernetes.io/projected/3b369691-a360-4ec3-ac10-b55af864f470-kube-api-access-wdh74\") pod \"e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-plcp4\" (UID: \"3b369691-a360-4ec3-ac10-b55af864f470\") " pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-plcp4"
Oct 30 01:08:17.022481 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:17.022464    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b369691-a360-4ec3-ac10-b55af864f470-workload-socket\") pod \"e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-plcp4\" (UID: \"3b369691-a360-4ec3-ac10-b55af864f470\") " pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-plcp4"
Oct 30 01:08:17.123853 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:17.123812    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b369691-a360-4ec3-ac10-b55af864f470-istio-podinfo\") pod \"e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-plcp4\" (UID: \"3b369691-a360-4ec3-ac10-b55af864f470\") " pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-plcp4"
Oct 30 01:08:17.124055 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:17.123891    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b369691-a360-4ec3-ac10-b55af864f470-istio-data\") pod \"e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-plcp4\" (UID: \"3b369691-a360-4ec3-ac10-b55af864f470\") " pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-plcp4"
Oct 30 01:08:17.124055 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:17.123940    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b369691-a360-4ec3-ac10-b55af864f470-istio-envoy\") pod \"e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-plcp4\" (UID: \"3b369691-a360-4ec3-ac10-b55af864f470\") " pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-plcp4"
Oct 30 01:08:17.124055 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:17.123974    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b369691-a360-4ec3-ac10-b55af864f470-istiod-ca-cert\") pod \"e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-plcp4\" (UID: \"3b369691-a360-4ec3-ac10-b55af864f470\") " pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-plcp4"
Oct 30 01:08:17.124055 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:17.124004    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b369691-a360-4ec3-ac10-b55af864f470-credential-socket\") pod \"e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-plcp4\" (UID: \"3b369691-a360-4ec3-ac10-b55af864f470\") " pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-plcp4"
Oct 30 01:08:17.124055 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:17.124026    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b369691-a360-4ec3-ac10-b55af864f470-istio-token\") pod \"e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-plcp4\" (UID: \"3b369691-a360-4ec3-ac10-b55af864f470\") " pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-plcp4"
Oct 30 01:08:17.124055 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:17.124043    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdh74\" (UniqueName: \"kubernetes.io/projected/3b369691-a360-4ec3-ac10-b55af864f470-kube-api-access-wdh74\") pod \"e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-plcp4\" (UID: \"3b369691-a360-4ec3-ac10-b55af864f470\") " pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-plcp4"
Oct 30 01:08:17.124441 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:17.124072    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b369691-a360-4ec3-ac10-b55af864f470-workload-socket\") pod \"e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-plcp4\" (UID: \"3b369691-a360-4ec3-ac10-b55af864f470\") " pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-plcp4"
Oct 30 01:08:17.124441 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:17.124165    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b369691-a360-4ec3-ac10-b55af864f470-workload-certs\") pod \"e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-plcp4\" (UID: \"3b369691-a360-4ec3-ac10-b55af864f470\") " pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-plcp4"
Oct 30 01:08:17.124574 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:17.124476    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b369691-a360-4ec3-ac10-b55af864f470-istio-data\") pod \"e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-plcp4\" (UID: \"3b369691-a360-4ec3-ac10-b55af864f470\") " pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-plcp4"
Oct 30 01:08:17.124574 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:17.124562    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b369691-a360-4ec3-ac10-b55af864f470-workload-certs\") pod \"e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-plcp4\" (UID: \"3b369691-a360-4ec3-ac10-b55af864f470\") " pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-plcp4"
Oct 30 01:08:17.124687 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:17.124661    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b369691-a360-4ec3-ac10-b55af864f470-credential-socket\") pod \"e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-plcp4\" (UID: \"3b369691-a360-4ec3-ac10-b55af864f470\") " pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-plcp4"
Oct 30 01:08:17.124776 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:17.124755    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b369691-a360-4ec3-ac10-b55af864f470-workload-socket\") pod \"e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-plcp4\" (UID: \"3b369691-a360-4ec3-ac10-b55af864f470\") " pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-plcp4"
Oct 30 01:08:17.125346 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:17.125325    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b369691-a360-4ec3-ac10-b55af864f470-istiod-ca-cert\") pod \"e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-plcp4\" (UID: \"3b369691-a360-4ec3-ac10-b55af864f470\") " pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-plcp4"
Oct 30 01:08:17.126693 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:17.126665    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b369691-a360-4ec3-ac10-b55af864f470-istio-envoy\") pod \"e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-plcp4\" (UID: \"3b369691-a360-4ec3-ac10-b55af864f470\") " pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-plcp4"
Oct 30 01:08:17.126794 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:17.126701    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b369691-a360-4ec3-ac10-b55af864f470-istio-podinfo\") pod \"e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-plcp4\" (UID: \"3b369691-a360-4ec3-ac10-b55af864f470\") " pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-plcp4"
Oct 30 01:08:17.149661 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:17.149634    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdh74\" (UniqueName: \"kubernetes.io/projected/3b369691-a360-4ec3-ac10-b55af864f470-kube-api-access-wdh74\") pod \"e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-plcp4\" (UID: \"3b369691-a360-4ec3-ac10-b55af864f470\") " pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-plcp4"
Oct 30 01:08:17.149804 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:17.149781    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b369691-a360-4ec3-ac10-b55af864f470-istio-token\") pod \"e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-plcp4\" (UID: \"3b369691-a360-4ec3-ac10-b55af864f470\") " pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-plcp4"
Oct 30 01:08:17.249259 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:17.249218    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-plcp4"
Oct 30 01:08:17.579828 ip-10-0-128-7 kubenswrapper[5002]: W1030 01:08:17.579793    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b369691_a360_4ec3_ac10_b55af864f470.slice/crio-e6c052d4d4d1fceeead0ae63edb2066e42d5fe067b81d38974b66c56a7379dd6 WatchSource:0}: Error finding container e6c052d4d4d1fceeead0ae63edb2066e42d5fe067b81d38974b66c56a7379dd6: Status 404 returned error can't find the container with id e6c052d4d4d1fceeead0ae63edb2066e42d5fe067b81d38974b66c56a7379dd6
Oct 30 01:08:17.580298 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:17.580272    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-plcp4"]
Oct 30 01:08:17.582880 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:17.582846    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:08:17.582984 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:17.582927    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:08:17.582984 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:17.582976    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:08:18.278896 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:18.278858    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f51a915-9g29d"]
Oct 30 01:08:18.283425 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:18.283367    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f51a915-9g29d"
Oct 30 01:08:18.299940 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:18.299908    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f51a915-9g29d"]
Oct 30 01:08:18.308318 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:18.308223    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:08:18.308318 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:18.308274    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5phsh\""
Oct 30 01:08:18.308318 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:18.308296    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:08:18.313805 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:18.313750    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:08:18.366513 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:18.346471    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6sjt\" (UniqueName: \"kubernetes.io/projected/1ecc17fe-4d17-452c-9856-5ac462ffde03-kube-api-access-g6sjt\") pod \"maas-api-revoke-sub-5f51a915-9g29d\" (UID: \"1ecc17fe-4d17-452c-9856-5ac462ffde03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f51a915-9g29d"
Oct 30 01:08:18.366513 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:18.346584    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ecc17fe-4d17-452c-9856-5ac462ffde03-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5f51a915-9g29d\" (UID: \"1ecc17fe-4d17-452c-9856-5ac462ffde03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f51a915-9g29d"
Oct 30 01:08:18.366513 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:18.346805    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ecc17fe-4d17-452c-9856-5ac462ffde03-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5f51a915-9g29d\" (UID: \"1ecc17fe-4d17-452c-9856-5ac462ffde03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f51a915-9g29d"
Oct 30 01:08:18.448136 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:18.448096    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ecc17fe-4d17-452c-9856-5ac462ffde03-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5f51a915-9g29d\" (UID: \"1ecc17fe-4d17-452c-9856-5ac462ffde03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f51a915-9g29d"
Oct 30 01:08:18.448344 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:18.448315    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ecc17fe-4d17-452c-9856-5ac462ffde03-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5f51a915-9g29d\" (UID: \"1ecc17fe-4d17-452c-9856-5ac462ffde03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f51a915-9g29d"
Oct 30 01:08:18.448482 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:18.448455    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6sjt\" (UniqueName: \"kubernetes.io/projected/1ecc17fe-4d17-452c-9856-5ac462ffde03-kube-api-access-g6sjt\") pod \"maas-api-revoke-sub-5f51a915-9g29d\" (UID: \"1ecc17fe-4d17-452c-9856-5ac462ffde03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f51a915-9g29d"
Oct 30 01:08:18.448992 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:18.448967    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ecc17fe-4d17-452c-9856-5ac462ffde03-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5f51a915-9g29d\" (UID: \"1ecc17fe-4d17-452c-9856-5ac462ffde03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f51a915-9g29d"
Oct 30 01:08:18.477729 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:18.477692    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ecc17fe-4d17-452c-9856-5ac462ffde03-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5f51a915-9g29d\" (UID: \"1ecc17fe-4d17-452c-9856-5ac462ffde03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f51a915-9g29d"
Oct 30 01:08:18.480693 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:18.480649    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6sjt\" (UniqueName: \"kubernetes.io/projected/1ecc17fe-4d17-452c-9856-5ac462ffde03-kube-api-access-g6sjt\") pod \"maas-api-revoke-sub-5f51a915-9g29d\" (UID: \"1ecc17fe-4d17-452c-9856-5ac462ffde03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f51a915-9g29d"
Oct 30 01:08:18.577691 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:18.577596    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-plcp4" event={"ID":"3b369691-a360-4ec3-ac10-b55af864f470","Type":"ContainerStarted","Data":"e8c759c36a821ad9cfc29a7edf354f13c5b03ea0d7ffebe4ace4d2ab16c0f3ed"}
Oct 30 01:08:18.577691 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:18.577653    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-plcp4" event={"ID":"3b369691-a360-4ec3-ac10-b55af864f470","Type":"ContainerStarted","Data":"e6c052d4d4d1fceeead0ae63edb2066e42d5fe067b81d38974b66c56a7379dd6"}
Oct 30 01:08:18.601685 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:18.601657    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f51a915-9g29d"
Oct 30 01:08:18.627349 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:18.627299    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-plcp4" podStartSLOduration=2.627284989 podStartE2EDuration="2.627284989s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:08:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:08:18.625663871 +0000 UTC m=+2194.242096766" watchObservedRunningTime="2026-10-30 01:08:18.627284989 +0000 UTC m=+2194.243717868"
Oct 30 01:08:18.930990 ip-10-0-128-7 kubenswrapper[5002]: W1030 01:08:18.930961    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ecc17fe_4d17_452c_9856_5ac462ffde03.slice/crio-e922999110dfc4060323c94cd92661c39064ede5526e18a98aa935b9f1f5a36e WatchSource:0}: Error finding container e922999110dfc4060323c94cd92661c39064ede5526e18a98aa935b9f1f5a36e: Status 404 returned error can't find the container with id e922999110dfc4060323c94cd92661c39064ede5526e18a98aa935b9f1f5a36e
Oct 30 01:08:18.932361 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:18.932329    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f51a915-9g29d"]
Oct 30 01:08:19.031335 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:19.031305    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bafb00a1-hw4v7"]
Oct 30 01:08:19.035038 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:19.035012    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bafb00a1-hw4v7"]
Oct 30 01:08:19.252251 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:19.252216    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-plcp4"
Oct 30 01:08:19.256597 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:19.256560    5002 patch_prober.go:29] interesting pod/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-plcp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body=
Oct 30 01:08:19.256959 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:19.256924    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-plcp4" podUID="3b369691-a360-4ec3-ac10-b55af864f470" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused"
Oct 30 01:08:19.587922 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:19.587825    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f51a915-9g29d" event={"ID":"1ecc17fe-4d17-452c-9856-5ac462ffde03","Type":"ContainerStarted","Data":"f29efdc56bf33c25c25df51d44201cf57e77b2c01aeadce7f002e07ac4607714"}
Oct 30 01:08:19.587922 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:19.587890    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f51a915-9g29d" event={"ID":"1ecc17fe-4d17-452c-9856-5ac462ffde03","Type":"ContainerStarted","Data":"e922999110dfc4060323c94cd92661c39064ede5526e18a98aa935b9f1f5a36e"}
Oct 30 01:08:19.637761 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:19.637713    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f51a915-9g29d" podStartSLOduration=1.637698318 podStartE2EDuration="1.637698318s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:08:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:08:19.634134807 +0000 UTC m=+2195.250567686" watchObservedRunningTime="2026-10-30 01:08:19.637698318 +0000 UTC m=+2195.254131198"
Oct 30 01:08:20.263408 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:20.249883    5002 patch_prober.go:29] interesting pod/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-plcp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body=
Oct 30 01:08:20.263408 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:20.249948    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-plcp4" podUID="3b369691-a360-4ec3-ac10-b55af864f470" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused"
Oct 30 01:08:20.286966 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:20.286239    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-x66jp"]
Oct 30 01:08:20.394404 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:20.391606    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w6-009e78/auth-test-model-009e78-kserve-75bf4b99d5-gxw7n"
Oct 30 01:08:20.394404 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:20.391683    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-x66jp"]
Oct 30 01:08:20.431850 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:20.394884    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-x66jp"
Oct 30 01:08:20.453448 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:20.440451    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w5-93079c-openshift-default-dockercfg-9g4x2\""
Oct 30 01:08:20.453448 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:20.445470    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w6-009e78/auth-test-model-009e78-kserve-75bf4b99d5-gxw7n"
Oct 30 01:08:20.568401 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:20.515000    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37340929-9f26-4db9-ad7d-66d0bd14a7cd-istiod-ca-cert\") pod \"e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-x66jp\" (UID: \"37340929-9f26-4db9-ad7d-66d0bd14a7cd\") " pod="openshift-ingress/e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-x66jp"
Oct 30 01:08:20.568401 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:20.515053    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37340929-9f26-4db9-ad7d-66d0bd14a7cd-istio-data\") pod \"e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-x66jp\" (UID: \"37340929-9f26-4db9-ad7d-66d0bd14a7cd\") " pod="openshift-ingress/e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-x66jp"
Oct 30 01:08:20.568401 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:20.515113    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbx6c\" (UniqueName: \"kubernetes.io/projected/37340929-9f26-4db9-ad7d-66d0bd14a7cd-kube-api-access-lbx6c\") pod \"e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-x66jp\" (UID: \"37340929-9f26-4db9-ad7d-66d0bd14a7cd\") " pod="openshift-ingress/e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-x66jp"
Oct 30 01:08:20.568401 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:20.515143    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37340929-9f26-4db9-ad7d-66d0bd14a7cd-istio-token\") pod \"e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-x66jp\" (UID: \"37340929-9f26-4db9-ad7d-66d0bd14a7cd\") " pod="openshift-ingress/e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-x66jp"
Oct 30 01:08:20.568401 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:20.515200    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37340929-9f26-4db9-ad7d-66d0bd14a7cd-credential-socket\") pod \"e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-x66jp\" (UID: \"37340929-9f26-4db9-ad7d-66d0bd14a7cd\") " pod="openshift-ingress/e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-x66jp"
Oct 30 01:08:20.568401 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:20.515227    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37340929-9f26-4db9-ad7d-66d0bd14a7cd-istio-envoy\") pod \"e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-x66jp\" (UID: \"37340929-9f26-4db9-ad7d-66d0bd14a7cd\") " pod="openshift-ingress/e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-x66jp"
Oct 30 01:08:20.568401 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:20.515422    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37340929-9f26-4db9-ad7d-66d0bd14a7cd-istio-podinfo\") pod \"e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-x66jp\" (UID: \"37340929-9f26-4db9-ad7d-66d0bd14a7cd\") " pod="openshift-ingress/e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-x66jp"
Oct 30 01:08:20.568401 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:20.515471    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37340929-9f26-4db9-ad7d-66d0bd14a7cd-workload-socket\") pod \"e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-x66jp\" (UID: \"37340929-9f26-4db9-ad7d-66d0bd14a7cd\") " pod="openshift-ingress/e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-x66jp"
Oct 30 01:08:20.568401 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:20.515525    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37340929-9f26-4db9-ad7d-66d0bd14a7cd-workload-certs\") pod \"e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-x66jp\" (UID: \"37340929-9f26-4db9-ad7d-66d0bd14a7cd\") " pod="openshift-ingress/e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-x66jp"
Oct 30 01:08:20.618512 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:20.618472    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37340929-9f26-4db9-ad7d-66d0bd14a7cd-workload-certs\") pod \"e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-x66jp\" (UID: \"37340929-9f26-4db9-ad7d-66d0bd14a7cd\") " pod="openshift-ingress/e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-x66jp"
Oct 30 01:08:20.619003 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:20.618687    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37340929-9f26-4db9-ad7d-66d0bd14a7cd-istiod-ca-cert\") pod \"e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-x66jp\" (UID: \"37340929-9f26-4db9-ad7d-66d0bd14a7cd\") " pod="openshift-ingress/e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-x66jp"
Oct 30 01:08:20.619003 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:20.618730    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37340929-9f26-4db9-ad7d-66d0bd14a7cd-istio-data\") pod \"e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-x66jp\" (UID: \"37340929-9f26-4db9-ad7d-66d0bd14a7cd\") " pod="openshift-ingress/e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-x66jp"
Oct 30 01:08:20.619003 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:20.618780    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbx6c\" (UniqueName: \"kubernetes.io/projected/37340929-9f26-4db9-ad7d-66d0bd14a7cd-kube-api-access-lbx6c\") pod \"e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-x66jp\" (UID: \"37340929-9f26-4db9-ad7d-66d0bd14a7cd\") " pod="openshift-ingress/e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-x66jp"
Oct 30 01:08:20.619003 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:20.618809    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37340929-9f26-4db9-ad7d-66d0bd14a7cd-istio-token\") pod \"e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-x66jp\" (UID: \"37340929-9f26-4db9-ad7d-66d0bd14a7cd\") " pod="openshift-ingress/e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-x66jp"
Oct 30 01:08:20.619003 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:20.618906    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37340929-9f26-4db9-ad7d-66d0bd14a7cd-credential-socket\") pod \"e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-x66jp\" (UID: \"37340929-9f26-4db9-ad7d-66d0bd14a7cd\") " pod="openshift-ingress/e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-x66jp"
Oct 30 01:08:20.619003 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:20.618940    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37340929-9f26-4db9-ad7d-66d0bd14a7cd-istio-envoy\") pod \"e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-x66jp\" (UID: \"37340929-9f26-4db9-ad7d-66d0bd14a7cd\") " pod="openshift-ingress/e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-x66jp"
Oct 30 01:08:20.619316 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:20.619159    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37340929-9f26-4db9-ad7d-66d0bd14a7cd-istio-podinfo\") pod \"e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-x66jp\" (UID: \"37340929-9f26-4db9-ad7d-66d0bd14a7cd\") " pod="openshift-ingress/e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-x66jp"
Oct 30 01:08:20.619316 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:20.619221    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37340929-9f26-4db9-ad7d-66d0bd14a7cd-workload-socket\") pod \"e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-x66jp\" (UID: \"37340929-9f26-4db9-ad7d-66d0bd14a7cd\") " pod="openshift-ingress/e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-x66jp"
Oct 30 01:08:20.621177 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:20.621150    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37340929-9f26-4db9-ad7d-66d0bd14a7cd-workload-socket\") pod \"e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-x66jp\" (UID: \"37340929-9f26-4db9-ad7d-66d0bd14a7cd\") " pod="openshift-ingress/e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-x66jp"
Oct 30 01:08:20.621525 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:20.621502    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37340929-9f26-4db9-ad7d-66d0bd14a7cd-workload-certs\") pod \"e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-x66jp\" (UID: \"37340929-9f26-4db9-ad7d-66d0bd14a7cd\") " pod="openshift-ingress/e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-x66jp"
Oct 30 01:08:20.623464 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:20.623443    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37340929-9f26-4db9-ad7d-66d0bd14a7cd-istiod-ca-cert\") pod \"e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-x66jp\" (UID: \"37340929-9f26-4db9-ad7d-66d0bd14a7cd\") " pod="openshift-ingress/e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-x66jp"
Oct 30 01:08:20.627455 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:20.623734    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37340929-9f26-4db9-ad7d-66d0bd14a7cd-istio-data\") pod \"e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-x66jp\" (UID: \"37340929-9f26-4db9-ad7d-66d0bd14a7cd\") " pod="openshift-ingress/e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-x66jp"
Oct 30 01:08:20.629055 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:20.629024    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37340929-9f26-4db9-ad7d-66d0bd14a7cd-credential-socket\") pod \"e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-x66jp\" (UID: \"37340929-9f26-4db9-ad7d-66d0bd14a7cd\") " pod="openshift-ingress/e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-x66jp"
Oct 30 01:08:20.641416 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:20.639209    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37340929-9f26-4db9-ad7d-66d0bd14a7cd-istio-podinfo\") pod \"e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-x66jp\" (UID: \"37340929-9f26-4db9-ad7d-66d0bd14a7cd\") " pod="openshift-ingress/e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-x66jp"
Oct 30 01:08:20.642675 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:20.642647    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37340929-9f26-4db9-ad7d-66d0bd14a7cd-istio-envoy\") pod \"e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-x66jp\" (UID: \"37340929-9f26-4db9-ad7d-66d0bd14a7cd\") " pod="openshift-ingress/e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-x66jp"
Oct 30 01:08:20.647166 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:20.647141    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8cef991-175d-4b30-8d0b-33f05e92a992" path="/var/lib/kubelet/pods/a8cef991-175d-4b30-8d0b-33f05e92a992/volumes"
Oct 30 01:08:20.659442 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:20.654939    5002 generic.go:356] "Generic (PLEG): container finished" podID="1ecc17fe-4d17-452c-9856-5ac462ffde03" containerID="f29efdc56bf33c25c25df51d44201cf57e77b2c01aeadce7f002e07ac4607714" exitCode=0
Oct 30 01:08:20.659442 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:20.656125    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f51a915-9g29d" event={"ID":"1ecc17fe-4d17-452c-9856-5ac462ffde03","Type":"ContainerDied","Data":"f29efdc56bf33c25c25df51d44201cf57e77b2c01aeadce7f002e07ac4607714"}
Oct 30 01:08:20.659442 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:20.659019    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37340929-9f26-4db9-ad7d-66d0bd14a7cd-istio-token\") pod \"e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-x66jp\" (UID: \"37340929-9f26-4db9-ad7d-66d0bd14a7cd\") " pod="openshift-ingress/e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-x66jp"
Oct 30 01:08:20.682477 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:20.679183    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w6-009e78/auth-test-model-009e78-kserve-75bf4b99d5-gxw7n"
Oct 30 01:08:20.694070 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:20.693902    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbx6c\" (UniqueName: \"kubernetes.io/projected/37340929-9f26-4db9-ad7d-66d0bd14a7cd-kube-api-access-lbx6c\") pod \"e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-x66jp\" (UID: \"37340929-9f26-4db9-ad7d-66d0bd14a7cd\") " pod="openshift-ingress/e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-x66jp"
Oct 30 01:08:20.944066 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:20.943968    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-x66jp"
Oct 30 01:08:21.255467 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:21.250834    5002 patch_prober.go:29] interesting pod/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-plcp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body=
Oct 30 01:08:21.255467 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:21.250904    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-plcp4" podUID="3b369691-a360-4ec3-ac10-b55af864f470" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused"
Oct 30 01:08:21.808808 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:21.808070    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ef1084-f6m5q"]
Oct 30 01:08:21.820876 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:21.820850    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ef1084-f6m5q"]
Oct 30 01:08:22.106919 ip-10-0-128-7 kubenswrapper[5002]: W1030 01:08:22.106872    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37340929_9f26_4db9_ad7d_66d0bd14a7cd.slice/crio-cd1c33552899d0f1ef5bf878847a2befa2d20871324353534a73c5417bfb1aa4 WatchSource:0}: Error finding container cd1c33552899d0f1ef5bf878847a2befa2d20871324353534a73c5417bfb1aa4: Status 404 returned error can't find the container with id cd1c33552899d0f1ef5bf878847a2befa2d20871324353534a73c5417bfb1aa4
Oct 30 01:08:22.109893 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:22.109855    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:08:22.110018 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:22.109930    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:08:22.110018 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:22.109960    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:08:22.117349 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:22.116895    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-x66jp"]
Oct 30 01:08:22.194401 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:22.193677    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f51a915-9g29d"
Oct 30 01:08:22.252829 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:22.252723    5002 patch_prober.go:29] interesting pod/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-plcp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body=
Oct 30 01:08:22.252829 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:22.252776    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-plcp4" podUID="3b369691-a360-4ec3-ac10-b55af864f470" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused"
Oct 30 01:08:22.323031 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:22.322991    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1ecc17fe-4d17-452c-9856-5ac462ffde03-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ecc17fe-4d17-452c-9856-5ac462ffde03-maas-api-xks-ca\") pod \"1ecc17fe-4d17-452c-9856-5ac462ffde03\" (UID: \"1ecc17fe-4d17-452c-9856-5ac462ffde03\") "
Oct 30 01:08:22.323220 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:22.323055    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ecc17fe-4d17-452c-9856-5ac462ffde03-kube-api-access-g6sjt\" (UniqueName: \"kubernetes.io/projected/1ecc17fe-4d17-452c-9856-5ac462ffde03-kube-api-access-g6sjt\") pod \"1ecc17fe-4d17-452c-9856-5ac462ffde03\" (UID: \"1ecc17fe-4d17-452c-9856-5ac462ffde03\") "
Oct 30 01:08:22.323220 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:22.323075    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ecc17fe-4d17-452c-9856-5ac462ffde03-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ecc17fe-4d17-452c-9856-5ac462ffde03-maas-api-openshift-ca\") pod \"1ecc17fe-4d17-452c-9856-5ac462ffde03\" (UID: \"1ecc17fe-4d17-452c-9856-5ac462ffde03\") "
Oct 30 01:08:22.324074 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:22.324046    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ecc17fe-4d17-452c-9856-5ac462ffde03-maas-api-openshift-ca" pod "1ecc17fe-4d17-452c-9856-5ac462ffde03" (UID: "1ecc17fe-4d17-452c-9856-5ac462ffde03"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:08:22.327845 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:22.327816    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ecc17fe-4d17-452c-9856-5ac462ffde03-maas-api-xks-ca" pod "1ecc17fe-4d17-452c-9856-5ac462ffde03" (UID: "1ecc17fe-4d17-452c-9856-5ac462ffde03"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:08:22.328877 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:22.328849    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ecc17fe-4d17-452c-9856-5ac462ffde03-kube-api-access-g6sjt" pod "1ecc17fe-4d17-452c-9856-5ac462ffde03" (UID: "1ecc17fe-4d17-452c-9856-5ac462ffde03"). InnerVolumeSpecName "kube-api-access-g6sjt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:08:22.428373 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:22.427609    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ecc17fe-4d17-452c-9856-5ac462ffde03-maas-api-xks-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:08:22.428373 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:22.427650    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g6sjt\" (UniqueName: \"kubernetes.io/projected/1ecc17fe-4d17-452c-9856-5ac462ffde03-kube-api-access-g6sjt\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:08:22.428373 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:22.427666    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ecc17fe-4d17-452c-9856-5ac462ffde03-maas-api-openshift-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:08:22.519853 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:22.519812    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23d9111c-b9d3-4691-9545-0e0cf3689359" path="/var/lib/kubelet/pods/23d9111c-b9d3-4691-9545-0e0cf3689359/volumes"
Oct 30 01:08:22.825056 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:22.824963    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f51a915-9g29d"
Oct 30 01:08:22.825056 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:22.824991    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f51a915-9g29d" event={"ID":"1ecc17fe-4d17-452c-9856-5ac462ffde03","Type":"ContainerDied","Data":"e922999110dfc4060323c94cd92661c39064ede5526e18a98aa935b9f1f5a36e"}
Oct 30 01:08:22.825056 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:22.825052    5002 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e922999110dfc4060323c94cd92661c39064ede5526e18a98aa935b9f1f5a36e"
Oct 30 01:08:22.827119 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:22.827087    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-x66jp" event={"ID":"37340929-9f26-4db9-ad7d-66d0bd14a7cd","Type":"ContainerStarted","Data":"4e3d340fe1ad2d5300d74d0c655c804d66f0fd0a4e3a8c9b07a631aa16d34830"}
Oct 30 01:08:22.827260 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:22.827141    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-x66jp" event={"ID":"37340929-9f26-4db9-ad7d-66d0bd14a7cd","Type":"ContainerStarted","Data":"cd1c33552899d0f1ef5bf878847a2befa2d20871324353534a73c5417bfb1aa4"}
Oct 30 01:08:22.878401 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:22.878256    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-x66jp" podStartSLOduration=2.87823407 podStartE2EDuration="2.87823407s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:08:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:08:22.875154506 +0000 UTC m=+2198.491587388" watchObservedRunningTime="2026-10-30 01:08:22.87823407 +0000 UTC m=+2198.494666952"
Oct 30 01:08:22.944761 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:22.944638    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-x66jp"
Oct 30 01:08:23.256410 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:23.253767    5002 patch_prober.go:29] interesting pod/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-plcp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body=
Oct 30 01:08:23.256410 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:23.253832    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-plcp4" podUID="3b369691-a360-4ec3-ac10-b55af864f470" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused"
Oct 30 01:08:23.945850 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:23.945794    5002 patch_prober.go:29] interesting pod/e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-x66jp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" start-of-body=
Oct 30 01:08:23.946348 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:23.945870    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-x66jp" podUID="37340929-9f26-4db9-ad7d-66d0bd14a7cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused"
Oct 30 01:08:24.257817 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:24.257778    5002 patch_prober.go:29] interesting pod/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-plcp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body=
Oct 30 01:08:24.258085 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:24.258057    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-plcp4" podUID="3b369691-a360-4ec3-ac10-b55af864f470" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused"
Oct 30 01:08:24.631226 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:24.631191    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-ef1084-66c68bf584-58x9z"
Oct 30 01:08:24.714523 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:24.714483    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbe86602-c69c-4fff-b440-5e1ce9dd8e29-kube-api-access-bd9ph\" (UniqueName: \"kubernetes.io/projected/fbe86602-c69c-4fff-b440-5e1ce9dd8e29-kube-api-access-bd9ph\") pod \"fbe86602-c69c-4fff-b440-5e1ce9dd8e29\" (UID: \"fbe86602-c69c-4fff-b440-5e1ce9dd8e29\") "
Oct 30 01:08:24.714958 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:24.714934    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fbe86602-c69c-4fff-b440-5e1ce9dd8e29-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbe86602-c69c-4fff-b440-5e1ce9dd8e29-plugins-config-volume\") pod \"fbe86602-c69c-4fff-b440-5e1ce9dd8e29\" (UID: \"fbe86602-c69c-4fff-b440-5e1ce9dd8e29\") "
Oct 30 01:08:24.715362 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:24.715334    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbe86602-c69c-4fff-b440-5e1ce9dd8e29-plugins-config-volume" pod "fbe86602-c69c-4fff-b440-5e1ce9dd8e29" (UID: "fbe86602-c69c-4fff-b440-5e1ce9dd8e29"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:08:24.715833 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:24.715808    5002 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbe86602-c69c-4fff-b440-5e1ce9dd8e29-plugins-config-volume\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:08:24.718001 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:24.717974    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbe86602-c69c-4fff-b440-5e1ce9dd8e29-kube-api-access-bd9ph" pod "fbe86602-c69c-4fff-b440-5e1ce9dd8e29" (UID: "fbe86602-c69c-4fff-b440-5e1ce9dd8e29"). InnerVolumeSpecName "kube-api-access-bd9ph". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:08:24.803763 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:24.803735    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ef1084-64c6bbc5ff-rrjq8"
Oct 30 01:08:24.816404 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:24.816353    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d405389a-c3e3-4aa9-a7db-ec4f32166b46-kube-api-access-xdpfz\" (UniqueName: \"kubernetes.io/projected/d405389a-c3e3-4aa9-a7db-ec4f32166b46-kube-api-access-xdpfz\") pod \"d405389a-c3e3-4aa9-a7db-ec4f32166b46\" (UID: \"d405389a-c3e3-4aa9-a7db-ec4f32166b46\") "
Oct 30 01:08:24.816567 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:24.816445    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d405389a-c3e3-4aa9-a7db-ec4f32166b46-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d405389a-c3e3-4aa9-a7db-ec4f32166b46-plugins-config-volume\") pod \"d405389a-c3e3-4aa9-a7db-ec4f32166b46\" (UID: \"d405389a-c3e3-4aa9-a7db-ec4f32166b46\") "
Oct 30 01:08:24.816949 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:24.816927    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bd9ph\" (UniqueName: \"kubernetes.io/projected/fbe86602-c69c-4fff-b440-5e1ce9dd8e29-kube-api-access-bd9ph\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:08:24.817249 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:24.817229    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d405389a-c3e3-4aa9-a7db-ec4f32166b46-plugins-config-volume" pod "d405389a-c3e3-4aa9-a7db-ec4f32166b46" (UID: "d405389a-c3e3-4aa9-a7db-ec4f32166b46"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:08:24.819316 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:24.819265    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d405389a-c3e3-4aa9-a7db-ec4f32166b46-kube-api-access-xdpfz" pod "d405389a-c3e3-4aa9-a7db-ec4f32166b46" (UID: "d405389a-c3e3-4aa9-a7db-ec4f32166b46"). InnerVolumeSpecName "kube-api-access-xdpfz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:08:24.858407 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:24.858354    5002 generic.go:356] "Generic (PLEG): container finished" podID="fbe86602-c69c-4fff-b440-5e1ce9dd8e29" containerID="c9975e41fb734905a892f2391dacea87e78d1d17f742f146a44d1b91646a06b6" exitCode=0
Oct 30 01:08:24.867421 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:24.860270    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-ef1084-66c68bf584-58x9z" event={"ID":"fbe86602-c69c-4fff-b440-5e1ce9dd8e29","Type":"ContainerDied","Data":"c9975e41fb734905a892f2391dacea87e78d1d17f742f146a44d1b91646a06b6"}
Oct 30 01:08:24.867421 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:24.860393    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-ef1084-66c68bf584-58x9z" event={"ID":"fbe86602-c69c-4fff-b440-5e1ce9dd8e29","Type":"ContainerDied","Data":"892943be9b7d98e1085b9f6485ef884d7bbc6eff4fb195da5ac492c00a0395cc"}
Oct 30 01:08:24.867421 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:24.860432    5002 scope.go:122] "RemoveContainer" containerID="c9975e41fb734905a892f2391dacea87e78d1d17f742f146a44d1b91646a06b6"
Oct 30 01:08:24.867421 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:24.860592    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-ef1084-66c68bf584-58x9z"
Oct 30 01:08:24.923455 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:24.918188    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xdpfz\" (UniqueName: \"kubernetes.io/projected/d405389a-c3e3-4aa9-a7db-ec4f32166b46-kube-api-access-xdpfz\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:08:24.923455 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:24.918224    5002 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d405389a-c3e3-4aa9-a7db-ec4f32166b46-plugins-config-volume\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:08:24.930158 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:24.930121    5002 generic.go:356] "Generic (PLEG): container finished" podID="d405389a-c3e3-4aa9-a7db-ec4f32166b46" containerID="3674e1f28451e5276b398306dd73863073fdddf1e620e90074a493f0754b2013" exitCode=0
Oct 30 01:08:24.931007 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:24.930980    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ef1084-64c6bbc5ff-rrjq8"
Oct 30 01:08:24.931229 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:24.931200    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ef1084-64c6bbc5ff-rrjq8" event={"ID":"d405389a-c3e3-4aa9-a7db-ec4f32166b46","Type":"ContainerDied","Data":"3674e1f28451e5276b398306dd73863073fdddf1e620e90074a493f0754b2013"}
Oct 30 01:08:24.931299 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:24.931270    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ef1084-64c6bbc5ff-rrjq8" event={"ID":"d405389a-c3e3-4aa9-a7db-ec4f32166b46","Type":"ContainerDied","Data":"299840d781d8a8468828ef6861986fb0af5e02f7b5c7599a8b8560274a72cf1b"}
Oct 30 01:08:24.944517 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:24.944481    5002 patch_prober.go:29] interesting pod/e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-x66jp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" start-of-body=
Oct 30 01:08:24.944646 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:24.944551    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-x66jp" podUID="37340929-9f26-4db9-ad7d-66d0bd14a7cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused"
Oct 30 01:08:24.977564 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:24.976730    5002 scope.go:122] "RemoveContainer" containerID="c9975e41fb734905a892f2391dacea87e78d1d17f742f146a44d1b91646a06b6"
Oct 30 01:08:25.019643 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:08:24.989138    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9975e41fb734905a892f2391dacea87e78d1d17f742f146a44d1b91646a06b6\": container with ID starting with c9975e41fb734905a892f2391dacea87e78d1d17f742f146a44d1b91646a06b6 not found: ID does not exist" containerID="c9975e41fb734905a892f2391dacea87e78d1d17f742f146a44d1b91646a06b6"
Oct 30 01:08:25.019643 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:24.989188    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9975e41fb734905a892f2391dacea87e78d1d17f742f146a44d1b91646a06b6"} err="failed to get container status \"c9975e41fb734905a892f2391dacea87e78d1d17f742f146a44d1b91646a06b6\": rpc error: code = NotFound desc = could not find container \"c9975e41fb734905a892f2391dacea87e78d1d17f742f146a44d1b91646a06b6\": container with ID starting with c9975e41fb734905a892f2391dacea87e78d1d17f742f146a44d1b91646a06b6 not found: ID does not exist"
Oct 30 01:08:25.019643 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:24.989215    5002 scope.go:122] "RemoveContainer" containerID="3674e1f28451e5276b398306dd73863073fdddf1e620e90074a493f0754b2013"
Oct 30 01:08:25.073443 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:25.062472    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ef1084-66c68bf584-58x9z"]
Oct 30 01:08:25.150507 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:25.149161    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ef1084-66c68bf584-58x9z"]
Oct 30 01:08:25.244636 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:25.244600    5002 scope.go:122] "RemoveContainer" containerID="3674e1f28451e5276b398306dd73863073fdddf1e620e90074a493f0754b2013"
Oct 30 01:08:25.248705 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:25.248677    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ef1084-64c6bbc5ff-rrjq8"]
Oct 30 01:08:25.248854 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:25.248844    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ef1084-64c6bbc5ff-rrjq8"]
Oct 30 01:08:25.260435 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:08:25.256547    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3674e1f28451e5276b398306dd73863073fdddf1e620e90074a493f0754b2013\": container with ID starting with 3674e1f28451e5276b398306dd73863073fdddf1e620e90074a493f0754b2013 not found: ID does not exist" containerID="3674e1f28451e5276b398306dd73863073fdddf1e620e90074a493f0754b2013"
Oct 30 01:08:25.260435 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:25.256588    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3674e1f28451e5276b398306dd73863073fdddf1e620e90074a493f0754b2013"} err="failed to get container status \"3674e1f28451e5276b398306dd73863073fdddf1e620e90074a493f0754b2013\": rpc error: code = NotFound desc = could not find container \"3674e1f28451e5276b398306dd73863073fdddf1e620e90074a493f0754b2013\": container with ID starting with 3674e1f28451e5276b398306dd73863073fdddf1e620e90074a493f0754b2013 not found: ID does not exist"
Oct 30 01:08:25.281711 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:25.277655    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-plcp4"
Oct 30 01:08:25.281711 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:25.279194    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-plcp4"
Oct 30 01:08:25.295683 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:25.295654    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-plcp4"
Oct 30 01:08:25.945164 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:25.945077    5002 patch_prober.go:29] interesting pod/e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-x66jp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" start-of-body=
Oct 30 01:08:25.945164 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:25.945139    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-x66jp" podUID="37340929-9f26-4db9-ad7d-66d0bd14a7cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused"
Oct 30 01:08:26.498038 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:26.498001    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d405389a-c3e3-4aa9-a7db-ec4f32166b46" path="/var/lib/kubelet/pods/d405389a-c3e3-4aa9-a7db-ec4f32166b46/volumes"
Oct 30 01:08:26.498738 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:26.498715    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fbe86602-c69c-4fff-b440-5e1ce9dd8e29" path="/var/lib/kubelet/pods/fbe86602-c69c-4fff-b440-5e1ce9dd8e29/volumes"
Oct 30 01:08:26.944292 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:26.944257    5002 patch_prober.go:29] interesting pod/e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-x66jp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" start-of-body=
Oct 30 01:08:26.944609 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:26.944568    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-x66jp" podUID="37340929-9f26-4db9-ad7d-66d0bd14a7cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused"
Oct 30 01:08:27.948404 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:27.944761    5002 patch_prober.go:29] interesting pod/e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-x66jp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" start-of-body=
Oct 30 01:08:27.948404 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:27.944845    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-x66jp" podUID="37340929-9f26-4db9-ad7d-66d0bd14a7cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused"
Oct 30 01:08:28.694443 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:28.692698    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0265ca5-5rxfz"]
Oct 30 01:08:28.694730 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:28.694702    5002 cpu_manager.go:414] "containerMap: removing container" podUID="d405389a-c3e3-4aa9-a7db-ec4f32166b46" containerName="payload-pre-processing"
Oct 30 01:08:28.694822 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:28.694732    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d405389a-c3e3-4aa9-a7db-ec4f32166b46" containerName="payload-pre-processing"
Oct 30 01:08:28.694822 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:28.694759    5002 cpu_manager.go:414] "containerMap: removing container" podUID="1ecc17fe-4d17-452c-9856-5ac462ffde03" containerName="revoke-keys"
Oct 30 01:08:28.694822 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:28.694767    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ecc17fe-4d17-452c-9856-5ac462ffde03" containerName="revoke-keys"
Oct 30 01:08:28.694989 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:28.694850    5002 cpu_manager.go:414] "containerMap: removing container" podUID="fbe86602-c69c-4fff-b440-5e1ce9dd8e29" containerName="payload-processing"
Oct 30 01:08:28.694989 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:28.694860    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbe86602-c69c-4fff-b440-5e1ce9dd8e29" containerName="payload-processing"
Oct 30 01:08:28.695091 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:28.695041    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbe86602-c69c-4fff-b440-5e1ce9dd8e29" containerName="payload-processing"
Oct 30 01:08:28.695091 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:28.695060    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="d405389a-c3e3-4aa9-a7db-ec4f32166b46" containerName="payload-pre-processing"
Oct 30 01:08:28.695091 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:28.695072    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ecc17fe-4d17-452c-9856-5ac462ffde03" containerName="revoke-keys"
Oct 30 01:08:28.700636 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:28.700609    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0265ca5-5rxfz"
Oct 30 01:08:28.718322 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:28.715817    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0265ca5-5rxfz"]
Oct 30 01:08:28.722958 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:28.722930    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:08:28.723455 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:28.723430    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:08:28.731625 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:28.731594    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:08:28.731774 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:28.731660    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5phsh\""
Oct 30 01:08:28.741570 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:28.741526    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/822f4720-123e-47f1-b83e-da2c509d6e41-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b0265ca5-5rxfz\" (UID: \"822f4720-123e-47f1-b83e-da2c509d6e41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0265ca5-5rxfz"
Oct 30 01:08:28.741940 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:28.741913    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/822f4720-123e-47f1-b83e-da2c509d6e41-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b0265ca5-5rxfz\" (UID: \"822f4720-123e-47f1-b83e-da2c509d6e41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0265ca5-5rxfz"
Oct 30 01:08:28.742247 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:28.742229    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzm2k\" (UniqueName: \"kubernetes.io/projected/822f4720-123e-47f1-b83e-da2c509d6e41-kube-api-access-pzm2k\") pod \"maas-api-revoke-sub-b0265ca5-5rxfz\" (UID: \"822f4720-123e-47f1-b83e-da2c509d6e41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0265ca5-5rxfz"
Oct 30 01:08:28.843625 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:28.843586    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/822f4720-123e-47f1-b83e-da2c509d6e41-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b0265ca5-5rxfz\" (UID: \"822f4720-123e-47f1-b83e-da2c509d6e41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0265ca5-5rxfz"
Oct 30 01:08:28.843811 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:28.843645    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/822f4720-123e-47f1-b83e-da2c509d6e41-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b0265ca5-5rxfz\" (UID: \"822f4720-123e-47f1-b83e-da2c509d6e41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0265ca5-5rxfz"
Oct 30 01:08:28.843811 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:28.843727    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzm2k\" (UniqueName: \"kubernetes.io/projected/822f4720-123e-47f1-b83e-da2c509d6e41-kube-api-access-pzm2k\") pod \"maas-api-revoke-sub-b0265ca5-5rxfz\" (UID: \"822f4720-123e-47f1-b83e-da2c509d6e41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0265ca5-5rxfz"
Oct 30 01:08:28.844729 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:28.844686    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/822f4720-123e-47f1-b83e-da2c509d6e41-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b0265ca5-5rxfz\" (UID: \"822f4720-123e-47f1-b83e-da2c509d6e41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0265ca5-5rxfz"
Oct 30 01:08:28.846209 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:28.846183    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/822f4720-123e-47f1-b83e-da2c509d6e41-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b0265ca5-5rxfz\" (UID: \"822f4720-123e-47f1-b83e-da2c509d6e41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0265ca5-5rxfz"
Oct 30 01:08:28.869262 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:28.868804    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzm2k\" (UniqueName: \"kubernetes.io/projected/822f4720-123e-47f1-b83e-da2c509d6e41-kube-api-access-pzm2k\") pod \"maas-api-revoke-sub-b0265ca5-5rxfz\" (UID: \"822f4720-123e-47f1-b83e-da2c509d6e41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0265ca5-5rxfz"
Oct 30 01:08:29.001927 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:29.001828    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-x66jp"
Oct 30 01:08:29.002473 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:29.002416    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-x66jp"
Oct 30 01:08:29.003309 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:29.003286    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-x66jp"
Oct 30 01:08:29.022647 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:29.022616    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0265ca5-5rxfz"
Oct 30 01:08:29.353302 ip-10-0-128-7 kubenswrapper[5002]: W1030 01:08:29.353270    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod822f4720_123e_47f1_b83e_da2c509d6e41.slice/crio-60013195f1ed2af93c7b2e38206eff1f2f46aadb131f99b1410127a797565672 WatchSource:0}: Error finding container 60013195f1ed2af93c7b2e38206eff1f2f46aadb131f99b1410127a797565672: Status 404 returned error can't find the container with id 60013195f1ed2af93c7b2e38206eff1f2f46aadb131f99b1410127a797565672
Oct 30 01:08:29.354183 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:29.354153    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0265ca5-5rxfz"]
Oct 30 01:08:29.703426 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:29.702270    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-ef1084-openshift-default-68989c6b9f-lrmtq"]
Oct 30 01:08:29.703426 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:29.702611    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-ef1084-openshift-default-68989c6b9f-lrmtq" podUID="1618b1d3-81ce-4def-9498-43b3fdbc46a4" containerName="istio-proxy" containerID="cri-o://9f9d018de891196758eb912c2a779c5b7d19ea6841ae85b657848faa91f75975" gracePeriod=30
Oct 30 01:08:30.043146 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:30.043104    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0265ca5-5rxfz" event={"ID":"822f4720-123e-47f1-b83e-da2c509d6e41","Type":"ContainerStarted","Data":"a5969443ca43eeb7d91cc30253e980fb62e7bbd5c11fb6bc450fb2b21a61e0df"}
Oct 30 01:08:30.043687 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:30.043170    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0265ca5-5rxfz" event={"ID":"822f4720-123e-47f1-b83e-da2c509d6e41","Type":"ContainerStarted","Data":"60013195f1ed2af93c7b2e38206eff1f2f46aadb131f99b1410127a797565672"}
Oct 30 01:08:30.088719 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:30.088654    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0265ca5-5rxfz" podStartSLOduration=2.088637496 podStartE2EDuration="2.088637496s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:08:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:08:30.086482599 +0000 UTC m=+2205.702915491" watchObservedRunningTime="2026-10-30 01:08:30.088637496 +0000 UTC m=+2205.705070375"
Oct 30 01:08:31.051044 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:31.051010    5002 generic.go:356] "Generic (PLEG): container finished" podID="822f4720-123e-47f1-b83e-da2c509d6e41" containerID="a5969443ca43eeb7d91cc30253e980fb62e7bbd5c11fb6bc450fb2b21a61e0df" exitCode=0
Oct 30 01:08:31.051577 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:31.051100    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0265ca5-5rxfz" event={"ID":"822f4720-123e-47f1-b83e-da2c509d6e41","Type":"ContainerDied","Data":"a5969443ca43eeb7d91cc30253e980fb62e7bbd5c11fb6bc450fb2b21a61e0df"}
Oct 30 01:08:32.100657 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:32.100619    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-19bc68-grz78"]
Oct 30 01:08:32.105172 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:32.105139    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-19bc68-grz78"
Oct 30 01:08:32.122938 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:32.122906    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-19bc68-grz78"]
Oct 30 01:08:32.154509 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:32.153489    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e4336b3-ff70-4a4c-b3e6-0669cb308f87-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-19bc68-grz78\" (UID: \"7e4336b3-ff70-4a4c-b3e6-0669cb308f87\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-19bc68-grz78"
Oct 30 01:08:32.154509 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:32.153776    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e4336b3-ff70-4a4c-b3e6-0669cb308f87-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-19bc68-grz78\" (UID: \"7e4336b3-ff70-4a4c-b3e6-0669cb308f87\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-19bc68-grz78"
Oct 30 01:08:32.154509 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:32.153852    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xjlx\" (UniqueName: \"kubernetes.io/projected/7e4336b3-ff70-4a4c-b3e6-0669cb308f87-kube-api-access-8xjlx\") pod \"maas-api-revoke-keys-e2e-inf-b-19bc68-grz78\" (UID: \"7e4336b3-ff70-4a4c-b3e6-0669cb308f87\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-19bc68-grz78"
Oct 30 01:08:32.255874 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:32.255839    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e4336b3-ff70-4a4c-b3e6-0669cb308f87-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-19bc68-grz78\" (UID: \"7e4336b3-ff70-4a4c-b3e6-0669cb308f87\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-19bc68-grz78"
Oct 30 01:08:32.256021 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:32.255932    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xjlx\" (UniqueName: \"kubernetes.io/projected/7e4336b3-ff70-4a4c-b3e6-0669cb308f87-kube-api-access-8xjlx\") pod \"maas-api-revoke-keys-e2e-inf-b-19bc68-grz78\" (UID: \"7e4336b3-ff70-4a4c-b3e6-0669cb308f87\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-19bc68-grz78"
Oct 30 01:08:32.256168 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:32.256104    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e4336b3-ff70-4a4c-b3e6-0669cb308f87-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-19bc68-grz78\" (UID: \"7e4336b3-ff70-4a4c-b3e6-0669cb308f87\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-19bc68-grz78"
Oct 30 01:08:32.256815 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:32.256789    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e4336b3-ff70-4a4c-b3e6-0669cb308f87-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-19bc68-grz78\" (UID: \"7e4336b3-ff70-4a4c-b3e6-0669cb308f87\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-19bc68-grz78"
Oct 30 01:08:32.258506 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:32.258486    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e4336b3-ff70-4a4c-b3e6-0669cb308f87-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-19bc68-grz78\" (UID: \"7e4336b3-ff70-4a4c-b3e6-0669cb308f87\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-19bc68-grz78"
Oct 30 01:08:32.280865 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:32.280828    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xjlx\" (UniqueName: \"kubernetes.io/projected/7e4336b3-ff70-4a4c-b3e6-0669cb308f87-kube-api-access-8xjlx\") pod \"maas-api-revoke-keys-e2e-inf-b-19bc68-grz78\" (UID: \"7e4336b3-ff70-4a4c-b3e6-0669cb308f87\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-19bc68-grz78"
Oct 30 01:08:32.300019 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:32.299985    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0265ca5-5rxfz"
Oct 30 01:08:32.357478 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:32.357371    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/822f4720-123e-47f1-b83e-da2c509d6e41-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/822f4720-123e-47f1-b83e-da2c509d6e41-maas-api-openshift-ca\") pod \"822f4720-123e-47f1-b83e-da2c509d6e41\" (UID: \"822f4720-123e-47f1-b83e-da2c509d6e41\") "
Oct 30 01:08:32.357821 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:32.357789    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/822f4720-123e-47f1-b83e-da2c509d6e41-maas-api-openshift-ca" pod "822f4720-123e-47f1-b83e-da2c509d6e41" (UID: "822f4720-123e-47f1-b83e-da2c509d6e41"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:08:32.358012 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:32.357988    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/822f4720-123e-47f1-b83e-da2c509d6e41-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/822f4720-123e-47f1-b83e-da2c509d6e41-maas-api-xks-ca\") pod \"822f4720-123e-47f1-b83e-da2c509d6e41\" (UID: \"822f4720-123e-47f1-b83e-da2c509d6e41\") "
Oct 30 01:08:32.358142 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:32.358125    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/822f4720-123e-47f1-b83e-da2c509d6e41-kube-api-access-pzm2k\" (UniqueName: \"kubernetes.io/projected/822f4720-123e-47f1-b83e-da2c509d6e41-kube-api-access-pzm2k\") pod \"822f4720-123e-47f1-b83e-da2c509d6e41\" (UID: \"822f4720-123e-47f1-b83e-da2c509d6e41\") "
Oct 30 01:08:32.358918 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:32.358891    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/822f4720-123e-47f1-b83e-da2c509d6e41-maas-api-openshift-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:08:32.361929 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:32.361896    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/822f4720-123e-47f1-b83e-da2c509d6e41-maas-api-xks-ca" pod "822f4720-123e-47f1-b83e-da2c509d6e41" (UID: "822f4720-123e-47f1-b83e-da2c509d6e41"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:08:32.376994 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:32.376958    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/822f4720-123e-47f1-b83e-da2c509d6e41-kube-api-access-pzm2k" pod "822f4720-123e-47f1-b83e-da2c509d6e41" (UID: "822f4720-123e-47f1-b83e-da2c509d6e41"). InnerVolumeSpecName "kube-api-access-pzm2k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:08:32.423723 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:32.423693    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-19bc68-grz78"
Oct 30 01:08:32.460775 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:32.460737    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/822f4720-123e-47f1-b83e-da2c509d6e41-maas-api-xks-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:08:32.460775 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:32.460773    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pzm2k\" (UniqueName: \"kubernetes.io/projected/822f4720-123e-47f1-b83e-da2c509d6e41-kube-api-access-pzm2k\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:08:32.731157 ip-10-0-128-7 kubenswrapper[5002]: W1030 01:08:32.731123    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e4336b3_ff70_4a4c_b3e6_0669cb308f87.slice/crio-c669d88be1c47e8a5dc015f7ac6a8d562629f8724600c0153c5b01064c2a6635 WatchSource:0}: Error finding container c669d88be1c47e8a5dc015f7ac6a8d562629f8724600c0153c5b01064c2a6635: Status 404 returned error can't find the container with id c669d88be1c47e8a5dc015f7ac6a8d562629f8724600c0153c5b01064c2a6635
Oct 30 01:08:32.732745 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:32.732713    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-19bc68-grz78"]
Oct 30 01:08:33.076810 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:33.076775    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-19bc68-grz78" event={"ID":"7e4336b3-ff70-4a4c-b3e6-0669cb308f87","Type":"ContainerStarted","Data":"c669d88be1c47e8a5dc015f7ac6a8d562629f8724600c0153c5b01064c2a6635"}
Oct 30 01:08:33.078360 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:33.078330    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0265ca5-5rxfz" event={"ID":"822f4720-123e-47f1-b83e-da2c509d6e41","Type":"ContainerDied","Data":"60013195f1ed2af93c7b2e38206eff1f2f46aadb131f99b1410127a797565672"}
Oct 30 01:08:33.078490 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:33.078358    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0265ca5-5rxfz"
Oct 30 01:08:33.078490 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:33.078373    5002 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="60013195f1ed2af93c7b2e38206eff1f2f46aadb131f99b1410127a797565672"
Oct 30 01:08:34.090945 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:34.090853    5002 generic.go:356] "Generic (PLEG): container finished" podID="7e4336b3-ff70-4a4c-b3e6-0669cb308f87" containerID="29794e77725969ea6dc3724d69f11a205055df813994e45e491a58d2f5090d2e" exitCode=0
Oct 30 01:08:34.091351 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:34.090939    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-19bc68-grz78" event={"ID":"7e4336b3-ff70-4a4c-b3e6-0669cb308f87","Type":"ContainerDied","Data":"29794e77725969ea6dc3724d69f11a205055df813994e45e491a58d2f5090d2e"}
Oct 30 01:08:35.044069 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:35.044039    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ef1084-openshift-default-68989c6b9f-lrmtq"
Oct 30 01:08:35.106824 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:35.106792    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1618b1d3-81ce-4def-9498-43b3fdbc46a4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1618b1d3-81ce-4def-9498-43b3fdbc46a4-workload-certs\") pod \"1618b1d3-81ce-4def-9498-43b3fdbc46a4\" (UID: \"1618b1d3-81ce-4def-9498-43b3fdbc46a4\") "
Oct 30 01:08:35.107319 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:35.106851    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1618b1d3-81ce-4def-9498-43b3fdbc46a4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1618b1d3-81ce-4def-9498-43b3fdbc46a4-istio-data\") pod \"1618b1d3-81ce-4def-9498-43b3fdbc46a4\" (UID: \"1618b1d3-81ce-4def-9498-43b3fdbc46a4\") "
Oct 30 01:08:35.107319 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:35.106963    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1618b1d3-81ce-4def-9498-43b3fdbc46a4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1618b1d3-81ce-4def-9498-43b3fdbc46a4-istiod-ca-cert\") pod \"1618b1d3-81ce-4def-9498-43b3fdbc46a4\" (UID: \"1618b1d3-81ce-4def-9498-43b3fdbc46a4\") "
Oct 30 01:08:35.107319 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:35.107030    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1618b1d3-81ce-4def-9498-43b3fdbc46a4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1618b1d3-81ce-4def-9498-43b3fdbc46a4-workload-socket\") pod \"1618b1d3-81ce-4def-9498-43b3fdbc46a4\" (UID: \"1618b1d3-81ce-4def-9498-43b3fdbc46a4\") "
Oct 30 01:08:35.107319 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:35.107097    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1618b1d3-81ce-4def-9498-43b3fdbc46a4-workload-certs" pod "1618b1d3-81ce-4def-9498-43b3fdbc46a4" (UID: "1618b1d3-81ce-4def-9498-43b3fdbc46a4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:08:35.107319 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:35.107105    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1618b1d3-81ce-4def-9498-43b3fdbc46a4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1618b1d3-81ce-4def-9498-43b3fdbc46a4-istio-envoy\") pod \"1618b1d3-81ce-4def-9498-43b3fdbc46a4\" (UID: \"1618b1d3-81ce-4def-9498-43b3fdbc46a4\") "
Oct 30 01:08:35.107319 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:35.107136    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1618b1d3-81ce-4def-9498-43b3fdbc46a4-kube-api-access-txd2n\" (UniqueName: \"kubernetes.io/projected/1618b1d3-81ce-4def-9498-43b3fdbc46a4-kube-api-access-txd2n\") pod \"1618b1d3-81ce-4def-9498-43b3fdbc46a4\" (UID: \"1618b1d3-81ce-4def-9498-43b3fdbc46a4\") "
Oct 30 01:08:35.107319 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:35.107195    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1618b1d3-81ce-4def-9498-43b3fdbc46a4-istio-token\" (UniqueName: \"kubernetes.io/projected/1618b1d3-81ce-4def-9498-43b3fdbc46a4-istio-token\") pod \"1618b1d3-81ce-4def-9498-43b3fdbc46a4\" (UID: \"1618b1d3-81ce-4def-9498-43b3fdbc46a4\") "
Oct 30 01:08:35.107319 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:35.107289    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1618b1d3-81ce-4def-9498-43b3fdbc46a4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1618b1d3-81ce-4def-9498-43b3fdbc46a4-credential-socket\") pod \"1618b1d3-81ce-4def-9498-43b3fdbc46a4\" (UID: \"1618b1d3-81ce-4def-9498-43b3fdbc46a4\") "
Oct 30 01:08:35.107764 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:35.107433    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1618b1d3-81ce-4def-9498-43b3fdbc46a4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1618b1d3-81ce-4def-9498-43b3fdbc46a4-istio-podinfo\") pod \"1618b1d3-81ce-4def-9498-43b3fdbc46a4\" (UID: \"1618b1d3-81ce-4def-9498-43b3fdbc46a4\") "
Oct 30 01:08:35.107764 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:35.107441    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1618b1d3-81ce-4def-9498-43b3fdbc46a4-istiod-ca-cert" pod "1618b1d3-81ce-4def-9498-43b3fdbc46a4" (UID: "1618b1d3-81ce-4def-9498-43b3fdbc46a4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:08:35.107764 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:35.107674    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1618b1d3-81ce-4def-9498-43b3fdbc46a4-istio-data" pod "1618b1d3-81ce-4def-9498-43b3fdbc46a4" (UID: "1618b1d3-81ce-4def-9498-43b3fdbc46a4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:08:35.108152 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:35.108128    5002 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1618b1d3-81ce-4def-9498-43b3fdbc46a4-workload-certs\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:08:35.108152 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:35.108151    5002 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1618b1d3-81ce-4def-9498-43b3fdbc46a4-istio-data\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:08:35.108329 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:35.108167    5002 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1618b1d3-81ce-4def-9498-43b3fdbc46a4-istiod-ca-cert\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:08:35.108482 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:35.108457    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1618b1d3-81ce-4def-9498-43b3fdbc46a4-workload-socket" pod "1618b1d3-81ce-4def-9498-43b3fdbc46a4" (UID: "1618b1d3-81ce-4def-9498-43b3fdbc46a4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:08:35.108703 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:35.108677    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1618b1d3-81ce-4def-9498-43b3fdbc46a4-credential-socket" pod "1618b1d3-81ce-4def-9498-43b3fdbc46a4" (UID: "1618b1d3-81ce-4def-9498-43b3fdbc46a4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:08:35.114495 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:35.110590    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1618b1d3-81ce-4def-9498-43b3fdbc46a4-kube-api-access-txd2n" pod "1618b1d3-81ce-4def-9498-43b3fdbc46a4" (UID: "1618b1d3-81ce-4def-9498-43b3fdbc46a4"). InnerVolumeSpecName "kube-api-access-txd2n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:08:35.114495 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:35.110627    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1618b1d3-81ce-4def-9498-43b3fdbc46a4-istio-token" pod "1618b1d3-81ce-4def-9498-43b3fdbc46a4" (UID: "1618b1d3-81ce-4def-9498-43b3fdbc46a4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:08:35.114495 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:35.111586    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1618b1d3-81ce-4def-9498-43b3fdbc46a4-istio-podinfo" pod "1618b1d3-81ce-4def-9498-43b3fdbc46a4" (UID: "1618b1d3-81ce-4def-9498-43b3fdbc46a4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:08:35.114495 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:35.112483    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1618b1d3-81ce-4def-9498-43b3fdbc46a4-istio-envoy" pod "1618b1d3-81ce-4def-9498-43b3fdbc46a4" (UID: "1618b1d3-81ce-4def-9498-43b3fdbc46a4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:08:35.116426 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:35.116352    5002 generic.go:356] "Generic (PLEG): container finished" podID="1618b1d3-81ce-4def-9498-43b3fdbc46a4" containerID="9f9d018de891196758eb912c2a779c5b7d19ea6841ae85b657848faa91f75975" exitCode=0
Oct 30 01:08:35.116831 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:35.116808    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ef1084-openshift-default-68989c6b9f-lrmtq"
Oct 30 01:08:35.122678 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:35.121699    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ef1084-openshift-default-68989c6b9f-lrmtq" event={"ID":"1618b1d3-81ce-4def-9498-43b3fdbc46a4","Type":"ContainerDied","Data":"9f9d018de891196758eb912c2a779c5b7d19ea6841ae85b657848faa91f75975"}
Oct 30 01:08:35.122678 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:35.122151    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ef1084-openshift-default-68989c6b9f-lrmtq" event={"ID":"1618b1d3-81ce-4def-9498-43b3fdbc46a4","Type":"ContainerDied","Data":"af116c343b1d7035544afdf077769fae2d76cf81a278e0ae944ab7b9a02adfc9"}
Oct 30 01:08:35.122678 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:35.122171    5002 scope.go:122] "RemoveContainer" containerID="9f9d018de891196758eb912c2a779c5b7d19ea6841ae85b657848faa91f75975"
Oct 30 01:08:35.180962 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:35.180349    5002 scope.go:122] "RemoveContainer" containerID="b9bb212d7256f99518451eacffb2043a7ef5781b2b01367473460ad8117c5575"
Oct 30 01:08:35.211422 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:35.210847    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-ef1084-openshift-default-68989c6b9f-lrmtq"]
Oct 30 01:08:35.211422 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:35.211060    5002 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1618b1d3-81ce-4def-9498-43b3fdbc46a4-istio-podinfo\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:08:35.211422 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:35.211081    5002 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1618b1d3-81ce-4def-9498-43b3fdbc46a4-workload-socket\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:08:35.211422 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:35.211095    5002 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1618b1d3-81ce-4def-9498-43b3fdbc46a4-istio-envoy\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:08:35.211422 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:35.211110    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-txd2n\" (UniqueName: \"kubernetes.io/projected/1618b1d3-81ce-4def-9498-43b3fdbc46a4-kube-api-access-txd2n\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:08:35.211422 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:35.211124    5002 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1618b1d3-81ce-4def-9498-43b3fdbc46a4-istio-token\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:08:35.211422 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:35.211138    5002 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1618b1d3-81ce-4def-9498-43b3fdbc46a4-credential-socket\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:08:35.214115 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:35.213309    5002 scope.go:122] "RemoveContainer" containerID="9f9d018de891196758eb912c2a779c5b7d19ea6841ae85b657848faa91f75975"
Oct 30 01:08:35.214964 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:08:35.214815    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f9d018de891196758eb912c2a779c5b7d19ea6841ae85b657848faa91f75975\": container with ID starting with 9f9d018de891196758eb912c2a779c5b7d19ea6841ae85b657848faa91f75975 not found: ID does not exist" containerID="9f9d018de891196758eb912c2a779c5b7d19ea6841ae85b657848faa91f75975"
Oct 30 01:08:35.214964 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:35.214851    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f9d018de891196758eb912c2a779c5b7d19ea6841ae85b657848faa91f75975"} err="failed to get container status \"9f9d018de891196758eb912c2a779c5b7d19ea6841ae85b657848faa91f75975\": rpc error: code = NotFound desc = could not find container \"9f9d018de891196758eb912c2a779c5b7d19ea6841ae85b657848faa91f75975\": container with ID starting with 9f9d018de891196758eb912c2a779c5b7d19ea6841ae85b657848faa91f75975 not found: ID does not exist"
Oct 30 01:08:35.214964 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:35.214877    5002 scope.go:122] "RemoveContainer" containerID="b9bb212d7256f99518451eacffb2043a7ef5781b2b01367473460ad8117c5575"
Oct 30 01:08:35.215274 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:08:35.215185    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9bb212d7256f99518451eacffb2043a7ef5781b2b01367473460ad8117c5575\": container with ID starting with b9bb212d7256f99518451eacffb2043a7ef5781b2b01367473460ad8117c5575 not found: ID does not exist" containerID="b9bb212d7256f99518451eacffb2043a7ef5781b2b01367473460ad8117c5575"
Oct 30 01:08:35.215274 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:35.215220    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9bb212d7256f99518451eacffb2043a7ef5781b2b01367473460ad8117c5575"} err="failed to get container status \"b9bb212d7256f99518451eacffb2043a7ef5781b2b01367473460ad8117c5575\": rpc error: code = NotFound desc = could not find container \"b9bb212d7256f99518451eacffb2043a7ef5781b2b01367473460ad8117c5575\": container with ID starting with b9bb212d7256f99518451eacffb2043a7ef5781b2b01367473460ad8117c5575 not found: ID does not exist"
Oct 30 01:08:35.216923 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:35.216899    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-ef1084-openshift-default-68989c6b9f-lrmtq"]
Oct 30 01:08:35.340855 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:35.340831    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-19bc68-grz78"
Oct 30 01:08:35.414238 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:35.414204    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e4336b3-ff70-4a4c-b3e6-0669cb308f87-kube-api-access-8xjlx\" (UniqueName: \"kubernetes.io/projected/7e4336b3-ff70-4a4c-b3e6-0669cb308f87-kube-api-access-8xjlx\") pod \"7e4336b3-ff70-4a4c-b3e6-0669cb308f87\" (UID: \"7e4336b3-ff70-4a4c-b3e6-0669cb308f87\") "
Oct 30 01:08:35.414452 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:35.414323    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e4336b3-ff70-4a4c-b3e6-0669cb308f87-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e4336b3-ff70-4a4c-b3e6-0669cb308f87-maas-api-openshift-ca\") pod \"7e4336b3-ff70-4a4c-b3e6-0669cb308f87\" (UID: \"7e4336b3-ff70-4a4c-b3e6-0669cb308f87\") "
Oct 30 01:08:35.414525 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:35.414455    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7e4336b3-ff70-4a4c-b3e6-0669cb308f87-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e4336b3-ff70-4a4c-b3e6-0669cb308f87-maas-api-xks-ca\") pod \"7e4336b3-ff70-4a4c-b3e6-0669cb308f87\" (UID: \"7e4336b3-ff70-4a4c-b3e6-0669cb308f87\") "
Oct 30 01:08:35.414792 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:35.414758    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e4336b3-ff70-4a4c-b3e6-0669cb308f87-maas-api-openshift-ca" pod "7e4336b3-ff70-4a4c-b3e6-0669cb308f87" (UID: "7e4336b3-ff70-4a4c-b3e6-0669cb308f87"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:08:35.415510 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:35.415253    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e4336b3-ff70-4a4c-b3e6-0669cb308f87-maas-api-openshift-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:08:35.416794 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:35.416771    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e4336b3-ff70-4a4c-b3e6-0669cb308f87-kube-api-access-8xjlx" pod "7e4336b3-ff70-4a4c-b3e6-0669cb308f87" (UID: "7e4336b3-ff70-4a4c-b3e6-0669cb308f87"). InnerVolumeSpecName "kube-api-access-8xjlx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:08:35.417063 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:35.417040    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e4336b3-ff70-4a4c-b3e6-0669cb308f87-maas-api-xks-ca" pod "7e4336b3-ff70-4a4c-b3e6-0669cb308f87" (UID: "7e4336b3-ff70-4a4c-b3e6-0669cb308f87"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:08:35.563959 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:35.520990    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e4336b3-ff70-4a4c-b3e6-0669cb308f87-maas-api-xks-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:08:35.564484 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:35.564451    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8xjlx\" (UniqueName: \"kubernetes.io/projected/7e4336b3-ff70-4a4c-b3e6-0669cb308f87-kube-api-access-8xjlx\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:08:36.127776 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:36.127684    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-19bc68-grz78" event={"ID":"7e4336b3-ff70-4a4c-b3e6-0669cb308f87","Type":"ContainerDied","Data":"c669d88be1c47e8a5dc015f7ac6a8d562629f8724600c0153c5b01064c2a6635"}
Oct 30 01:08:36.127776 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:36.127708    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-19bc68-grz78"
Oct 30 01:08:36.127776 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:36.127735    5002 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c669d88be1c47e8a5dc015f7ac6a8d562629f8724600c0153c5b01064c2a6635"
Oct 30 01:08:36.514928 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:36.514883    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1618b1d3-81ce-4def-9498-43b3fdbc46a4" path="/var/lib/kubelet/pods/1618b1d3-81ce-4def-9498-43b3fdbc46a4/volumes"
Oct 30 01:08:42.627266 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:42.627223    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-19bc68-54864556d8-n2648"]
Oct 30 01:08:42.776576 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:42.776442    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-19bc68-59cc54d647-zvgzf"]
Oct 30 01:08:42.899766 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:08:42.891362    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbjdq" containerName="extractor"
Oct 30 01:08:47.032284 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:47.032242    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f874e14-m6h2d"]
Oct 30 01:08:47.045363 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:47.045329    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f874e14-m6h2d"]
Oct 30 01:08:47.669800 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:47.669722    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-19bc68-54864556d8-n2648" podUID="64d5b052-11b9-46e0-b159-d43050f9778b" containerName="payload-processing" containerID="cri-o://d533ea1be077eb77213d543d0426907e005d9d03ebf2c45f128ab3a0d37e37af" gracePeriod=25
Oct 30 01:08:47.721220 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:47.721186    5002 scope.go:122] "RemoveContainer" containerID="62f03545c7803e2527c8fcd846da98daaf12fec8f9baf68f92257693c12cf9c4"
Oct 30 01:08:47.777704 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:47.777635    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-19bc68-59cc54d647-zvgzf" podUID="9edfd30b-6f37-4849-9dd8-b5b294c003d7" containerName="payload-pre-processing" containerID="cri-o://d9403cef7ffdc8103ed3d3008cb8846b1ea63e616d3edd548cf1323ce66b7c88" gracePeriod=25
Oct 30 01:08:47.785841 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:47.785811    5002 scope.go:122] "RemoveContainer" containerID="b20841f2f72abc8027f221c1090145bd17a4676d10931ca810b6877230ccf502"
Oct 30 01:08:47.796869 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:47.796848    5002 scope.go:122] "RemoveContainer" containerID="8c2cf7d8d68bb93abf721f230173bd6209acfca141428bfd55415834c4e1eacb"
Oct 30 01:08:47.808761 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:47.808727    5002 scope.go:122] "RemoveContainer" containerID="6b47c750fe98ef9ef3ee221b1171fbc2b633dccad9a283684553dd72673b1bda"
Oct 30 01:08:47.995310 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:47.995282    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-19bc68-54864556d8-n2648"
Oct 30 01:08:48.206403 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:48.203227    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64d5b052-11b9-46e0-b159-d43050f9778b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64d5b052-11b9-46e0-b159-d43050f9778b-plugins-config-volume\") pod \"64d5b052-11b9-46e0-b159-d43050f9778b\" (UID: \"64d5b052-11b9-46e0-b159-d43050f9778b\") "
Oct 30 01:08:48.206403 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:48.203302    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64d5b052-11b9-46e0-b159-d43050f9778b-kube-api-access-vp655\" (UniqueName: \"kubernetes.io/projected/64d5b052-11b9-46e0-b159-d43050f9778b-kube-api-access-vp655\") pod \"64d5b052-11b9-46e0-b159-d43050f9778b\" (UID: \"64d5b052-11b9-46e0-b159-d43050f9778b\") "
Oct 30 01:08:48.206403 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:48.204936    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64d5b052-11b9-46e0-b159-d43050f9778b-plugins-config-volume" pod "64d5b052-11b9-46e0-b159-d43050f9778b" (UID: "64d5b052-11b9-46e0-b159-d43050f9778b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:08:48.215422 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:48.212179    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-48df96/rate-test-model-48df96-kserve-5b9bb8959c-pd7z9"]
Oct 30 01:08:48.216279 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:48.216248    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64d5b052-11b9-46e0-b159-d43050f9778b-kube-api-access-vp655" pod "64d5b052-11b9-46e0-b159-d43050f9778b" (UID: "64d5b052-11b9-46e0-b159-d43050f9778b"). InnerVolumeSpecName "kube-api-access-vp655". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:08:48.275416 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:48.275350    5002 generic.go:356] "Generic (PLEG): container finished" podID="64d5b052-11b9-46e0-b159-d43050f9778b" containerID="d533ea1be077eb77213d543d0426907e005d9d03ebf2c45f128ab3a0d37e37af" exitCode=0
Oct 30 01:08:48.275608 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:48.275451    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-19bc68-54864556d8-n2648"
Oct 30 01:08:48.275608 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:48.275457    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-19bc68-54864556d8-n2648" event={"ID":"64d5b052-11b9-46e0-b159-d43050f9778b","Type":"ContainerDied","Data":"d533ea1be077eb77213d543d0426907e005d9d03ebf2c45f128ab3a0d37e37af"}
Oct 30 01:08:48.275608 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:48.275511    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-19bc68-54864556d8-n2648" event={"ID":"64d5b052-11b9-46e0-b159-d43050f9778b","Type":"ContainerDied","Data":"943813a0faf51e81dc3e8874d3c0a48039fe14a7d65f1aecbba577e3d4a20105"}
Oct 30 01:08:48.275608 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:48.275557    5002 scope.go:122] "RemoveContainer" containerID="d533ea1be077eb77213d543d0426907e005d9d03ebf2c45f128ab3a0d37e37af"
Oct 30 01:08:48.288274 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:48.288254    5002 scope.go:122] "RemoveContainer" containerID="d533ea1be077eb77213d543d0426907e005d9d03ebf2c45f128ab3a0d37e37af"
Oct 30 01:08:48.288678 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:08:48.288651    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d533ea1be077eb77213d543d0426907e005d9d03ebf2c45f128ab3a0d37e37af\": container with ID starting with d533ea1be077eb77213d543d0426907e005d9d03ebf2c45f128ab3a0d37e37af not found: ID does not exist" containerID="d533ea1be077eb77213d543d0426907e005d9d03ebf2c45f128ab3a0d37e37af"
Oct 30 01:08:48.288792 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:48.288689    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d533ea1be077eb77213d543d0426907e005d9d03ebf2c45f128ab3a0d37e37af"} err="failed to get container status \"d533ea1be077eb77213d543d0426907e005d9d03ebf2c45f128ab3a0d37e37af\": rpc error: code = NotFound desc = could not find container \"d533ea1be077eb77213d543d0426907e005d9d03ebf2c45f128ab3a0d37e37af\": container with ID starting with d533ea1be077eb77213d543d0426907e005d9d03ebf2c45f128ab3a0d37e37af not found: ID does not exist"
Oct 30 01:08:48.346444 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:48.323842    5002 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64d5b052-11b9-46e0-b159-d43050f9778b-plugins-config-volume\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:08:48.346444 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:48.323885    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vp655\" (UniqueName: \"kubernetes.io/projected/64d5b052-11b9-46e0-b159-d43050f9778b-kube-api-access-vp655\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:08:48.391444 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:48.386312    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-19bc68-54864556d8-n2648"]
Oct 30 01:08:48.408540 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:48.402557    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-19bc68-54864556d8-n2648"]
Oct 30 01:08:48.497250 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:48.497144    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64d5b052-11b9-46e0-b159-d43050f9778b" path="/var/lib/kubelet/pods/64d5b052-11b9-46e0-b159-d43050f9778b/volumes"
Oct 30 01:08:48.497915 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:48.497894    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f879b587-9122-48b1-bca2-cecb3d22067e" path="/var/lib/kubelet/pods/f879b587-9122-48b1-bca2-cecb3d22067e/volumes"
Oct 30 01:08:53.007053 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:53.007012    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-19bc68-59cc54d647-zvgzf" podUID="9edfd30b-6f37-4849-9dd8-b5b294c003d7" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:08:57.030778 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:57.030738    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82b7e656-hb79m"]
Oct 30 01:08:57.044264 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:57.044235    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82b7e656-hb79m"]
Oct 30 01:08:58.495833 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:08:58.495795    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ccd498da-017b-4a42-a806-21b31d3fe326" path="/var/lib/kubelet/pods/ccd498da-017b-4a42-a806-21b31d3fe326/volumes"
Oct 30 01:09:03.006076 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:03.006037    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-19bc68-59cc54d647-zvgzf" podUID="9edfd30b-6f37-4849-9dd8-b5b294c003d7" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:09:03.248508 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:03.248443    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w1-48df96/rate-test-model-48df96-kserve-5b9bb8959c-pd7z9" podUID="fab2daef-ee8c-44f5-8e11-1d11372976a1" containerName="main" containerID="cri-o://e2deec5d8ea1cb5c9d0b6a5462bcf50f11d3ec726456e69252ce73f5798efb8e" gracePeriod=45
Oct 30 01:09:03.259312 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:03.259237    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w1-48df96/rate-test-model-48df96-kserve-5b9bb8959c-pd7z9" podUID="fab2daef-ee8c-44f5-8e11-1d11372976a1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.177:8000/ready\": dial tcp 10.132.0.177:8000: connect: connection refused"
Oct 30 01:09:03.478139 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:03.478104    5002 generic.go:356] "Generic (PLEG): container finished" podID="fab2daef-ee8c-44f5-8e11-1d11372976a1" containerID="e2deec5d8ea1cb5c9d0b6a5462bcf50f11d3ec726456e69252ce73f5798efb8e" exitCode=0
Oct 30 01:09:03.478341 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:03.478165    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-48df96/rate-test-model-48df96-kserve-5b9bb8959c-pd7z9" event={"ID":"fab2daef-ee8c-44f5-8e11-1d11372976a1","Type":"ContainerDied","Data":"e2deec5d8ea1cb5c9d0b6a5462bcf50f11d3ec726456e69252ce73f5798efb8e"}
Oct 30 01:09:03.534375 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:03.534343    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-48df96/rate-test-model-48df96-kserve-5b9bb8959c-pd7z9"
Oct 30 01:09:03.575635 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:03.575595    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fab2daef-ee8c-44f5-8e11-1d11372976a1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fab2daef-ee8c-44f5-8e11-1d11372976a1-tmp-dir\") pod \"fab2daef-ee8c-44f5-8e11-1d11372976a1\" (UID: \"fab2daef-ee8c-44f5-8e11-1d11372976a1\") "
Oct 30 01:09:03.575839 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:03.575725    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fab2daef-ee8c-44f5-8e11-1d11372976a1-home\" (UniqueName: \"kubernetes.io/empty-dir/fab2daef-ee8c-44f5-8e11-1d11372976a1-home\") pod \"fab2daef-ee8c-44f5-8e11-1d11372976a1\" (UID: \"fab2daef-ee8c-44f5-8e11-1d11372976a1\") "
Oct 30 01:09:03.575839 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:03.575785    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fab2daef-ee8c-44f5-8e11-1d11372976a1-tls-certs\" (UniqueName: \"kubernetes.io/secret/fab2daef-ee8c-44f5-8e11-1d11372976a1-tls-certs\") pod \"fab2daef-ee8c-44f5-8e11-1d11372976a1\" (UID: \"fab2daef-ee8c-44f5-8e11-1d11372976a1\") "
Oct 30 01:09:03.575839 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:03.575802    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fab2daef-ee8c-44f5-8e11-1d11372976a1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fab2daef-ee8c-44f5-8e11-1d11372976a1-model-cache\") pod \"fab2daef-ee8c-44f5-8e11-1d11372976a1\" (UID: \"fab2daef-ee8c-44f5-8e11-1d11372976a1\") "
Oct 30 01:09:03.576026 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:03.575861    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fab2daef-ee8c-44f5-8e11-1d11372976a1-kube-api-access-fn4qw\" (UniqueName: \"kubernetes.io/projected/fab2daef-ee8c-44f5-8e11-1d11372976a1-kube-api-access-fn4qw\") pod \"fab2daef-ee8c-44f5-8e11-1d11372976a1\" (UID: \"fab2daef-ee8c-44f5-8e11-1d11372976a1\") "
Oct 30 01:09:03.576026 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:03.575871    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fab2daef-ee8c-44f5-8e11-1d11372976a1-tmp-dir" pod "fab2daef-ee8c-44f5-8e11-1d11372976a1" (UID: "fab2daef-ee8c-44f5-8e11-1d11372976a1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:09:03.576123 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:03.576025    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fab2daef-ee8c-44f5-8e11-1d11372976a1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/fab2daef-ee8c-44f5-8e11-1d11372976a1-dshm\") pod \"fab2daef-ee8c-44f5-8e11-1d11372976a1\" (UID: \"fab2daef-ee8c-44f5-8e11-1d11372976a1\") "
Oct 30 01:09:03.576123 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:03.576035    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fab2daef-ee8c-44f5-8e11-1d11372976a1-home" pod "fab2daef-ee8c-44f5-8e11-1d11372976a1" (UID: "fab2daef-ee8c-44f5-8e11-1d11372976a1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:09:03.576123 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:03.576074    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fab2daef-ee8c-44f5-8e11-1d11372976a1-model-cache" pod "fab2daef-ee8c-44f5-8e11-1d11372976a1" (UID: "fab2daef-ee8c-44f5-8e11-1d11372976a1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:09:03.576967 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:03.576792    5002 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fab2daef-ee8c-44f5-8e11-1d11372976a1-home\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:09:03.576967 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:03.576814    5002 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fab2daef-ee8c-44f5-8e11-1d11372976a1-model-cache\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:09:03.576967 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:03.576825    5002 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fab2daef-ee8c-44f5-8e11-1d11372976a1-tmp-dir\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:09:03.579042 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:03.578658    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fab2daef-ee8c-44f5-8e11-1d11372976a1-dshm" pod "fab2daef-ee8c-44f5-8e11-1d11372976a1" (UID: "fab2daef-ee8c-44f5-8e11-1d11372976a1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:09:03.579042 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:03.578696    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fab2daef-ee8c-44f5-8e11-1d11372976a1-tls-certs" pod "fab2daef-ee8c-44f5-8e11-1d11372976a1" (UID: "fab2daef-ee8c-44f5-8e11-1d11372976a1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:09:03.579468 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:03.579135    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fab2daef-ee8c-44f5-8e11-1d11372976a1-kube-api-access-fn4qw" pod "fab2daef-ee8c-44f5-8e11-1d11372976a1" (UID: "fab2daef-ee8c-44f5-8e11-1d11372976a1"). InnerVolumeSpecName "kube-api-access-fn4qw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:09:03.678039 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:03.678006    5002 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fab2daef-ee8c-44f5-8e11-1d11372976a1-dshm\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:09:03.678039 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:03.678036    5002 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fab2daef-ee8c-44f5-8e11-1d11372976a1-tls-certs\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:09:03.678039 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:03.678047    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fn4qw\" (UniqueName: \"kubernetes.io/projected/fab2daef-ee8c-44f5-8e11-1d11372976a1-kube-api-access-fn4qw\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:09:04.503326 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:04.503290    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-48df96/rate-test-model-48df96-kserve-5b9bb8959c-pd7z9"
Oct 30 01:09:04.524273 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:04.522772    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-48df96/rate-test-model-48df96-kserve-5b9bb8959c-pd7z9" event={"ID":"fab2daef-ee8c-44f5-8e11-1d11372976a1","Type":"ContainerDied","Data":"7e444533a3b72ee0763c06d9f4a72e92d5f90955f272991cc8d195611761ccc6"}
Oct 30 01:09:04.524273 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:04.522819    5002 scope.go:122] "RemoveContainer" containerID="e2deec5d8ea1cb5c9d0b6a5462bcf50f11d3ec726456e69252ce73f5798efb8e"
Oct 30 01:09:04.586525 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:04.586264    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-48df96/rate-test-model-48df96-kserve-5b9bb8959c-pd7z9"]
Oct 30 01:09:04.590158 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:04.590114    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w1-48df96/rate-test-model-48df96-kserve-5b9bb8959c-pd7z9"]
Oct 30 01:09:05.160834 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:05.160794    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24080105-hzdnf"]
Oct 30 01:09:05.166934 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:05.166900    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24080105-hzdnf"]
Oct 30 01:09:05.243447 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:05.243409    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-19bc68-grz78"]
Oct 30 01:09:05.250963 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:05.250930    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-19bc68-grz78"]
Oct 30 01:09:05.305329 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:05.305288    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-73f667-openshift-default-7bcfd5c9b9-7fcm7"]
Oct 30 01:09:05.306297 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:05.306272    5002 cpu_manager.go:414] "containerMap: removing container" podUID="822f4720-123e-47f1-b83e-da2c509d6e41" containerName="revoke-keys"
Oct 30 01:09:05.306457 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:05.306299    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="822f4720-123e-47f1-b83e-da2c509d6e41" containerName="revoke-keys"
Oct 30 01:09:05.306457 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:05.306321    5002 cpu_manager.go:414] "containerMap: removing container" podUID="7e4336b3-ff70-4a4c-b3e6-0669cb308f87" containerName="revoke-keys"
Oct 30 01:09:05.306457 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:05.306329    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e4336b3-ff70-4a4c-b3e6-0669cb308f87" containerName="revoke-keys"
Oct 30 01:09:05.306457 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:05.306347    5002 cpu_manager.go:414] "containerMap: removing container" podUID="64d5b052-11b9-46e0-b159-d43050f9778b" containerName="payload-processing"
Oct 30 01:09:05.306457 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:05.306355    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64d5b052-11b9-46e0-b159-d43050f9778b" containerName="payload-processing"
Oct 30 01:09:05.306457 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:05.306369    5002 cpu_manager.go:414] "containerMap: removing container" podUID="fab2daef-ee8c-44f5-8e11-1d11372976a1" containerName="main"
Oct 30 01:09:05.306457 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:05.306392    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fab2daef-ee8c-44f5-8e11-1d11372976a1" containerName="main"
Oct 30 01:09:05.306457 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:05.306417    5002 cpu_manager.go:414] "containerMap: removing container" podUID="1618b1d3-81ce-4def-9498-43b3fdbc46a4" containerName="istio-proxy"
Oct 30 01:09:05.306457 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:05.306425    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1618b1d3-81ce-4def-9498-43b3fdbc46a4" containerName="istio-proxy"
Oct 30 01:09:05.306457 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:05.306446    5002 cpu_manager.go:414] "containerMap: removing container" podUID="1618b1d3-81ce-4def-9498-43b3fdbc46a4" containerName="istio-proxy"
Oct 30 01:09:05.306457 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:05.306453    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1618b1d3-81ce-4def-9498-43b3fdbc46a4" containerName="istio-proxy"
Oct 30 01:09:05.307069 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:05.306664    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="822f4720-123e-47f1-b83e-da2c509d6e41" containerName="revoke-keys"
Oct 30 01:09:05.307069 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:05.306681    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e4336b3-ff70-4a4c-b3e6-0669cb308f87" containerName="revoke-keys"
Oct 30 01:09:05.307069 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:05.306694    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="64d5b052-11b9-46e0-b159-d43050f9778b" containerName="payload-processing"
Oct 30 01:09:05.307069 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:05.306703    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="fab2daef-ee8c-44f5-8e11-1d11372976a1" containerName="main"
Oct 30 01:09:05.307069 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:05.306715    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="1618b1d3-81ce-4def-9498-43b3fdbc46a4" containerName="istio-proxy"
Oct 30 01:09:05.307069 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:05.306725    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="1618b1d3-81ce-4def-9498-43b3fdbc46a4" containerName="istio-proxy"
Oct 30 01:09:05.312253 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:05.312231    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-73f667-openshift-default-7bcfd5c9b9-7fcm7"
Oct 30 01:09:05.329918 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:05.329891    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-73f667-openshift-default-dockercfg-bvg5x\""
Oct 30 01:09:05.335681 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:05.335654    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-73f667-openshift-default-7bcfd5c9b9-7fcm7"]
Oct 30 01:09:05.448804 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:05.448762    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0eb54f67-9f54-468e-a88c-ba3fbcaa6d2d-workload-certs\") pod \"e2e-ipp-a-73f667-openshift-default-7bcfd5c9b9-7fcm7\" (UID: \"0eb54f67-9f54-468e-a88c-ba3fbcaa6d2d\") " pod="openshift-ingress/e2e-ipp-a-73f667-openshift-default-7bcfd5c9b9-7fcm7"
Oct 30 01:09:05.448995 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:05.448952    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0eb54f67-9f54-468e-a88c-ba3fbcaa6d2d-istio-envoy\") pod \"e2e-ipp-a-73f667-openshift-default-7bcfd5c9b9-7fcm7\" (UID: \"0eb54f67-9f54-468e-a88c-ba3fbcaa6d2d\") " pod="openshift-ingress/e2e-ipp-a-73f667-openshift-default-7bcfd5c9b9-7fcm7"
Oct 30 01:09:05.449070 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:05.448991    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0eb54f67-9f54-468e-a88c-ba3fbcaa6d2d-istiod-ca-cert\") pod \"e2e-ipp-a-73f667-openshift-default-7bcfd5c9b9-7fcm7\" (UID: \"0eb54f67-9f54-468e-a88c-ba3fbcaa6d2d\") " pod="openshift-ingress/e2e-ipp-a-73f667-openshift-default-7bcfd5c9b9-7fcm7"
Oct 30 01:09:05.449180 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:05.449148    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5m967\" (UniqueName: \"kubernetes.io/projected/0eb54f67-9f54-468e-a88c-ba3fbcaa6d2d-kube-api-access-5m967\") pod \"e2e-ipp-a-73f667-openshift-default-7bcfd5c9b9-7fcm7\" (UID: \"0eb54f67-9f54-468e-a88c-ba3fbcaa6d2d\") " pod="openshift-ingress/e2e-ipp-a-73f667-openshift-default-7bcfd5c9b9-7fcm7"
Oct 30 01:09:05.449290 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:05.449262    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0eb54f67-9f54-468e-a88c-ba3fbcaa6d2d-istio-podinfo\") pod \"e2e-ipp-a-73f667-openshift-default-7bcfd5c9b9-7fcm7\" (UID: \"0eb54f67-9f54-468e-a88c-ba3fbcaa6d2d\") " pod="openshift-ingress/e2e-ipp-a-73f667-openshift-default-7bcfd5c9b9-7fcm7"
Oct 30 01:09:05.449338 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:05.449324    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0eb54f67-9f54-468e-a88c-ba3fbcaa6d2d-istio-token\") pod \"e2e-ipp-a-73f667-openshift-default-7bcfd5c9b9-7fcm7\" (UID: \"0eb54f67-9f54-468e-a88c-ba3fbcaa6d2d\") " pod="openshift-ingress/e2e-ipp-a-73f667-openshift-default-7bcfd5c9b9-7fcm7"
Oct 30 01:09:05.449497 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:05.449478    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0eb54f67-9f54-468e-a88c-ba3fbcaa6d2d-workload-socket\") pod \"e2e-ipp-a-73f667-openshift-default-7bcfd5c9b9-7fcm7\" (UID: \"0eb54f67-9f54-468e-a88c-ba3fbcaa6d2d\") " pod="openshift-ingress/e2e-ipp-a-73f667-openshift-default-7bcfd5c9b9-7fcm7"
Oct 30 01:09:05.449569 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:05.449523    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0eb54f67-9f54-468e-a88c-ba3fbcaa6d2d-credential-socket\") pod \"e2e-ipp-a-73f667-openshift-default-7bcfd5c9b9-7fcm7\" (UID: \"0eb54f67-9f54-468e-a88c-ba3fbcaa6d2d\") " pod="openshift-ingress/e2e-ipp-a-73f667-openshift-default-7bcfd5c9b9-7fcm7"
Oct 30 01:09:05.449569 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:05.449559    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0eb54f67-9f54-468e-a88c-ba3fbcaa6d2d-istio-data\") pod \"e2e-ipp-a-73f667-openshift-default-7bcfd5c9b9-7fcm7\" (UID: \"0eb54f67-9f54-468e-a88c-ba3fbcaa6d2d\") " pod="openshift-ingress/e2e-ipp-a-73f667-openshift-default-7bcfd5c9b9-7fcm7"
Oct 30 01:09:05.550946 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:05.550910    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5m967\" (UniqueName: \"kubernetes.io/projected/0eb54f67-9f54-468e-a88c-ba3fbcaa6d2d-kube-api-access-5m967\") pod \"e2e-ipp-a-73f667-openshift-default-7bcfd5c9b9-7fcm7\" (UID: \"0eb54f67-9f54-468e-a88c-ba3fbcaa6d2d\") " pod="openshift-ingress/e2e-ipp-a-73f667-openshift-default-7bcfd5c9b9-7fcm7"
Oct 30 01:09:05.551504 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:05.551007    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0eb54f67-9f54-468e-a88c-ba3fbcaa6d2d-istio-podinfo\") pod \"e2e-ipp-a-73f667-openshift-default-7bcfd5c9b9-7fcm7\" (UID: \"0eb54f67-9f54-468e-a88c-ba3fbcaa6d2d\") " pod="openshift-ingress/e2e-ipp-a-73f667-openshift-default-7bcfd5c9b9-7fcm7"
Oct 30 01:09:05.551504 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:05.551048    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0eb54f67-9f54-468e-a88c-ba3fbcaa6d2d-istio-token\") pod \"e2e-ipp-a-73f667-openshift-default-7bcfd5c9b9-7fcm7\" (UID: \"0eb54f67-9f54-468e-a88c-ba3fbcaa6d2d\") " pod="openshift-ingress/e2e-ipp-a-73f667-openshift-default-7bcfd5c9b9-7fcm7"
Oct 30 01:09:05.551504 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:05.551175    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0eb54f67-9f54-468e-a88c-ba3fbcaa6d2d-workload-socket\") pod \"e2e-ipp-a-73f667-openshift-default-7bcfd5c9b9-7fcm7\" (UID: \"0eb54f67-9f54-468e-a88c-ba3fbcaa6d2d\") " pod="openshift-ingress/e2e-ipp-a-73f667-openshift-default-7bcfd5c9b9-7fcm7"
Oct 30 01:09:05.551504 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:05.551212    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0eb54f67-9f54-468e-a88c-ba3fbcaa6d2d-credential-socket\") pod \"e2e-ipp-a-73f667-openshift-default-7bcfd5c9b9-7fcm7\" (UID: \"0eb54f67-9f54-468e-a88c-ba3fbcaa6d2d\") " pod="openshift-ingress/e2e-ipp-a-73f667-openshift-default-7bcfd5c9b9-7fcm7"
Oct 30 01:09:05.551504 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:05.551249    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0eb54f67-9f54-468e-a88c-ba3fbcaa6d2d-istio-data\") pod \"e2e-ipp-a-73f667-openshift-default-7bcfd5c9b9-7fcm7\" (UID: \"0eb54f67-9f54-468e-a88c-ba3fbcaa6d2d\") " pod="openshift-ingress/e2e-ipp-a-73f667-openshift-default-7bcfd5c9b9-7fcm7"
Oct 30 01:09:05.551504 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:05.551301    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0eb54f67-9f54-468e-a88c-ba3fbcaa6d2d-workload-certs\") pod \"e2e-ipp-a-73f667-openshift-default-7bcfd5c9b9-7fcm7\" (UID: \"0eb54f67-9f54-468e-a88c-ba3fbcaa6d2d\") " pod="openshift-ingress/e2e-ipp-a-73f667-openshift-default-7bcfd5c9b9-7fcm7"
Oct 30 01:09:05.551504 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:05.551444    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0eb54f67-9f54-468e-a88c-ba3fbcaa6d2d-istio-envoy\") pod \"e2e-ipp-a-73f667-openshift-default-7bcfd5c9b9-7fcm7\" (UID: \"0eb54f67-9f54-468e-a88c-ba3fbcaa6d2d\") " pod="openshift-ingress/e2e-ipp-a-73f667-openshift-default-7bcfd5c9b9-7fcm7"
Oct 30 01:09:05.551504 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:05.551470    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0eb54f67-9f54-468e-a88c-ba3fbcaa6d2d-istiod-ca-cert\") pod \"e2e-ipp-a-73f667-openshift-default-7bcfd5c9b9-7fcm7\" (UID: \"0eb54f67-9f54-468e-a88c-ba3fbcaa6d2d\") " pod="openshift-ingress/e2e-ipp-a-73f667-openshift-default-7bcfd5c9b9-7fcm7"
Oct 30 01:09:05.551991 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:05.551587    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0eb54f67-9f54-468e-a88c-ba3fbcaa6d2d-workload-socket\") pod \"e2e-ipp-a-73f667-openshift-default-7bcfd5c9b9-7fcm7\" (UID: \"0eb54f67-9f54-468e-a88c-ba3fbcaa6d2d\") " pod="openshift-ingress/e2e-ipp-a-73f667-openshift-default-7bcfd5c9b9-7fcm7"
Oct 30 01:09:05.551991 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:05.551636    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0eb54f67-9f54-468e-a88c-ba3fbcaa6d2d-credential-socket\") pod \"e2e-ipp-a-73f667-openshift-default-7bcfd5c9b9-7fcm7\" (UID: \"0eb54f67-9f54-468e-a88c-ba3fbcaa6d2d\") " pod="openshift-ingress/e2e-ipp-a-73f667-openshift-default-7bcfd5c9b9-7fcm7"
Oct 30 01:09:05.551991 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:05.551705    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0eb54f67-9f54-468e-a88c-ba3fbcaa6d2d-istio-data\") pod \"e2e-ipp-a-73f667-openshift-default-7bcfd5c9b9-7fcm7\" (UID: \"0eb54f67-9f54-468e-a88c-ba3fbcaa6d2d\") " pod="openshift-ingress/e2e-ipp-a-73f667-openshift-default-7bcfd5c9b9-7fcm7"
Oct 30 01:09:05.551991 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:05.551826    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0eb54f67-9f54-468e-a88c-ba3fbcaa6d2d-workload-certs\") pod \"e2e-ipp-a-73f667-openshift-default-7bcfd5c9b9-7fcm7\" (UID: \"0eb54f67-9f54-468e-a88c-ba3fbcaa6d2d\") " pod="openshift-ingress/e2e-ipp-a-73f667-openshift-default-7bcfd5c9b9-7fcm7"
Oct 30 01:09:05.552317 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:05.552296    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0eb54f67-9f54-468e-a88c-ba3fbcaa6d2d-istiod-ca-cert\") pod \"e2e-ipp-a-73f667-openshift-default-7bcfd5c9b9-7fcm7\" (UID: \"0eb54f67-9f54-468e-a88c-ba3fbcaa6d2d\") " pod="openshift-ingress/e2e-ipp-a-73f667-openshift-default-7bcfd5c9b9-7fcm7"
Oct 30 01:09:05.553728 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:05.553706    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0eb54f67-9f54-468e-a88c-ba3fbcaa6d2d-istio-envoy\") pod \"e2e-ipp-a-73f667-openshift-default-7bcfd5c9b9-7fcm7\" (UID: \"0eb54f67-9f54-468e-a88c-ba3fbcaa6d2d\") " pod="openshift-ingress/e2e-ipp-a-73f667-openshift-default-7bcfd5c9b9-7fcm7"
Oct 30 01:09:05.554233 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:05.554217    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0eb54f67-9f54-468e-a88c-ba3fbcaa6d2d-istio-podinfo\") pod \"e2e-ipp-a-73f667-openshift-default-7bcfd5c9b9-7fcm7\" (UID: \"0eb54f67-9f54-468e-a88c-ba3fbcaa6d2d\") " pod="openshift-ingress/e2e-ipp-a-73f667-openshift-default-7bcfd5c9b9-7fcm7"
Oct 30 01:09:05.576607 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:05.576576    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0eb54f67-9f54-468e-a88c-ba3fbcaa6d2d-istio-token\") pod \"e2e-ipp-a-73f667-openshift-default-7bcfd5c9b9-7fcm7\" (UID: \"0eb54f67-9f54-468e-a88c-ba3fbcaa6d2d\") " pod="openshift-ingress/e2e-ipp-a-73f667-openshift-default-7bcfd5c9b9-7fcm7"
Oct 30 01:09:05.576854 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:05.576830    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5m967\" (UniqueName: \"kubernetes.io/projected/0eb54f67-9f54-468e-a88c-ba3fbcaa6d2d-kube-api-access-5m967\") pod \"e2e-ipp-a-73f667-openshift-default-7bcfd5c9b9-7fcm7\" (UID: \"0eb54f67-9f54-468e-a88c-ba3fbcaa6d2d\") " pod="openshift-ingress/e2e-ipp-a-73f667-openshift-default-7bcfd5c9b9-7fcm7"
Oct 30 01:09:05.643693 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:05.643432    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-73f667-openshift-default-7bcfd5c9b9-7fcm7"
Oct 30 01:09:06.197514 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:06.197477    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-73f667-openshift-default-7bcfd5c9b9-7fcm7"]
Oct 30 01:09:06.207264 ip-10-0-128-7 kubenswrapper[5002]: W1030 01:09:06.207227    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0eb54f67_9f54_468e_a88c_ba3fbcaa6d2d.slice/crio-7829bb756c04b45ee557577cd7a23125b209ab7dbf965046e1d5ec87ff3fdddc WatchSource:0}: Error finding container 7829bb756c04b45ee557577cd7a23125b209ab7dbf965046e1d5ec87ff3fdddc: Status 404 returned error can't find the container with id 7829bb756c04b45ee557577cd7a23125b209ab7dbf965046e1d5ec87ff3fdddc
Oct 30 01:09:06.210400 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:06.210345    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:09:06.210506 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:06.210484    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:09:06.210548 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:06.210536    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:09:06.497370 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:06.497296    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e4336b3-ff70-4a4c-b3e6-0669cb308f87" path="/var/lib/kubelet/pods/7e4336b3-ff70-4a4c-b3e6-0669cb308f87/volumes"
Oct 30 01:09:06.497846 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:06.497831    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8adb29be-3746-4500-9889-ea465d532657" path="/var/lib/kubelet/pods/8adb29be-3746-4500-9889-ea465d532657/volumes"
Oct 30 01:09:06.498260 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:06.498248    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fab2daef-ee8c-44f5-8e11-1d11372976a1" path="/var/lib/kubelet/pods/fab2daef-ee8c-44f5-8e11-1d11372976a1/volumes"
Oct 30 01:09:06.540362 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:06.540324    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-73f667-openshift-default-7bcfd5c9b9-7fcm7" event={"ID":"0eb54f67-9f54-468e-a88c-ba3fbcaa6d2d","Type":"ContainerStarted","Data":"c7ea4c5c15a25ce02e96b7fcc3c1265f8042d3ca9174740c5341b3b07aa2fdd1"}
Oct 30 01:09:06.540579 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:06.540460    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-73f667-openshift-default-7bcfd5c9b9-7fcm7" event={"ID":"0eb54f67-9f54-468e-a88c-ba3fbcaa6d2d","Type":"ContainerStarted","Data":"7829bb756c04b45ee557577cd7a23125b209ab7dbf965046e1d5ec87ff3fdddc"}
Oct 30 01:09:06.644044 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:06.644011    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-73f667-openshift-default-7bcfd5c9b9-7fcm7"
Oct 30 01:09:06.665935 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:06.665882    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-73f667-openshift-default-7bcfd5c9b9-7fcm7" podStartSLOduration=1.665867867 podStartE2EDuration="1.665867867s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:09:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:09:06.662237023 +0000 UTC m=+2242.278669903" watchObservedRunningTime="2026-10-30 01:09:06.665867867 +0000 UTC m=+2242.282300798"
Oct 30 01:09:07.651373 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:07.651339    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-73f667-openshift-default-7bcfd5c9b9-7fcm7"
Oct 30 01:09:08.100855 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:08.100816    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-19bc68-59cc54d647-zvgzf"
Oct 30 01:09:08.235638 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:08.235559    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9edfd30b-6f37-4849-9dd8-b5b294c003d7-kube-api-access-p6xqk\" (UniqueName: \"kubernetes.io/projected/9edfd30b-6f37-4849-9dd8-b5b294c003d7-kube-api-access-p6xqk\") pod \"9edfd30b-6f37-4849-9dd8-b5b294c003d7\" (UID: \"9edfd30b-6f37-4849-9dd8-b5b294c003d7\") "
Oct 30 01:09:08.235638 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:08.235638    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9edfd30b-6f37-4849-9dd8-b5b294c003d7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9edfd30b-6f37-4849-9dd8-b5b294c003d7-plugins-config-volume\") pod \"9edfd30b-6f37-4849-9dd8-b5b294c003d7\" (UID: \"9edfd30b-6f37-4849-9dd8-b5b294c003d7\") "
Oct 30 01:09:08.236030 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:08.236006    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9edfd30b-6f37-4849-9dd8-b5b294c003d7-plugins-config-volume" pod "9edfd30b-6f37-4849-9dd8-b5b294c003d7" (UID: "9edfd30b-6f37-4849-9dd8-b5b294c003d7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:09:08.236110 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:08.236091    5002 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9edfd30b-6f37-4849-9dd8-b5b294c003d7-plugins-config-volume\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:09:08.237801 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:08.237780    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9edfd30b-6f37-4849-9dd8-b5b294c003d7-kube-api-access-p6xqk" pod "9edfd30b-6f37-4849-9dd8-b5b294c003d7" (UID: "9edfd30b-6f37-4849-9dd8-b5b294c003d7"). InnerVolumeSpecName "kube-api-access-p6xqk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:09:08.337930 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:08.337893    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p6xqk\" (UniqueName: \"kubernetes.io/projected/9edfd30b-6f37-4849-9dd8-b5b294c003d7-kube-api-access-p6xqk\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:09:08.558305 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:08.558265    5002 generic.go:356] "Generic (PLEG): container finished" podID="9edfd30b-6f37-4849-9dd8-b5b294c003d7" containerID="d9403cef7ffdc8103ed3d3008cb8846b1ea63e616d3edd548cf1323ce66b7c88" exitCode=0
Oct 30 01:09:08.558519 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:08.558310    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-19bc68-59cc54d647-zvgzf"
Oct 30 01:09:08.558519 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:08.558348    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-19bc68-59cc54d647-zvgzf" event={"ID":"9edfd30b-6f37-4849-9dd8-b5b294c003d7","Type":"ContainerDied","Data":"d9403cef7ffdc8103ed3d3008cb8846b1ea63e616d3edd548cf1323ce66b7c88"}
Oct 30 01:09:08.558519 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:08.558421    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-19bc68-59cc54d647-zvgzf" event={"ID":"9edfd30b-6f37-4849-9dd8-b5b294c003d7","Type":"ContainerDied","Data":"fccbea97d001348be473676d73ca0a358c620a51280a97aeac71b330a8ada7b0"}
Oct 30 01:09:08.558519 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:08.558448    5002 scope.go:122] "RemoveContainer" containerID="d9403cef7ffdc8103ed3d3008cb8846b1ea63e616d3edd548cf1323ce66b7c88"
Oct 30 01:09:08.558984 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:08.558958    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-73f667-openshift-default-7bcfd5c9b9-7fcm7"
Oct 30 01:09:08.560228 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:08.560207    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-73f667-openshift-default-7bcfd5c9b9-7fcm7"
Oct 30 01:09:08.569725 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:08.569700    5002 scope.go:122] "RemoveContainer" containerID="d9403cef7ffdc8103ed3d3008cb8846b1ea63e616d3edd548cf1323ce66b7c88"
Oct 30 01:09:08.570089 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:09:08.570065    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9403cef7ffdc8103ed3d3008cb8846b1ea63e616d3edd548cf1323ce66b7c88\": container with ID starting with d9403cef7ffdc8103ed3d3008cb8846b1ea63e616d3edd548cf1323ce66b7c88 not found: ID does not exist" containerID="d9403cef7ffdc8103ed3d3008cb8846b1ea63e616d3edd548cf1323ce66b7c88"
Oct 30 01:09:08.570197 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:08.570099    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9403cef7ffdc8103ed3d3008cb8846b1ea63e616d3edd548cf1323ce66b7c88"} err="failed to get container status \"d9403cef7ffdc8103ed3d3008cb8846b1ea63e616d3edd548cf1323ce66b7c88\": rpc error: code = NotFound desc = could not find container \"d9403cef7ffdc8103ed3d3008cb8846b1ea63e616d3edd548cf1323ce66b7c88\": container with ID starting with d9403cef7ffdc8103ed3d3008cb8846b1ea63e616d3edd548cf1323ce66b7c88 not found: ID does not exist"
Oct 30 01:09:08.616964 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:08.616927    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-19bc68-59cc54d647-zvgzf"]
Oct 30 01:09:08.630326 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:08.630292    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-19bc68-59cc54d647-zvgzf"]
Oct 30 01:09:10.496686 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:10.496650    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9edfd30b-6f37-4849-9dd8-b5b294c003d7" path="/var/lib/kubelet/pods/9edfd30b-6f37-4849-9dd8-b5b294c003d7/volumes"
Oct 30 01:09:12.866985 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:09:12.866945    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbjdq" containerName="extractor"
Oct 30 01:09:12.952809 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:12.952772    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-19bc68-openshift-default-7b6485f77f-s2xkw"]
Oct 30 01:09:12.953065 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:12.953036    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-19bc68-openshift-default-7b6485f77f-s2xkw" podUID="9ae75f2d-c292-43f7-a59c-b1a09261c412" containerName="istio-proxy" containerID="cri-o://08cfeae333f5df907e1e0bc809b425d2345451236767828cf70d27df76f6f3d3" gracePeriod=30
Oct 30 01:09:14.889354 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:14.889308    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w6-009e78-openshift-default-5674f58c7-9fkcq"]
Oct 30 01:09:14.890265 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:14.890243    5002 cpu_manager.go:414] "containerMap: removing container" podUID="9edfd30b-6f37-4849-9dd8-b5b294c003d7" containerName="payload-pre-processing"
Oct 30 01:09:14.890311 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:14.890269    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9edfd30b-6f37-4849-9dd8-b5b294c003d7" containerName="payload-pre-processing"
Oct 30 01:09:14.890482 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:14.890469    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="9edfd30b-6f37-4849-9dd8-b5b294c003d7" containerName="payload-pre-processing"
Oct 30 01:09:14.895059 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:14.895037    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-009e78-openshift-default-5674f58c7-9fkcq"
Oct 30 01:09:14.916394 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:14.916342    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-009e78-openshift-default-5674f58c7-9fkcq"]
Oct 30 01:09:14.990251 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:14.990221    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w6-009e78-openshift-default-dockercfg-c68np\""
Oct 30 01:09:15.025031 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:15.024989    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed247ba4-ebad-4ef0-bffe-5f85a2794a05-workload-socket\") pod \"e2e-shared-a-w6-009e78-openshift-default-5674f58c7-9fkcq\" (UID: \"ed247ba4-ebad-4ef0-bffe-5f85a2794a05\") " pod="openshift-ingress/e2e-shared-a-w6-009e78-openshift-default-5674f58c7-9fkcq"
Oct 30 01:09:15.025248 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:15.025052    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hrbr\" (UniqueName: \"kubernetes.io/projected/ed247ba4-ebad-4ef0-bffe-5f85a2794a05-kube-api-access-7hrbr\") pod \"e2e-shared-a-w6-009e78-openshift-default-5674f58c7-9fkcq\" (UID: \"ed247ba4-ebad-4ef0-bffe-5f85a2794a05\") " pod="openshift-ingress/e2e-shared-a-w6-009e78-openshift-default-5674f58c7-9fkcq"
Oct 30 01:09:15.025248 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:15.025193    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed247ba4-ebad-4ef0-bffe-5f85a2794a05-istio-envoy\") pod \"e2e-shared-a-w6-009e78-openshift-default-5674f58c7-9fkcq\" (UID: \"ed247ba4-ebad-4ef0-bffe-5f85a2794a05\") " pod="openshift-ingress/e2e-shared-a-w6-009e78-openshift-default-5674f58c7-9fkcq"
Oct 30 01:09:15.025248 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:15.025239    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed247ba4-ebad-4ef0-bffe-5f85a2794a05-credential-socket\") pod \"e2e-shared-a-w6-009e78-openshift-default-5674f58c7-9fkcq\" (UID: \"ed247ba4-ebad-4ef0-bffe-5f85a2794a05\") " pod="openshift-ingress/e2e-shared-a-w6-009e78-openshift-default-5674f58c7-9fkcq"
Oct 30 01:09:15.025491 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:15.025356    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed247ba4-ebad-4ef0-bffe-5f85a2794a05-workload-certs\") pod \"e2e-shared-a-w6-009e78-openshift-default-5674f58c7-9fkcq\" (UID: \"ed247ba4-ebad-4ef0-bffe-5f85a2794a05\") " pod="openshift-ingress/e2e-shared-a-w6-009e78-openshift-default-5674f58c7-9fkcq"
Oct 30 01:09:15.025491 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:15.025479    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed247ba4-ebad-4ef0-bffe-5f85a2794a05-istio-token\") pod \"e2e-shared-a-w6-009e78-openshift-default-5674f58c7-9fkcq\" (UID: \"ed247ba4-ebad-4ef0-bffe-5f85a2794a05\") " pod="openshift-ingress/e2e-shared-a-w6-009e78-openshift-default-5674f58c7-9fkcq"
Oct 30 01:09:15.025614 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:15.025590    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed247ba4-ebad-4ef0-bffe-5f85a2794a05-istio-podinfo\") pod \"e2e-shared-a-w6-009e78-openshift-default-5674f58c7-9fkcq\" (UID: \"ed247ba4-ebad-4ef0-bffe-5f85a2794a05\") " pod="openshift-ingress/e2e-shared-a-w6-009e78-openshift-default-5674f58c7-9fkcq"
Oct 30 01:09:15.025672 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:15.025636    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed247ba4-ebad-4ef0-bffe-5f85a2794a05-istiod-ca-cert\") pod \"e2e-shared-a-w6-009e78-openshift-default-5674f58c7-9fkcq\" (UID: \"ed247ba4-ebad-4ef0-bffe-5f85a2794a05\") " pod="openshift-ingress/e2e-shared-a-w6-009e78-openshift-default-5674f58c7-9fkcq"
Oct 30 01:09:15.025737 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:15.025694    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed247ba4-ebad-4ef0-bffe-5f85a2794a05-istio-data\") pod \"e2e-shared-a-w6-009e78-openshift-default-5674f58c7-9fkcq\" (UID: \"ed247ba4-ebad-4ef0-bffe-5f85a2794a05\") " pod="openshift-ingress/e2e-shared-a-w6-009e78-openshift-default-5674f58c7-9fkcq"
Oct 30 01:09:15.126897 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:15.126854    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed247ba4-ebad-4ef0-bffe-5f85a2794a05-istio-podinfo\") pod \"e2e-shared-a-w6-009e78-openshift-default-5674f58c7-9fkcq\" (UID: \"ed247ba4-ebad-4ef0-bffe-5f85a2794a05\") " pod="openshift-ingress/e2e-shared-a-w6-009e78-openshift-default-5674f58c7-9fkcq"
Oct 30 01:09:15.127104 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:15.126913    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed247ba4-ebad-4ef0-bffe-5f85a2794a05-istiod-ca-cert\") pod \"e2e-shared-a-w6-009e78-openshift-default-5674f58c7-9fkcq\" (UID: \"ed247ba4-ebad-4ef0-bffe-5f85a2794a05\") " pod="openshift-ingress/e2e-shared-a-w6-009e78-openshift-default-5674f58c7-9fkcq"
Oct 30 01:09:15.127104 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:15.126954    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed247ba4-ebad-4ef0-bffe-5f85a2794a05-istio-data\") pod \"e2e-shared-a-w6-009e78-openshift-default-5674f58c7-9fkcq\" (UID: \"ed247ba4-ebad-4ef0-bffe-5f85a2794a05\") " pod="openshift-ingress/e2e-shared-a-w6-009e78-openshift-default-5674f58c7-9fkcq"
Oct 30 01:09:15.127104 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:15.127029    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed247ba4-ebad-4ef0-bffe-5f85a2794a05-workload-socket\") pod \"e2e-shared-a-w6-009e78-openshift-default-5674f58c7-9fkcq\" (UID: \"ed247ba4-ebad-4ef0-bffe-5f85a2794a05\") " pod="openshift-ingress/e2e-shared-a-w6-009e78-openshift-default-5674f58c7-9fkcq"
Oct 30 01:09:15.127275 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:15.127105    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hrbr\" (UniqueName: \"kubernetes.io/projected/ed247ba4-ebad-4ef0-bffe-5f85a2794a05-kube-api-access-7hrbr\") pod \"e2e-shared-a-w6-009e78-openshift-default-5674f58c7-9fkcq\" (UID: \"ed247ba4-ebad-4ef0-bffe-5f85a2794a05\") " pod="openshift-ingress/e2e-shared-a-w6-009e78-openshift-default-5674f58c7-9fkcq"
Oct 30 01:09:15.127602 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:15.127576    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed247ba4-ebad-4ef0-bffe-5f85a2794a05-istio-envoy\") pod \"e2e-shared-a-w6-009e78-openshift-default-5674f58c7-9fkcq\" (UID: \"ed247ba4-ebad-4ef0-bffe-5f85a2794a05\") " pod="openshift-ingress/e2e-shared-a-w6-009e78-openshift-default-5674f58c7-9fkcq"
Oct 30 01:09:15.127852 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:15.127808    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed247ba4-ebad-4ef0-bffe-5f85a2794a05-credential-socket\") pod \"e2e-shared-a-w6-009e78-openshift-default-5674f58c7-9fkcq\" (UID: \"ed247ba4-ebad-4ef0-bffe-5f85a2794a05\") " pod="openshift-ingress/e2e-shared-a-w6-009e78-openshift-default-5674f58c7-9fkcq"
Oct 30 01:09:15.128246 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:15.127617    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed247ba4-ebad-4ef0-bffe-5f85a2794a05-workload-socket\") pod \"e2e-shared-a-w6-009e78-openshift-default-5674f58c7-9fkcq\" (UID: \"ed247ba4-ebad-4ef0-bffe-5f85a2794a05\") " pod="openshift-ingress/e2e-shared-a-w6-009e78-openshift-default-5674f58c7-9fkcq"
Oct 30 01:09:15.128454 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:15.127577    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed247ba4-ebad-4ef0-bffe-5f85a2794a05-istio-data\") pod \"e2e-shared-a-w6-009e78-openshift-default-5674f58c7-9fkcq\" (UID: \"ed247ba4-ebad-4ef0-bffe-5f85a2794a05\") " pod="openshift-ingress/e2e-shared-a-w6-009e78-openshift-default-5674f58c7-9fkcq"
Oct 30 01:09:15.128454 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:15.127874    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed247ba4-ebad-4ef0-bffe-5f85a2794a05-istiod-ca-cert\") pod \"e2e-shared-a-w6-009e78-openshift-default-5674f58c7-9fkcq\" (UID: \"ed247ba4-ebad-4ef0-bffe-5f85a2794a05\") " pod="openshift-ingress/e2e-shared-a-w6-009e78-openshift-default-5674f58c7-9fkcq"
Oct 30 01:09:15.128619 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:15.128130    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed247ba4-ebad-4ef0-bffe-5f85a2794a05-credential-socket\") pod \"e2e-shared-a-w6-009e78-openshift-default-5674f58c7-9fkcq\" (UID: \"ed247ba4-ebad-4ef0-bffe-5f85a2794a05\") " pod="openshift-ingress/e2e-shared-a-w6-009e78-openshift-default-5674f58c7-9fkcq"
Oct 30 01:09:15.128619 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:15.128239    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed247ba4-ebad-4ef0-bffe-5f85a2794a05-workload-certs\") pod \"e2e-shared-a-w6-009e78-openshift-default-5674f58c7-9fkcq\" (UID: \"ed247ba4-ebad-4ef0-bffe-5f85a2794a05\") " pod="openshift-ingress/e2e-shared-a-w6-009e78-openshift-default-5674f58c7-9fkcq"
Oct 30 01:09:15.128619 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:15.128492    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed247ba4-ebad-4ef0-bffe-5f85a2794a05-workload-certs\") pod \"e2e-shared-a-w6-009e78-openshift-default-5674f58c7-9fkcq\" (UID: \"ed247ba4-ebad-4ef0-bffe-5f85a2794a05\") " pod="openshift-ingress/e2e-shared-a-w6-009e78-openshift-default-5674f58c7-9fkcq"
Oct 30 01:09:15.128619 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:15.128583    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed247ba4-ebad-4ef0-bffe-5f85a2794a05-istio-token\") pod \"e2e-shared-a-w6-009e78-openshift-default-5674f58c7-9fkcq\" (UID: \"ed247ba4-ebad-4ef0-bffe-5f85a2794a05\") " pod="openshift-ingress/e2e-shared-a-w6-009e78-openshift-default-5674f58c7-9fkcq"
Oct 30 01:09:15.129986 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:15.129952    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed247ba4-ebad-4ef0-bffe-5f85a2794a05-istio-envoy\") pod \"e2e-shared-a-w6-009e78-openshift-default-5674f58c7-9fkcq\" (UID: \"ed247ba4-ebad-4ef0-bffe-5f85a2794a05\") " pod="openshift-ingress/e2e-shared-a-w6-009e78-openshift-default-5674f58c7-9fkcq"
Oct 30 01:09:15.130528 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:15.130501    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed247ba4-ebad-4ef0-bffe-5f85a2794a05-istio-podinfo\") pod \"e2e-shared-a-w6-009e78-openshift-default-5674f58c7-9fkcq\" (UID: \"ed247ba4-ebad-4ef0-bffe-5f85a2794a05\") " pod="openshift-ingress/e2e-shared-a-w6-009e78-openshift-default-5674f58c7-9fkcq"
Oct 30 01:09:15.154093 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:15.154029    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hrbr\" (UniqueName: \"kubernetes.io/projected/ed247ba4-ebad-4ef0-bffe-5f85a2794a05-kube-api-access-7hrbr\") pod \"e2e-shared-a-w6-009e78-openshift-default-5674f58c7-9fkcq\" (UID: \"ed247ba4-ebad-4ef0-bffe-5f85a2794a05\") " pod="openshift-ingress/e2e-shared-a-w6-009e78-openshift-default-5674f58c7-9fkcq"
Oct 30 01:09:15.155919 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:15.155894    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed247ba4-ebad-4ef0-bffe-5f85a2794a05-istio-token\") pod \"e2e-shared-a-w6-009e78-openshift-default-5674f58c7-9fkcq\" (UID: \"ed247ba4-ebad-4ef0-bffe-5f85a2794a05\") " pod="openshift-ingress/e2e-shared-a-w6-009e78-openshift-default-5674f58c7-9fkcq"
Oct 30 01:09:15.211723 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:15.211680    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-009e78-openshift-default-5674f58c7-9fkcq"
Oct 30 01:09:15.610926 ip-10-0-128-7 kubenswrapper[5002]: W1030 01:09:15.610891    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded247ba4_ebad_4ef0_bffe_5f85a2794a05.slice/crio-7032a5d9c27cdc753448ebd7f9d851562d8d31f45f14afc8dbc29178bdabd388 WatchSource:0}: Error finding container 7032a5d9c27cdc753448ebd7f9d851562d8d31f45f14afc8dbc29178bdabd388: Status 404 returned error can't find the container with id 7032a5d9c27cdc753448ebd7f9d851562d8d31f45f14afc8dbc29178bdabd388
Oct 30 01:09:15.611194 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:15.611170    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-009e78-openshift-default-5674f58c7-9fkcq"]
Oct 30 01:09:15.614063 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:15.614021    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:09:15.614163 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:15.614110    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:09:15.614163 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:15.614153    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:09:15.710738 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:15.710703    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-009e78-openshift-default-5674f58c7-9fkcq" event={"ID":"ed247ba4-ebad-4ef0-bffe-5f85a2794a05","Type":"ContainerStarted","Data":"07890073533e0669fe179142ae54a40b9c0c01eb2025c91f7e78c8098ff6c414"}
Oct 30 01:09:15.710874 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:15.710769    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-009e78-openshift-default-5674f58c7-9fkcq" event={"ID":"ed247ba4-ebad-4ef0-bffe-5f85a2794a05","Type":"ContainerStarted","Data":"7032a5d9c27cdc753448ebd7f9d851562d8d31f45f14afc8dbc29178bdabd388"}
Oct 30 01:09:15.765519 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:15.765449    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w6-009e78-openshift-default-5674f58c7-9fkcq" podStartSLOduration=1.7654270890000001 podStartE2EDuration="1.765427089s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:09:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:09:15.758702336 +0000 UTC m=+2251.375135215" watchObservedRunningTime="2026-10-30 01:09:15.765427089 +0000 UTC m=+2251.381859971"
Oct 30 01:09:16.212431 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:16.212389    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-009e78-openshift-default-5674f58c7-9fkcq"
Oct 30 01:09:16.214419 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:16.214368    5002 patch_prober.go:29] interesting pod/e2e-shared-a-w6-009e78-openshift-default-5674f58c7-9fkcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" start-of-body=
Oct 30 01:09:16.214551 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:16.214445    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-009e78-openshift-default-5674f58c7-9fkcq" podUID="ed247ba4-ebad-4ef0-bffe-5f85a2794a05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused"
Oct 30 01:09:17.214405 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:17.212794    5002 patch_prober.go:29] interesting pod/e2e-shared-a-w6-009e78-openshift-default-5674f58c7-9fkcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" start-of-body=
Oct 30 01:09:17.214405 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:17.212859    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-009e78-openshift-default-5674f58c7-9fkcq" podUID="ed247ba4-ebad-4ef0-bffe-5f85a2794a05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused"
Oct 30 01:09:17.985710 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:17.984627    5002 patch_prober.go:29] interesting pod/e2e-inf-b-19bc68-openshift-default-7b6485f77f-s2xkw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" start-of-body=
Oct 30 01:09:17.985710 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:17.984689    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-19bc68-openshift-default-7b6485f77f-s2xkw" podUID="9ae75f2d-c292-43f7-a59c-b1a09261c412" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused"
Oct 30 01:09:18.212549 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:18.212508    5002 patch_prober.go:29] interesting pod/e2e-shared-a-w6-009e78-openshift-default-5674f58c7-9fkcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" start-of-body=
Oct 30 01:09:18.212742 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:18.212584    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-009e78-openshift-default-5674f58c7-9fkcq" podUID="ed247ba4-ebad-4ef0-bffe-5f85a2794a05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused"
Oct 30 01:09:18.325276 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:18.325245    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-19bc68-openshift-default-7b6485f77f-s2xkw"
Oct 30 01:09:18.413240 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:18.413200    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9ae75f2d-c292-43f7-a59c-b1a09261c412-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ae75f2d-c292-43f7-a59c-b1a09261c412-istio-podinfo\") pod \"9ae75f2d-c292-43f7-a59c-b1a09261c412\" (UID: \"9ae75f2d-c292-43f7-a59c-b1a09261c412\") "
Oct 30 01:09:18.413487 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:18.413279    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ae75f2d-c292-43f7-a59c-b1a09261c412-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ae75f2d-c292-43f7-a59c-b1a09261c412-istiod-ca-cert\") pod \"9ae75f2d-c292-43f7-a59c-b1a09261c412\" (UID: \"9ae75f2d-c292-43f7-a59c-b1a09261c412\") "
Oct 30 01:09:18.413487 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:18.413460    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ae75f2d-c292-43f7-a59c-b1a09261c412-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ae75f2d-c292-43f7-a59c-b1a09261c412-credential-socket\") pod \"9ae75f2d-c292-43f7-a59c-b1a09261c412\" (UID: \"9ae75f2d-c292-43f7-a59c-b1a09261c412\") "
Oct 30 01:09:18.413621 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:18.413578    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ae75f2d-c292-43f7-a59c-b1a09261c412-istio-token\" (UniqueName: \"kubernetes.io/projected/9ae75f2d-c292-43f7-a59c-b1a09261c412-istio-token\") pod \"9ae75f2d-c292-43f7-a59c-b1a09261c412\" (UID: \"9ae75f2d-c292-43f7-a59c-b1a09261c412\") "
Oct 30 01:09:18.413679 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:18.413636    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ae75f2d-c292-43f7-a59c-b1a09261c412-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ae75f2d-c292-43f7-a59c-b1a09261c412-istio-envoy\") pod \"9ae75f2d-c292-43f7-a59c-b1a09261c412\" (UID: \"9ae75f2d-c292-43f7-a59c-b1a09261c412\") "
Oct 30 01:09:18.413751 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:18.413673    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ae75f2d-c292-43f7-a59c-b1a09261c412-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ae75f2d-c292-43f7-a59c-b1a09261c412-workload-certs\") pod \"9ae75f2d-c292-43f7-a59c-b1a09261c412\" (UID: \"9ae75f2d-c292-43f7-a59c-b1a09261c412\") "
Oct 30 01:09:18.413751 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:18.413736    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ae75f2d-c292-43f7-a59c-b1a09261c412-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ae75f2d-c292-43f7-a59c-b1a09261c412-istio-data\") pod \"9ae75f2d-c292-43f7-a59c-b1a09261c412\" (UID: \"9ae75f2d-c292-43f7-a59c-b1a09261c412\") "
Oct 30 01:09:18.413751 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:18.413735    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ae75f2d-c292-43f7-a59c-b1a09261c412-credential-socket" pod "9ae75f2d-c292-43f7-a59c-b1a09261c412" (UID: "9ae75f2d-c292-43f7-a59c-b1a09261c412"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:09:18.413941 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:18.413754    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ae75f2d-c292-43f7-a59c-b1a09261c412-istiod-ca-cert" pod "9ae75f2d-c292-43f7-a59c-b1a09261c412" (UID: "9ae75f2d-c292-43f7-a59c-b1a09261c412"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:09:18.413941 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:18.413778    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ae75f2d-c292-43f7-a59c-b1a09261c412-kube-api-access-bv8tn\" (UniqueName: \"kubernetes.io/projected/9ae75f2d-c292-43f7-a59c-b1a09261c412-kube-api-access-bv8tn\") pod \"9ae75f2d-c292-43f7-a59c-b1a09261c412\" (UID: \"9ae75f2d-c292-43f7-a59c-b1a09261c412\") "
Oct 30 01:09:18.414055 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:18.413935    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ae75f2d-c292-43f7-a59c-b1a09261c412-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ae75f2d-c292-43f7-a59c-b1a09261c412-workload-socket\") pod \"9ae75f2d-c292-43f7-a59c-b1a09261c412\" (UID: \"9ae75f2d-c292-43f7-a59c-b1a09261c412\") "
Oct 30 01:09:18.414055 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:18.413964    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ae75f2d-c292-43f7-a59c-b1a09261c412-workload-certs" pod "9ae75f2d-c292-43f7-a59c-b1a09261c412" (UID: "9ae75f2d-c292-43f7-a59c-b1a09261c412"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:09:18.414055 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:18.414023    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ae75f2d-c292-43f7-a59c-b1a09261c412-istio-data" pod "9ae75f2d-c292-43f7-a59c-b1a09261c412" (UID: "9ae75f2d-c292-43f7-a59c-b1a09261c412"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:09:18.414310 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:18.414258    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ae75f2d-c292-43f7-a59c-b1a09261c412-workload-socket" pod "9ae75f2d-c292-43f7-a59c-b1a09261c412" (UID: "9ae75f2d-c292-43f7-a59c-b1a09261c412"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:09:18.414707 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:18.414679    5002 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ae75f2d-c292-43f7-a59c-b1a09261c412-workload-socket\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:09:18.414812 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:18.414712    5002 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ae75f2d-c292-43f7-a59c-b1a09261c412-istiod-ca-cert\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:09:18.414812 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:18.414727    5002 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ae75f2d-c292-43f7-a59c-b1a09261c412-credential-socket\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:09:18.414812 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:18.414741    5002 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ae75f2d-c292-43f7-a59c-b1a09261c412-workload-certs\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:09:18.414812 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:18.414756    5002 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ae75f2d-c292-43f7-a59c-b1a09261c412-istio-data\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:09:18.426713 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:18.416955    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ae75f2d-c292-43f7-a59c-b1a09261c412-istio-envoy" pod "9ae75f2d-c292-43f7-a59c-b1a09261c412" (UID: "9ae75f2d-c292-43f7-a59c-b1a09261c412"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:09:18.426713 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:18.416975    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ae75f2d-c292-43f7-a59c-b1a09261c412-istio-token" pod "9ae75f2d-c292-43f7-a59c-b1a09261c412" (UID: "9ae75f2d-c292-43f7-a59c-b1a09261c412"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:09:18.426713 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:18.417375    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9ae75f2d-c292-43f7-a59c-b1a09261c412-istio-podinfo" pod "9ae75f2d-c292-43f7-a59c-b1a09261c412" (UID: "9ae75f2d-c292-43f7-a59c-b1a09261c412"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:09:18.426713 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:18.423506    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ae75f2d-c292-43f7-a59c-b1a09261c412-kube-api-access-bv8tn" pod "9ae75f2d-c292-43f7-a59c-b1a09261c412" (UID: "9ae75f2d-c292-43f7-a59c-b1a09261c412"). InnerVolumeSpecName "kube-api-access-bv8tn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:09:18.516274 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:18.516246    5002 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ae75f2d-c292-43f7-a59c-b1a09261c412-istio-podinfo\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:09:18.516274 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:18.516272    5002 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ae75f2d-c292-43f7-a59c-b1a09261c412-istio-token\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:09:18.516274 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:18.516281    5002 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ae75f2d-c292-43f7-a59c-b1a09261c412-istio-envoy\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:09:18.516548 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:18.516289    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bv8tn\" (UniqueName: \"kubernetes.io/projected/9ae75f2d-c292-43f7-a59c-b1a09261c412-kube-api-access-bv8tn\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:09:18.736093 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:18.736057    5002 generic.go:356] "Generic (PLEG): container finished" podID="9ae75f2d-c292-43f7-a59c-b1a09261c412" containerID="08cfeae333f5df907e1e0bc809b425d2345451236767828cf70d27df76f6f3d3" exitCode=0
Oct 30 01:09:18.736265 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:18.736107    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-19bc68-openshift-default-7b6485f77f-s2xkw"
Oct 30 01:09:18.736265 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:18.736130    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-19bc68-openshift-default-7b6485f77f-s2xkw" event={"ID":"9ae75f2d-c292-43f7-a59c-b1a09261c412","Type":"ContainerDied","Data":"08cfeae333f5df907e1e0bc809b425d2345451236767828cf70d27df76f6f3d3"}
Oct 30 01:09:18.736265 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:18.736186    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-19bc68-openshift-default-7b6485f77f-s2xkw" event={"ID":"9ae75f2d-c292-43f7-a59c-b1a09261c412","Type":"ContainerDied","Data":"38d7495c4366b067976057983bb979e5dd0483f4608d7c530a53fde118d0e7f7"}
Oct 30 01:09:18.736265 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:18.736213    5002 scope.go:122] "RemoveContainer" containerID="08cfeae333f5df907e1e0bc809b425d2345451236767828cf70d27df76f6f3d3"
Oct 30 01:09:18.747284 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:18.747262    5002 scope.go:122] "RemoveContainer" containerID="08cfeae333f5df907e1e0bc809b425d2345451236767828cf70d27df76f6f3d3"
Oct 30 01:09:18.747626 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:09:18.747600    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08cfeae333f5df907e1e0bc809b425d2345451236767828cf70d27df76f6f3d3\": container with ID starting with 08cfeae333f5df907e1e0bc809b425d2345451236767828cf70d27df76f6f3d3 not found: ID does not exist" containerID="08cfeae333f5df907e1e0bc809b425d2345451236767828cf70d27df76f6f3d3"
Oct 30 01:09:18.747723 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:18.747635    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08cfeae333f5df907e1e0bc809b425d2345451236767828cf70d27df76f6f3d3"} err="failed to get container status \"08cfeae333f5df907e1e0bc809b425d2345451236767828cf70d27df76f6f3d3\": rpc error: code = NotFound desc = could not find container \"08cfeae333f5df907e1e0bc809b425d2345451236767828cf70d27df76f6f3d3\": container with ID starting with 08cfeae333f5df907e1e0bc809b425d2345451236767828cf70d27df76f6f3d3 not found: ID does not exist"
Oct 30 01:09:18.811731 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:18.811601    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-19bc68-openshift-default-7b6485f77f-s2xkw"]
Oct 30 01:09:18.817423 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:18.817368    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-19bc68-openshift-default-7b6485f77f-s2xkw"]
Oct 30 01:09:19.212660 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:19.212622    5002 patch_prober.go:29] interesting pod/e2e-shared-a-w6-009e78-openshift-default-5674f58c7-9fkcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" start-of-body=
Oct 30 01:09:19.212838 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:19.212689    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-009e78-openshift-default-5674f58c7-9fkcq" podUID="ed247ba4-ebad-4ef0-bffe-5f85a2794a05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused"
Oct 30 01:09:20.212099 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:20.212065    5002 patch_prober.go:29] interesting pod/e2e-shared-a-w6-009e78-openshift-default-5674f58c7-9fkcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" start-of-body=
Oct 30 01:09:20.212572 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:20.212121    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-009e78-openshift-default-5674f58c7-9fkcq" podUID="ed247ba4-ebad-4ef0-bffe-5f85a2794a05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused"
Oct 30 01:09:20.221910 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:20.221875    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f64dc0-b4f9c4899-cnjhz"]
Oct 30 01:09:20.222665 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:20.222640    5002 cpu_manager.go:414] "containerMap: removing container" podUID="9ae75f2d-c292-43f7-a59c-b1a09261c412" containerName="istio-proxy"
Oct 30 01:09:20.222732 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:20.222666    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ae75f2d-c292-43f7-a59c-b1a09261c412" containerName="istio-proxy"
Oct 30 01:09:20.222822 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:20.222812    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ae75f2d-c292-43f7-a59c-b1a09261c412" containerName="istio-proxy"
Oct 30 01:09:20.228234 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:20.228209    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f64dc0-b4f9c4899-cnjhz"
Oct 30 01:09:20.245511 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:20.245481    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-f64dc0-dockercfg-zr99j\""
Oct 30 01:09:20.246499 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:20.246473    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f64dc0-b4f9c4899-cnjhz"]
Oct 30 01:09:20.247950 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:20.247919    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fn82f\" (UniqueName: \"kubernetes.io/projected/b02a73f0-b8fd-4b73-8437-c940fc676a67-kube-api-access-fn82f\") pod \"payload-pre-processing-e2e-shared-a-w2-f64dc0-b4f9c4899-cnjhz\" (UID: \"b02a73f0-b8fd-4b73-8437-c940fc676a67\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f64dc0-b4f9c4899-cnjhz"
Oct 30 01:09:20.248300 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:20.248265    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b02a73f0-b8fd-4b73-8437-c940fc676a67-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-f64dc0-b4f9c4899-cnjhz\" (UID: \"b02a73f0-b8fd-4b73-8437-c940fc676a67\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f64dc0-b4f9c4899-cnjhz"
Oct 30 01:09:20.268405 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:20.261785    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-f64dc0\""
Oct 30 01:09:20.401410 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:20.400616    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fn82f\" (UniqueName: \"kubernetes.io/projected/b02a73f0-b8fd-4b73-8437-c940fc676a67-kube-api-access-fn82f\") pod \"payload-pre-processing-e2e-shared-a-w2-f64dc0-b4f9c4899-cnjhz\" (UID: \"b02a73f0-b8fd-4b73-8437-c940fc676a67\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f64dc0-b4f9c4899-cnjhz"
Oct 30 01:09:20.401410 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:20.400833    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b02a73f0-b8fd-4b73-8437-c940fc676a67-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-f64dc0-b4f9c4899-cnjhz\" (UID: \"b02a73f0-b8fd-4b73-8437-c940fc676a67\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f64dc0-b4f9c4899-cnjhz"
Oct 30 01:09:20.405603 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:20.401730    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b02a73f0-b8fd-4b73-8437-c940fc676a67-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-f64dc0-b4f9c4899-cnjhz\" (UID: \"b02a73f0-b8fd-4b73-8437-c940fc676a67\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f64dc0-b4f9c4899-cnjhz"
Oct 30 01:09:20.549333 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:20.547070    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ae75f2d-c292-43f7-a59c-b1a09261c412" path="/var/lib/kubelet/pods/9ae75f2d-c292-43f7-a59c-b1a09261c412/volumes"
Oct 30 01:09:20.592632 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:20.592590    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fn82f\" (UniqueName: \"kubernetes.io/projected/b02a73f0-b8fd-4b73-8437-c940fc676a67-kube-api-access-fn82f\") pod \"payload-pre-processing-e2e-shared-a-w2-f64dc0-b4f9c4899-cnjhz\" (UID: \"b02a73f0-b8fd-4b73-8437-c940fc676a67\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f64dc0-b4f9c4899-cnjhz"
Oct 30 01:09:20.844476 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:20.844373    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f64dc0-b4f9c4899-cnjhz"
Oct 30 01:09:21.224265 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:21.224145    5002 patch_prober.go:29] interesting pod/e2e-shared-a-w6-009e78-openshift-default-5674f58c7-9fkcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" start-of-body=
Oct 30 01:09:21.224265 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:21.224201    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-009e78-openshift-default-5674f58c7-9fkcq" podUID="ed247ba4-ebad-4ef0-bffe-5f85a2794a05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused"
Oct 30 01:09:21.230103 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:21.230075    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f64dc0-b4f9c4899-cnjhz"]
Oct 30 01:09:21.804207 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:21.804165    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f64dc0-b4f9c4899-cnjhz" event={"ID":"b02a73f0-b8fd-4b73-8437-c940fc676a67","Type":"ContainerStarted","Data":"d08a9e109ffb7402396024741367b71a6d600823cc44ad3bc0143da58e28d5c2"}
Oct 30 01:09:21.804450 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:21.804234    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f64dc0-b4f9c4899-cnjhz" event={"ID":"b02a73f0-b8fd-4b73-8437-c940fc676a67","Type":"ContainerStarted","Data":"d5efea925739cd5072eb8cbabaf92be67996001da603f6adf23eeedec353d75a"}
Oct 30 01:09:21.804450 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:21.804264    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f64dc0-b4f9c4899-cnjhz"
Oct 30 01:09:21.858277 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:21.858206    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f64dc0-b4f9c4899-cnjhz" podStartSLOduration=1.858186877 podStartE2EDuration="1.858186877s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:09:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:09:21.849081521 +0000 UTC m=+2257.465514417" watchObservedRunningTime="2026-10-30 01:09:21.858186877 +0000 UTC m=+2257.474619759"
Oct 30 01:09:22.213166 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:22.213127    5002 patch_prober.go:29] interesting pod/e2e-shared-a-w6-009e78-openshift-default-5674f58c7-9fkcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" start-of-body=
Oct 30 01:09:22.213343 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:22.213186    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-009e78-openshift-default-5674f58c7-9fkcq" podUID="ed247ba4-ebad-4ef0-bffe-5f85a2794a05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused"
Oct 30 01:09:23.217411 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:23.217346    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-009e78-openshift-default-5674f58c7-9fkcq"
Oct 30 01:09:23.217826 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:23.217750    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-009e78-openshift-default-5674f58c7-9fkcq"
Oct 30 01:09:23.218513 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:23.218491    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-009e78-openshift-default-5674f58c7-9fkcq"
Oct 30 01:09:32.262287 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:32.262252    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7d8z4"]
Oct 30 01:09:32.826062 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:32.826029    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f64dc0-b4f9c4899-cnjhz"
Oct 30 01:09:39.041355 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:39.041316    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-72d8b73b-khp4t"]
Oct 30 01:09:39.046836 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:39.046814    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72d8b73b-khp4t"
Oct 30 01:09:39.066720 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:39.066689    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-72d8b73b-khp4t"]
Oct 30 01:09:39.078652 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:39.078619    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:09:39.078855 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:39.078672    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:09:39.078855 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:39.078689    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5phsh\""
Oct 30 01:09:39.079091 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:39.078729    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:09:39.190843 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:39.190796    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13cd761c-6990-4b8d-afb1-25fb24baaa8f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-72d8b73b-khp4t\" (UID: \"13cd761c-6990-4b8d-afb1-25fb24baaa8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72d8b73b-khp4t"
Oct 30 01:09:39.191043 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:39.190983    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13cd761c-6990-4b8d-afb1-25fb24baaa8f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-72d8b73b-khp4t\" (UID: \"13cd761c-6990-4b8d-afb1-25fb24baaa8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72d8b73b-khp4t"
Oct 30 01:09:39.191145 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:39.191128    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpf8g\" (UniqueName: \"kubernetes.io/projected/13cd761c-6990-4b8d-afb1-25fb24baaa8f-kube-api-access-hpf8g\") pod \"maas-api-revoke-sub-72d8b73b-khp4t\" (UID: \"13cd761c-6990-4b8d-afb1-25fb24baaa8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72d8b73b-khp4t"
Oct 30 01:09:39.293219 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:39.293124    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13cd761c-6990-4b8d-afb1-25fb24baaa8f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-72d8b73b-khp4t\" (UID: \"13cd761c-6990-4b8d-afb1-25fb24baaa8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72d8b73b-khp4t"
Oct 30 01:09:39.293417 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:39.293288    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpf8g\" (UniqueName: \"kubernetes.io/projected/13cd761c-6990-4b8d-afb1-25fb24baaa8f-kube-api-access-hpf8g\") pod \"maas-api-revoke-sub-72d8b73b-khp4t\" (UID: \"13cd761c-6990-4b8d-afb1-25fb24baaa8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72d8b73b-khp4t"
Oct 30 01:09:39.293493 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:39.293443    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13cd761c-6990-4b8d-afb1-25fb24baaa8f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-72d8b73b-khp4t\" (UID: \"13cd761c-6990-4b8d-afb1-25fb24baaa8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72d8b73b-khp4t"
Oct 30 01:09:39.293991 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:39.293962    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13cd761c-6990-4b8d-afb1-25fb24baaa8f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-72d8b73b-khp4t\" (UID: \"13cd761c-6990-4b8d-afb1-25fb24baaa8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72d8b73b-khp4t"
Oct 30 01:09:39.296097 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:39.296072    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13cd761c-6990-4b8d-afb1-25fb24baaa8f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-72d8b73b-khp4t\" (UID: \"13cd761c-6990-4b8d-afb1-25fb24baaa8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72d8b73b-khp4t"
Oct 30 01:09:39.318249 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:39.318221    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpf8g\" (UniqueName: \"kubernetes.io/projected/13cd761c-6990-4b8d-afb1-25fb24baaa8f-kube-api-access-hpf8g\") pod \"maas-api-revoke-sub-72d8b73b-khp4t\" (UID: \"13cd761c-6990-4b8d-afb1-25fb24baaa8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72d8b73b-khp4t"
Oct 30 01:09:39.368679 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:39.368645    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72d8b73b-khp4t"
Oct 30 01:09:39.469764 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:39.469716    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8f3b80a-prpt5"]
Oct 30 01:09:39.477016 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:39.475980    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8f3b80a-prpt5"
Oct 30 01:09:39.497349 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:39.497311    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8f3b80a-prpt5"]
Oct 30 01:09:39.600434 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:39.600307    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxwxh\" (UniqueName: \"kubernetes.io/projected/ae9c53bd-abf9-4166-ab45-f9d77de3f8e6-kube-api-access-vxwxh\") pod \"maas-api-revoke-sub-e8f3b80a-prpt5\" (UID: \"ae9c53bd-abf9-4166-ab45-f9d77de3f8e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8f3b80a-prpt5"
Oct 30 01:09:39.600434 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:39.600402    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae9c53bd-abf9-4166-ab45-f9d77de3f8e6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e8f3b80a-prpt5\" (UID: \"ae9c53bd-abf9-4166-ab45-f9d77de3f8e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8f3b80a-prpt5"
Oct 30 01:09:39.600738 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:39.600716    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae9c53bd-abf9-4166-ab45-f9d77de3f8e6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e8f3b80a-prpt5\" (UID: \"ae9c53bd-abf9-4166-ab45-f9d77de3f8e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8f3b80a-prpt5"
Oct 30 01:09:39.683124 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:39.682655    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-72d8b73b-khp4t"]
Oct 30 01:09:39.685936 ip-10-0-128-7 kubenswrapper[5002]: W1030 01:09:39.685902    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13cd761c_6990_4b8d_afb1_25fb24baaa8f.slice/crio-df9b387f314b747207eb54aeeb16dee72868356fe34e650a404ed9523e3fc2ee WatchSource:0}: Error finding container df9b387f314b747207eb54aeeb16dee72868356fe34e650a404ed9523e3fc2ee: Status 404 returned error can't find the container with id df9b387f314b747207eb54aeeb16dee72868356fe34e650a404ed9523e3fc2ee
Oct 30 01:09:39.703497 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:39.703468    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae9c53bd-abf9-4166-ab45-f9d77de3f8e6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e8f3b80a-prpt5\" (UID: \"ae9c53bd-abf9-4166-ab45-f9d77de3f8e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8f3b80a-prpt5"
Oct 30 01:09:39.703640 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:39.703617    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxwxh\" (UniqueName: \"kubernetes.io/projected/ae9c53bd-abf9-4166-ab45-f9d77de3f8e6-kube-api-access-vxwxh\") pod \"maas-api-revoke-sub-e8f3b80a-prpt5\" (UID: \"ae9c53bd-abf9-4166-ab45-f9d77de3f8e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8f3b80a-prpt5"
Oct 30 01:09:39.703814 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:39.703681    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae9c53bd-abf9-4166-ab45-f9d77de3f8e6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e8f3b80a-prpt5\" (UID: \"ae9c53bd-abf9-4166-ab45-f9d77de3f8e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8f3b80a-prpt5"
Oct 30 01:09:39.704187 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:39.704160    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae9c53bd-abf9-4166-ab45-f9d77de3f8e6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e8f3b80a-prpt5\" (UID: \"ae9c53bd-abf9-4166-ab45-f9d77de3f8e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8f3b80a-prpt5"
Oct 30 01:09:39.706264 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:39.706242    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae9c53bd-abf9-4166-ab45-f9d77de3f8e6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e8f3b80a-prpt5\" (UID: \"ae9c53bd-abf9-4166-ab45-f9d77de3f8e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8f3b80a-prpt5"
Oct 30 01:09:39.729890 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:39.729860    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxwxh\" (UniqueName: \"kubernetes.io/projected/ae9c53bd-abf9-4166-ab45-f9d77de3f8e6-kube-api-access-vxwxh\") pod \"maas-api-revoke-sub-e8f3b80a-prpt5\" (UID: \"ae9c53bd-abf9-4166-ab45-f9d77de3f8e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8f3b80a-prpt5"
Oct 30 01:09:39.812831 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:39.812789    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8f3b80a-prpt5"
Oct 30 01:09:40.205962 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:40.205931    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0eae29ca-mmdxp"]
Oct 30 01:09:40.219589 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:40.215059    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72d8b73b-khp4t" event={"ID":"13cd761c-6990-4b8d-afb1-25fb24baaa8f","Type":"ContainerStarted","Data":"df9b387f314b747207eb54aeeb16dee72868356fe34e650a404ed9523e3fc2ee"}
Oct 30 01:09:40.221928 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:40.221896    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0eae29ca-mmdxp"]
Oct 30 01:09:40.226347 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:40.226278    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8f3b80a-prpt5"]
Oct 30 01:09:40.497964 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:40.497926    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2a2f913-82ba-406c-9cd8-3fb4d39d0763" path="/var/lib/kubelet/pods/e2a2f913-82ba-406c-9cd8-3fb4d39d0763/volumes"
Oct 30 01:09:41.224997 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:41.224957    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8f3b80a-prpt5" event={"ID":"ae9c53bd-abf9-4166-ab45-f9d77de3f8e6","Type":"ContainerStarted","Data":"fe4a875ed48a56d8c739283d5856f58dbb4235f6424e3315bfd8cfcd4ccb23aa"}
Oct 30 01:09:41.225494 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:41.225023    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8f3b80a-prpt5" event={"ID":"ae9c53bd-abf9-4166-ab45-f9d77de3f8e6","Type":"ContainerStarted","Data":"7d6825cab128966f3920bd8b79ff5e501be47e3019a26671809b2758b65930ab"}
Oct 30 01:09:41.226678 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:41.226651    5002 generic.go:356] "Generic (PLEG): container finished" podID="13cd761c-6990-4b8d-afb1-25fb24baaa8f" containerID="3b5761da010b184e6034f5db5338864a803f372d99a7e16a4d5bd3c498358def" exitCode=0
Oct 30 01:09:41.226773 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:41.226731    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72d8b73b-khp4t" event={"ID":"13cd761c-6990-4b8d-afb1-25fb24baaa8f","Type":"ContainerDied","Data":"3b5761da010b184e6034f5db5338864a803f372d99a7e16a4d5bd3c498358def"}
Oct 30 01:09:41.270322 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:41.270268    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8f3b80a-prpt5" podStartSLOduration=2.270248003 podStartE2EDuration="2.270248003s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:09:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:09:41.269220157 +0000 UTC m=+2276.885653036" watchObservedRunningTime="2026-10-30 01:09:41.270248003 +0000 UTC m=+2276.886680884"
Oct 30 01:09:42.236396 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:42.236345    5002 generic.go:356] "Generic (PLEG): container finished" podID="ae9c53bd-abf9-4166-ab45-f9d77de3f8e6" containerID="fe4a875ed48a56d8c739283d5856f58dbb4235f6424e3315bfd8cfcd4ccb23aa" exitCode=0
Oct 30 01:09:42.236892 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:42.236436    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8f3b80a-prpt5" event={"ID":"ae9c53bd-abf9-4166-ab45-f9d77de3f8e6","Type":"ContainerDied","Data":"fe4a875ed48a56d8c739283d5856f58dbb4235f6424e3315bfd8cfcd4ccb23aa"}
Oct 30 01:09:42.425150 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:42.425124    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72d8b73b-khp4t"
Oct 30 01:09:42.606708 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:42.600046    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/13cd761c-6990-4b8d-afb1-25fb24baaa8f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13cd761c-6990-4b8d-afb1-25fb24baaa8f-maas-api-xks-ca\") pod \"13cd761c-6990-4b8d-afb1-25fb24baaa8f\" (UID: \"13cd761c-6990-4b8d-afb1-25fb24baaa8f\") "
Oct 30 01:09:42.606708 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:42.600175    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13cd761c-6990-4b8d-afb1-25fb24baaa8f-kube-api-access-hpf8g\" (UniqueName: \"kubernetes.io/projected/13cd761c-6990-4b8d-afb1-25fb24baaa8f-kube-api-access-hpf8g\") pod \"13cd761c-6990-4b8d-afb1-25fb24baaa8f\" (UID: \"13cd761c-6990-4b8d-afb1-25fb24baaa8f\") "
Oct 30 01:09:42.606708 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:42.600324    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13cd761c-6990-4b8d-afb1-25fb24baaa8f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13cd761c-6990-4b8d-afb1-25fb24baaa8f-maas-api-openshift-ca\") pod \"13cd761c-6990-4b8d-afb1-25fb24baaa8f\" (UID: \"13cd761c-6990-4b8d-afb1-25fb24baaa8f\") "
Oct 30 01:09:42.617440 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:42.607225    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13cd761c-6990-4b8d-afb1-25fb24baaa8f-maas-api-xks-ca" pod "13cd761c-6990-4b8d-afb1-25fb24baaa8f" (UID: "13cd761c-6990-4b8d-afb1-25fb24baaa8f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:09:42.636407 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:42.621782    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13cd761c-6990-4b8d-afb1-25fb24baaa8f-maas-api-openshift-ca" pod "13cd761c-6990-4b8d-afb1-25fb24baaa8f" (UID: "13cd761c-6990-4b8d-afb1-25fb24baaa8f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:09:42.636407 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:42.626796    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13cd761c-6990-4b8d-afb1-25fb24baaa8f-kube-api-access-hpf8g" pod "13cd761c-6990-4b8d-afb1-25fb24baaa8f" (UID: "13cd761c-6990-4b8d-afb1-25fb24baaa8f"). InnerVolumeSpecName "kube-api-access-hpf8g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:09:42.636407 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:42.628771    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13cd761c-6990-4b8d-afb1-25fb24baaa8f-maas-api-xks-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:09:42.730306 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:42.730265    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13cd761c-6990-4b8d-afb1-25fb24baaa8f-maas-api-openshift-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:09:42.730306 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:42.730300    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hpf8g\" (UniqueName: \"kubernetes.io/projected/13cd761c-6990-4b8d-afb1-25fb24baaa8f-kube-api-access-hpf8g\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:09:42.864604 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:09:42.864511    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbjdq" containerName="extractor"
Oct 30 01:09:43.245086 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:43.245054    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72d8b73b-khp4t"
Oct 30 01:09:43.245654 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:43.245087    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72d8b73b-khp4t" event={"ID":"13cd761c-6990-4b8d-afb1-25fb24baaa8f","Type":"ContainerDied","Data":"df9b387f314b747207eb54aeeb16dee72868356fe34e650a404ed9523e3fc2ee"}
Oct 30 01:09:43.245654 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:43.245134    5002 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="df9b387f314b747207eb54aeeb16dee72868356fe34e650a404ed9523e3fc2ee"
Oct 30 01:09:43.425392 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:43.425348    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8f3b80a-prpt5"
Oct 30 01:09:43.440364 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:43.440335    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ae9c53bd-abf9-4166-ab45-f9d77de3f8e6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae9c53bd-abf9-4166-ab45-f9d77de3f8e6-maas-api-xks-ca\") pod \"ae9c53bd-abf9-4166-ab45-f9d77de3f8e6\" (UID: \"ae9c53bd-abf9-4166-ab45-f9d77de3f8e6\") "
Oct 30 01:09:43.440526 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:43.440516    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae9c53bd-abf9-4166-ab45-f9d77de3f8e6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae9c53bd-abf9-4166-ab45-f9d77de3f8e6-maas-api-openshift-ca\") pod \"ae9c53bd-abf9-4166-ab45-f9d77de3f8e6\" (UID: \"ae9c53bd-abf9-4166-ab45-f9d77de3f8e6\") "
Oct 30 01:09:43.440592 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:43.440580    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae9c53bd-abf9-4166-ab45-f9d77de3f8e6-kube-api-access-vxwxh\" (UniqueName: \"kubernetes.io/projected/ae9c53bd-abf9-4166-ab45-f9d77de3f8e6-kube-api-access-vxwxh\") pod \"ae9c53bd-abf9-4166-ab45-f9d77de3f8e6\" (UID: \"ae9c53bd-abf9-4166-ab45-f9d77de3f8e6\") "
Oct 30 01:09:43.440845 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:43.440822    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae9c53bd-abf9-4166-ab45-f9d77de3f8e6-maas-api-openshift-ca" pod "ae9c53bd-abf9-4166-ab45-f9d77de3f8e6" (UID: "ae9c53bd-abf9-4166-ab45-f9d77de3f8e6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:09:43.441119 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:43.441101    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae9c53bd-abf9-4166-ab45-f9d77de3f8e6-maas-api-openshift-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:09:43.442575 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:43.442545    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae9c53bd-abf9-4166-ab45-f9d77de3f8e6-maas-api-xks-ca" pod "ae9c53bd-abf9-4166-ab45-f9d77de3f8e6" (UID: "ae9c53bd-abf9-4166-ab45-f9d77de3f8e6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:09:43.442681 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:43.442655    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae9c53bd-abf9-4166-ab45-f9d77de3f8e6-kube-api-access-vxwxh" pod "ae9c53bd-abf9-4166-ab45-f9d77de3f8e6" (UID: "ae9c53bd-abf9-4166-ab45-f9d77de3f8e6"). InnerVolumeSpecName "kube-api-access-vxwxh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:09:43.542111 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:43.542026    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae9c53bd-abf9-4166-ab45-f9d77de3f8e6-maas-api-xks-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:09:43.542111 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:43.542066    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vxwxh\" (UniqueName: \"kubernetes.io/projected/ae9c53bd-abf9-4166-ab45-f9d77de3f8e6-kube-api-access-vxwxh\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:09:44.255975 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:44.255935    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8f3b80a-prpt5" event={"ID":"ae9c53bd-abf9-4166-ab45-f9d77de3f8e6","Type":"ContainerDied","Data":"7d6825cab128966f3920bd8b79ff5e501be47e3019a26671809b2758b65930ab"}
Oct 30 01:09:44.255975 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:44.255960    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8f3b80a-prpt5"
Oct 30 01:09:44.256489 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:44.256000    5002 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7d6825cab128966f3920bd8b79ff5e501be47e3019a26671809b2758b65930ab"
Oct 30 01:09:47.929103 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:47.929063    5002 scope.go:122] "RemoveContainer" containerID="1e7b12d93baea8c06d9e54fc2de5d6d472a12460c643a242b77a4099c94600f3"
Oct 30 01:09:47.940575 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:47.940551    5002 scope.go:122] "RemoveContainer" containerID="4013367e10d219e9261f7d3d6d66784279bf130cb51cc7286e141144d06b9854"
Oct 30 01:09:47.950517 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:47.950495    5002 scope.go:122] "RemoveContainer" containerID="8aa4f34cd77904f69d199495513315de890394b15aade7ecaf8c503e29ced588"
Oct 30 01:09:47.962024 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:47.961997    5002 scope.go:122] "RemoveContainer" containerID="42100265a44327bd748e2466a4c867c3d416211077d70d4bc0b7291d3e3cc1d2"
Oct 30 01:09:49.393195 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:49.393159    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7edda4b3-jnr46"]
Oct 30 01:09:49.393844 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:49.393825    5002 cpu_manager.go:414] "containerMap: removing container" podUID="ae9c53bd-abf9-4166-ab45-f9d77de3f8e6" containerName="revoke-keys"
Oct 30 01:09:49.393895 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:49.393845    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae9c53bd-abf9-4166-ab45-f9d77de3f8e6" containerName="revoke-keys"
Oct 30 01:09:49.393895 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:49.393887    5002 cpu_manager.go:414] "containerMap: removing container" podUID="13cd761c-6990-4b8d-afb1-25fb24baaa8f" containerName="revoke-keys"
Oct 30 01:09:49.393895 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:49.393892    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13cd761c-6990-4b8d-afb1-25fb24baaa8f" containerName="revoke-keys"
Oct 30 01:09:49.393998 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:49.393990    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="13cd761c-6990-4b8d-afb1-25fb24baaa8f" containerName="revoke-keys"
Oct 30 01:09:49.394036 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:49.394003    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae9c53bd-abf9-4166-ab45-f9d77de3f8e6" containerName="revoke-keys"
Oct 30 01:09:49.399279 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:49.399234    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7edda4b3-jnr46"
Oct 30 01:09:49.416193 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:49.416148    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:09:49.417502 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:49.417479    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7edda4b3-jnr46"]
Oct 30 01:09:49.428314 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:49.428285    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5phsh\""
Oct 30 01:09:49.428533 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:49.428361    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:09:49.430076 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:49.430055    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:09:49.438112 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:49.438081    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2kglx\" (UniqueName: \"kubernetes.io/projected/25e6159a-609a-483f-b679-3e64db06178c-kube-api-access-2kglx\") pod \"maas-api-revoke-sub-7edda4b3-jnr46\" (UID: \"25e6159a-609a-483f-b679-3e64db06178c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7edda4b3-jnr46"
Oct 30 01:09:49.438455 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:49.438429    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25e6159a-609a-483f-b679-3e64db06178c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7edda4b3-jnr46\" (UID: \"25e6159a-609a-483f-b679-3e64db06178c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7edda4b3-jnr46"
Oct 30 01:09:49.438560 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:49.438513    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25e6159a-609a-483f-b679-3e64db06178c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7edda4b3-jnr46\" (UID: \"25e6159a-609a-483f-b679-3e64db06178c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7edda4b3-jnr46"
Oct 30 01:09:49.539729 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:49.539688    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kglx\" (UniqueName: \"kubernetes.io/projected/25e6159a-609a-483f-b679-3e64db06178c-kube-api-access-2kglx\") pod \"maas-api-revoke-sub-7edda4b3-jnr46\" (UID: \"25e6159a-609a-483f-b679-3e64db06178c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7edda4b3-jnr46"
Oct 30 01:09:49.539911 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:49.539863    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25e6159a-609a-483f-b679-3e64db06178c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7edda4b3-jnr46\" (UID: \"25e6159a-609a-483f-b679-3e64db06178c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7edda4b3-jnr46"
Oct 30 01:09:49.539911 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:49.539898    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25e6159a-609a-483f-b679-3e64db06178c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7edda4b3-jnr46\" (UID: \"25e6159a-609a-483f-b679-3e64db06178c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7edda4b3-jnr46"
Oct 30 01:09:49.540582 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:49.540551    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25e6159a-609a-483f-b679-3e64db06178c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7edda4b3-jnr46\" (UID: \"25e6159a-609a-483f-b679-3e64db06178c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7edda4b3-jnr46"
Oct 30 01:09:49.542289 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:49.542271    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25e6159a-609a-483f-b679-3e64db06178c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7edda4b3-jnr46\" (UID: \"25e6159a-609a-483f-b679-3e64db06178c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7edda4b3-jnr46"
Oct 30 01:09:49.568277 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:49.568247    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kglx\" (UniqueName: \"kubernetes.io/projected/25e6159a-609a-483f-b679-3e64db06178c-kube-api-access-2kglx\") pod \"maas-api-revoke-sub-7edda4b3-jnr46\" (UID: \"25e6159a-609a-483f-b679-3e64db06178c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7edda4b3-jnr46"
Oct 30 01:09:49.715548 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:49.715513    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7edda4b3-jnr46"
Oct 30 01:09:49.884814 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:49.884775    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95e2da69-n7tqv"]
Oct 30 01:09:49.890188 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:49.890166    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95e2da69-n7tqv"
Oct 30 01:09:49.909103 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:49.909072    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95e2da69-n7tqv"]
Oct 30 01:09:49.946705 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:49.946661    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdxm6\" (UniqueName: \"kubernetes.io/projected/ec788b52-0c21-48a4-b329-27ce6a65433a-kube-api-access-qdxm6\") pod \"maas-api-revoke-sub-95e2da69-n7tqv\" (UID: \"ec788b52-0c21-48a4-b329-27ce6a65433a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95e2da69-n7tqv"
Oct 30 01:09:49.946909 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:49.946793    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec788b52-0c21-48a4-b329-27ce6a65433a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-95e2da69-n7tqv\" (UID: \"ec788b52-0c21-48a4-b329-27ce6a65433a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95e2da69-n7tqv"
Oct 30 01:09:49.947119 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:49.947007    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec788b52-0c21-48a4-b329-27ce6a65433a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-95e2da69-n7tqv\" (UID: \"ec788b52-0c21-48a4-b329-27ce6a65433a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95e2da69-n7tqv"
Oct 30 01:09:50.025607 ip-10-0-128-7 kubenswrapper[5002]: W1030 01:09:50.025571    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25e6159a_609a_483f_b679_3e64db06178c.slice/crio-de126819770468e78cb663dacb75001885bb8592f4c88f48681bb4122bb44baa WatchSource:0}: Error finding container de126819770468e78cb663dacb75001885bb8592f4c88f48681bb4122bb44baa: Status 404 returned error can't find the container with id de126819770468e78cb663dacb75001885bb8592f4c88f48681bb4122bb44baa
Oct 30 01:09:50.026041 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:50.026017    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7edda4b3-jnr46"]
Oct 30 01:09:50.048599 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:50.048574    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec788b52-0c21-48a4-b329-27ce6a65433a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-95e2da69-n7tqv\" (UID: \"ec788b52-0c21-48a4-b329-27ce6a65433a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95e2da69-n7tqv"
Oct 30 01:09:50.048726 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:50.048658    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdxm6\" (UniqueName: \"kubernetes.io/projected/ec788b52-0c21-48a4-b329-27ce6a65433a-kube-api-access-qdxm6\") pod \"maas-api-revoke-sub-95e2da69-n7tqv\" (UID: \"ec788b52-0c21-48a4-b329-27ce6a65433a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95e2da69-n7tqv"
Oct 30 01:09:50.048793 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:50.048738    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec788b52-0c21-48a4-b329-27ce6a65433a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-95e2da69-n7tqv\" (UID: \"ec788b52-0c21-48a4-b329-27ce6a65433a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95e2da69-n7tqv"
Oct 30 01:09:50.049536 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:50.049510    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec788b52-0c21-48a4-b329-27ce6a65433a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-95e2da69-n7tqv\" (UID: \"ec788b52-0c21-48a4-b329-27ce6a65433a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95e2da69-n7tqv"
Oct 30 01:09:50.051427 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:50.051405    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec788b52-0c21-48a4-b329-27ce6a65433a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-95e2da69-n7tqv\" (UID: \"ec788b52-0c21-48a4-b329-27ce6a65433a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95e2da69-n7tqv"
Oct 30 01:09:50.075509 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:50.075474    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdxm6\" (UniqueName: \"kubernetes.io/projected/ec788b52-0c21-48a4-b329-27ce6a65433a-kube-api-access-qdxm6\") pod \"maas-api-revoke-sub-95e2da69-n7tqv\" (UID: \"ec788b52-0c21-48a4-b329-27ce6a65433a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95e2da69-n7tqv"
Oct 30 01:09:50.206001 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:50.205968    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95e2da69-n7tqv"
Oct 30 01:09:50.311688 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:50.311653    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7edda4b3-jnr46" event={"ID":"25e6159a-609a-483f-b679-3e64db06178c","Type":"ContainerStarted","Data":"de126819770468e78cb663dacb75001885bb8592f4c88f48681bb4122bb44baa"}
Oct 30 01:09:50.558270 ip-10-0-128-7 kubenswrapper[5002]: W1030 01:09:50.558193    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec788b52_0c21_48a4_b329_27ce6a65433a.slice/crio-6774328160b192d8dbebbfdc5c0776a53c64d62c193d6404e68b89ebb4c04eb4 WatchSource:0}: Error finding container 6774328160b192d8dbebbfdc5c0776a53c64d62c193d6404e68b89ebb4c04eb4: Status 404 returned error can't find the container with id 6774328160b192d8dbebbfdc5c0776a53c64d62c193d6404e68b89ebb4c04eb4
Oct 30 01:09:50.570863 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:50.570828    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95e2da69-n7tqv"]
Oct 30 01:09:51.352412 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:51.347931    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95e2da69-n7tqv" event={"ID":"ec788b52-0c21-48a4-b329-27ce6a65433a","Type":"ContainerStarted","Data":"849a7fedf6cc4a703eb8a93ffffbcc279e421b0a09a239d0e21dc998aecfa174"}
Oct 30 01:09:51.352412 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:51.347996    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95e2da69-n7tqv" event={"ID":"ec788b52-0c21-48a4-b329-27ce6a65433a","Type":"ContainerStarted","Data":"6774328160b192d8dbebbfdc5c0776a53c64d62c193d6404e68b89ebb4c04eb4"}
Oct 30 01:09:51.352412 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:51.349660    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7edda4b3-jnr46" event={"ID":"25e6159a-609a-483f-b679-3e64db06178c","Type":"ContainerStarted","Data":"59ca3a29d72645f2b65acf6309c4ce820a7d20cf2ab8b5d51f0e80dfbd80127e"}
Oct 30 01:09:51.393147 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:51.393088    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95e2da69-n7tqv" podStartSLOduration=2.393074708 podStartE2EDuration="2.393074708s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:09:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:09:51.392086155 +0000 UTC m=+2287.008519053" watchObservedRunningTime="2026-10-30 01:09:51.393074708 +0000 UTC m=+2287.009507587"
Oct 30 01:09:52.359955 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:52.359918    5002 generic.go:356] "Generic (PLEG): container finished" podID="ec788b52-0c21-48a4-b329-27ce6a65433a" containerID="849a7fedf6cc4a703eb8a93ffffbcc279e421b0a09a239d0e21dc998aecfa174" exitCode=0
Oct 30 01:09:52.360468 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:52.360000    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95e2da69-n7tqv" event={"ID":"ec788b52-0c21-48a4-b329-27ce6a65433a","Type":"ContainerDied","Data":"849a7fedf6cc4a703eb8a93ffffbcc279e421b0a09a239d0e21dc998aecfa174"}
Oct 30 01:09:52.361973 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:52.361947    5002 generic.go:356] "Generic (PLEG): container finished" podID="25e6159a-609a-483f-b679-3e64db06178c" containerID="59ca3a29d72645f2b65acf6309c4ce820a7d20cf2ab8b5d51f0e80dfbd80127e" exitCode=0
Oct 30 01:09:52.362075 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:52.361975    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7edda4b3-jnr46" event={"ID":"25e6159a-609a-483f-b679-3e64db06178c","Type":"ContainerDied","Data":"59ca3a29d72645f2b65acf6309c4ce820a7d20cf2ab8b5d51f0e80dfbd80127e"}
Oct 30 01:09:52.575138 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:52.575098    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7edda4b3-jnr46"
Oct 30 01:09:52.657391 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:52.657283    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25e6159a-609a-483f-b679-3e64db06178c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25e6159a-609a-483f-b679-3e64db06178c-maas-api-openshift-ca\") pod \"25e6159a-609a-483f-b679-3e64db06178c\" (UID: \"25e6159a-609a-483f-b679-3e64db06178c\") "
Oct 30 01:09:52.657569 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:52.657529    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/25e6159a-609a-483f-b679-3e64db06178c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25e6159a-609a-483f-b679-3e64db06178c-maas-api-xks-ca\") pod \"25e6159a-609a-483f-b679-3e64db06178c\" (UID: \"25e6159a-609a-483f-b679-3e64db06178c\") "
Oct 30 01:09:52.657687 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:52.657672    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25e6159a-609a-483f-b679-3e64db06178c-kube-api-access-2kglx\" (UniqueName: \"kubernetes.io/projected/25e6159a-609a-483f-b679-3e64db06178c-kube-api-access-2kglx\") pod \"25e6159a-609a-483f-b679-3e64db06178c\" (UID: \"25e6159a-609a-483f-b679-3e64db06178c\") "
Oct 30 01:09:52.657746 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:52.657681    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25e6159a-609a-483f-b679-3e64db06178c-maas-api-openshift-ca" pod "25e6159a-609a-483f-b679-3e64db06178c" (UID: "25e6159a-609a-483f-b679-3e64db06178c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:09:52.658171 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:52.658151    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25e6159a-609a-483f-b679-3e64db06178c-maas-api-openshift-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:09:52.659792 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:52.659768    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25e6159a-609a-483f-b679-3e64db06178c-maas-api-xks-ca" pod "25e6159a-609a-483f-b679-3e64db06178c" (UID: "25e6159a-609a-483f-b679-3e64db06178c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:09:52.659979 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:52.659956    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25e6159a-609a-483f-b679-3e64db06178c-kube-api-access-2kglx" pod "25e6159a-609a-483f-b679-3e64db06178c" (UID: "25e6159a-609a-483f-b679-3e64db06178c"). InnerVolumeSpecName "kube-api-access-2kglx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:09:52.760295 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:52.760256    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2kglx\" (UniqueName: \"kubernetes.io/projected/25e6159a-609a-483f-b679-3e64db06178c-kube-api-access-2kglx\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:09:52.760295 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:52.760291    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25e6159a-609a-483f-b679-3e64db06178c-maas-api-xks-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:09:53.372827 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:53.372794    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7edda4b3-jnr46"
Oct 30 01:09:53.373305 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:53.372821    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7edda4b3-jnr46" event={"ID":"25e6159a-609a-483f-b679-3e64db06178c","Type":"ContainerDied","Data":"de126819770468e78cb663dacb75001885bb8592f4c88f48681bb4122bb44baa"}
Oct 30 01:09:53.373305 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:53.372874    5002 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="de126819770468e78cb663dacb75001885bb8592f4c88f48681bb4122bb44baa"
Oct 30 01:09:53.596169 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:53.596135    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95e2da69-n7tqv"
Oct 30 01:09:53.686083 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:53.686052    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec788b52-0c21-48a4-b329-27ce6a65433a-kube-api-access-qdxm6\" (UniqueName: \"kubernetes.io/projected/ec788b52-0c21-48a4-b329-27ce6a65433a-kube-api-access-qdxm6\") pod \"ec788b52-0c21-48a4-b329-27ce6a65433a\" (UID: \"ec788b52-0c21-48a4-b329-27ce6a65433a\") "
Oct 30 01:09:53.686289 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:53.686216    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ec788b52-0c21-48a4-b329-27ce6a65433a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec788b52-0c21-48a4-b329-27ce6a65433a-maas-api-xks-ca\") pod \"ec788b52-0c21-48a4-b329-27ce6a65433a\" (UID: \"ec788b52-0c21-48a4-b329-27ce6a65433a\") "
Oct 30 01:09:53.686440 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:53.686376    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec788b52-0c21-48a4-b329-27ce6a65433a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec788b52-0c21-48a4-b329-27ce6a65433a-maas-api-openshift-ca\") pod \"ec788b52-0c21-48a4-b329-27ce6a65433a\" (UID: \"ec788b52-0c21-48a4-b329-27ce6a65433a\") "
Oct 30 01:09:53.686971 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:53.686943    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec788b52-0c21-48a4-b329-27ce6a65433a-maas-api-openshift-ca" pod "ec788b52-0c21-48a4-b329-27ce6a65433a" (UID: "ec788b52-0c21-48a4-b329-27ce6a65433a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:09:53.687151 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:53.687132    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec788b52-0c21-48a4-b329-27ce6a65433a-maas-api-openshift-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:09:53.689257 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:53.689225    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec788b52-0c21-48a4-b329-27ce6a65433a-maas-api-xks-ca" pod "ec788b52-0c21-48a4-b329-27ce6a65433a" (UID: "ec788b52-0c21-48a4-b329-27ce6a65433a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:09:53.691354 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:53.691326    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec788b52-0c21-48a4-b329-27ce6a65433a-kube-api-access-qdxm6" pod "ec788b52-0c21-48a4-b329-27ce6a65433a" (UID: "ec788b52-0c21-48a4-b329-27ce6a65433a"). InnerVolumeSpecName "kube-api-access-qdxm6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:09:53.788997 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:53.788955    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qdxm6\" (UniqueName: \"kubernetes.io/projected/ec788b52-0c21-48a4-b329-27ce6a65433a-kube-api-access-qdxm6\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:09:53.788997 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:53.788991    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec788b52-0c21-48a4-b329-27ce6a65433a-maas-api-xks-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:09:54.085640 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:54.085556    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-j9nfb"]
Oct 30 01:09:54.085797 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:54.085757    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-j9nfb" podUID="c72b5051-4287-4298-b179-4e39b2535eb5" containerName="istio-proxy" containerID="cri-o://7ee356c780ab592d8054cf3342ed989c829d92c2815ee050d38beec4a7d48e3b" gracePeriod=30
Oct 30 01:09:54.386184 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:54.386091    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95e2da69-n7tqv"
Oct 30 01:09:54.386184 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:54.386121    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95e2da69-n7tqv" event={"ID":"ec788b52-0c21-48a4-b329-27ce6a65433a","Type":"ContainerDied","Data":"6774328160b192d8dbebbfdc5c0776a53c64d62c193d6404e68b89ebb4c04eb4"}
Oct 30 01:09:54.386809 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:54.386215    5002 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6774328160b192d8dbebbfdc5c0776a53c64d62c193d6404e68b89ebb4c04eb4"
Oct 30 01:09:59.454059 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:59.454026    5002 generic.go:356] "Generic (PLEG): container finished" podID="c72b5051-4287-4298-b179-4e39b2535eb5" containerID="7ee356c780ab592d8054cf3342ed989c829d92c2815ee050d38beec4a7d48e3b" exitCode=0
Oct 30 01:09:59.454502 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:59.454129    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-j9nfb" event={"ID":"c72b5051-4287-4298-b179-4e39b2535eb5","Type":"ContainerDied","Data":"7ee356c780ab592d8054cf3342ed989c829d92c2815ee050d38beec4a7d48e3b"}
Oct 30 01:09:59.489641 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:59.489611    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-j9nfb"
Oct 30 01:09:59.600068 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:59.600033    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c72b5051-4287-4298-b179-4e39b2535eb5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c72b5051-4287-4298-b179-4e39b2535eb5-workload-socket\") pod \"c72b5051-4287-4298-b179-4e39b2535eb5\" (UID: \"c72b5051-4287-4298-b179-4e39b2535eb5\") "
Oct 30 01:09:59.600068 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:59.600073    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c72b5051-4287-4298-b179-4e39b2535eb5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c72b5051-4287-4298-b179-4e39b2535eb5-istiod-ca-cert\") pod \"c72b5051-4287-4298-b179-4e39b2535eb5\" (UID: \"c72b5051-4287-4298-b179-4e39b2535eb5\") "
Oct 30 01:09:59.600334 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:59.600136    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c72b5051-4287-4298-b179-4e39b2535eb5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c72b5051-4287-4298-b179-4e39b2535eb5-istio-envoy\") pod \"c72b5051-4287-4298-b179-4e39b2535eb5\" (UID: \"c72b5051-4287-4298-b179-4e39b2535eb5\") "
Oct 30 01:09:59.600334 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:59.600184    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c72b5051-4287-4298-b179-4e39b2535eb5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c72b5051-4287-4298-b179-4e39b2535eb5-istio-podinfo\") pod \"c72b5051-4287-4298-b179-4e39b2535eb5\" (UID: \"c72b5051-4287-4298-b179-4e39b2535eb5\") "
Oct 30 01:09:59.600334 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:59.600215    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c72b5051-4287-4298-b179-4e39b2535eb5-kube-api-access-mplhb\" (UniqueName: \"kubernetes.io/projected/c72b5051-4287-4298-b179-4e39b2535eb5-kube-api-access-mplhb\") pod \"c72b5051-4287-4298-b179-4e39b2535eb5\" (UID: \"c72b5051-4287-4298-b179-4e39b2535eb5\") "
Oct 30 01:09:59.600334 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:59.600233    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c72b5051-4287-4298-b179-4e39b2535eb5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c72b5051-4287-4298-b179-4e39b2535eb5-istio-data\") pod \"c72b5051-4287-4298-b179-4e39b2535eb5\" (UID: \"c72b5051-4287-4298-b179-4e39b2535eb5\") "
Oct 30 01:09:59.600334 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:59.600264    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c72b5051-4287-4298-b179-4e39b2535eb5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c72b5051-4287-4298-b179-4e39b2535eb5-workload-certs\") pod \"c72b5051-4287-4298-b179-4e39b2535eb5\" (UID: \"c72b5051-4287-4298-b179-4e39b2535eb5\") "
Oct 30 01:09:59.600334 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:59.600306    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c72b5051-4287-4298-b179-4e39b2535eb5-istio-token\" (UniqueName: \"kubernetes.io/projected/c72b5051-4287-4298-b179-4e39b2535eb5-istio-token\") pod \"c72b5051-4287-4298-b179-4e39b2535eb5\" (UID: \"c72b5051-4287-4298-b179-4e39b2535eb5\") "
Oct 30 01:09:59.600334 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:59.600329    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c72b5051-4287-4298-b179-4e39b2535eb5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c72b5051-4287-4298-b179-4e39b2535eb5-credential-socket\") pod \"c72b5051-4287-4298-b179-4e39b2535eb5\" (UID: \"c72b5051-4287-4298-b179-4e39b2535eb5\") "
Oct 30 01:09:59.600810 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:59.600634    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c72b5051-4287-4298-b179-4e39b2535eb5-credential-socket" pod "c72b5051-4287-4298-b179-4e39b2535eb5" (UID: "c72b5051-4287-4298-b179-4e39b2535eb5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:09:59.600810 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:59.600798    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c72b5051-4287-4298-b179-4e39b2535eb5-workload-socket" pod "c72b5051-4287-4298-b179-4e39b2535eb5" (UID: "c72b5051-4287-4298-b179-4e39b2535eb5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:09:59.601359 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:59.601017    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c72b5051-4287-4298-b179-4e39b2535eb5-istiod-ca-cert" pod "c72b5051-4287-4298-b179-4e39b2535eb5" (UID: "c72b5051-4287-4298-b179-4e39b2535eb5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:09:59.601359 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:59.601170    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c72b5051-4287-4298-b179-4e39b2535eb5-workload-certs" pod "c72b5051-4287-4298-b179-4e39b2535eb5" (UID: "c72b5051-4287-4298-b179-4e39b2535eb5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:09:59.601359 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:59.601333    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c72b5051-4287-4298-b179-4e39b2535eb5-istio-data" pod "c72b5051-4287-4298-b179-4e39b2535eb5" (UID: "c72b5051-4287-4298-b179-4e39b2535eb5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:09:59.601884 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:59.601861    5002 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c72b5051-4287-4298-b179-4e39b2535eb5-workload-socket\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:09:59.601979 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:59.601898    5002 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c72b5051-4287-4298-b179-4e39b2535eb5-istiod-ca-cert\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:09:59.601979 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:59.601914    5002 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c72b5051-4287-4298-b179-4e39b2535eb5-istio-data\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:09:59.601979 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:59.601926    5002 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c72b5051-4287-4298-b179-4e39b2535eb5-workload-certs\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:09:59.601979 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:59.601939    5002 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c72b5051-4287-4298-b179-4e39b2535eb5-credential-socket\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:09:59.604170 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:59.603783    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c72b5051-4287-4298-b179-4e39b2535eb5-istio-podinfo" pod "c72b5051-4287-4298-b179-4e39b2535eb5" (UID: "c72b5051-4287-4298-b179-4e39b2535eb5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:09:59.604282 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:59.604226    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c72b5051-4287-4298-b179-4e39b2535eb5-kube-api-access-mplhb" pod "c72b5051-4287-4298-b179-4e39b2535eb5" (UID: "c72b5051-4287-4298-b179-4e39b2535eb5"). InnerVolumeSpecName "kube-api-access-mplhb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:09:59.616860 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:59.616821    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c72b5051-4287-4298-b179-4e39b2535eb5-istio-envoy" pod "c72b5051-4287-4298-b179-4e39b2535eb5" (UID: "c72b5051-4287-4298-b179-4e39b2535eb5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:09:59.618185 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:59.618155    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c72b5051-4287-4298-b179-4e39b2535eb5-istio-token" pod "c72b5051-4287-4298-b179-4e39b2535eb5" (UID: "c72b5051-4287-4298-b179-4e39b2535eb5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:09:59.704142 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:59.704040    5002 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c72b5051-4287-4298-b179-4e39b2535eb5-istio-envoy\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:09:59.704142 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:59.704076    5002 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c72b5051-4287-4298-b179-4e39b2535eb5-istio-podinfo\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:09:59.704142 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:59.704090    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mplhb\" (UniqueName: \"kubernetes.io/projected/c72b5051-4287-4298-b179-4e39b2535eb5-kube-api-access-mplhb\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:09:59.704142 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:09:59.704106    5002 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c72b5051-4287-4298-b179-4e39b2535eb5-istio-token\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:10:00.138730 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:10:00.138634    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-549b3758-gs27m"]
Oct 30 01:10:00.156364 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:10:00.156311    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-549b3758-gs27m"]
Oct 30 01:10:00.466808 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:10:00.466777    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-j9nfb"
Oct 30 01:10:00.467288 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:10:00.466796    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-j9nfb" event={"ID":"c72b5051-4287-4298-b179-4e39b2535eb5","Type":"ContainerDied","Data":"5d1785dd7f6c048e6aeefa299860d74aa1bf0268712f6f72530738b92befc864"}
Oct 30 01:10:00.467288 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:10:00.466904    5002 scope.go:122] "RemoveContainer" containerID="7ee356c780ab592d8054cf3342ed989c829d92c2815ee050d38beec4a7d48e3b"
Oct 30 01:10:00.499836 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:10:00.499804    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc015bf9-5c24-4af9-b931-69203486dd49" path="/var/lib/kubelet/pods/cc015bf9-5c24-4af9-b931-69203486dd49/volumes"
Oct 30 01:10:00.531587 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:10:00.531545    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-j9nfb"]
Oct 30 01:10:00.546429 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:10:00.546353    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-j9nfb"]
Oct 30 01:10:00.924522 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:10:00.924423    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a882968-pm4vv"]
Oct 30 01:10:00.925725 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:10:00.925693    5002 cpu_manager.go:414] "containerMap: removing container" podUID="c72b5051-4287-4298-b179-4e39b2535eb5" containerName="istio-proxy"
Oct 30 01:10:00.941475 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:10:00.937625    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c72b5051-4287-4298-b179-4e39b2535eb5" containerName="istio-proxy"
Oct 30 01:10:00.941475 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:10:00.937769    5002 cpu_manager.go:414] "containerMap: removing container" podUID="25e6159a-609a-483f-b679-3e64db06178c" containerName="revoke-keys"
Oct 30 01:10:00.941475 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:10:00.937783    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25e6159a-609a-483f-b679-3e64db06178c" containerName="revoke-keys"
Oct 30 01:10:00.941475 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:10:00.937807    5002 cpu_manager.go:414] "containerMap: removing container" podUID="ec788b52-0c21-48a4-b329-27ce6a65433a" containerName="revoke-keys"
Oct 30 01:10:00.941475 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:10:00.937816    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec788b52-0c21-48a4-b329-27ce6a65433a" containerName="revoke-keys"
Oct 30 01:10:00.941475 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:10:00.938105    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="c72b5051-4287-4298-b179-4e39b2535eb5" containerName="istio-proxy"
Oct 30 01:10:00.941475 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:10:00.938140    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="25e6159a-609a-483f-b679-3e64db06178c" containerName="revoke-keys"
Oct 30 01:10:00.941475 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:10:00.938151    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec788b52-0c21-48a4-b329-27ce6a65433a" containerName="revoke-keys"
Oct 30 01:10:00.954100 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:10:00.954068    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a882968-pm4vv"]
Oct 30 01:10:00.954431 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:10:00.954348    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a882968-pm4vv"
Oct 30 01:10:00.979438 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:10:00.978724    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:10:00.979623 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:10:00.979462    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:10:01.031060 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:10:01.031027    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbr4j\" (UniqueName: \"kubernetes.io/projected/5dab5c4c-0cf8-430f-b6ba-b81775ffa60a-kube-api-access-xbr4j\") pod \"maas-api-revoke-sub-6a882968-pm4vv\" (UID: \"5dab5c4c-0cf8-430f-b6ba-b81775ffa60a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a882968-pm4vv"
Oct 30 01:10:01.031301 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:10:01.031278    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5dab5c4c-0cf8-430f-b6ba-b81775ffa60a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6a882968-pm4vv\" (UID: \"5dab5c4c-0cf8-430f-b6ba-b81775ffa60a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a882968-pm4vv"
Oct 30 01:10:01.031490 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:10:01.031471    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5dab5c4c-0cf8-430f-b6ba-b81775ffa60a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6a882968-pm4vv\" (UID: \"5dab5c4c-0cf8-430f-b6ba-b81775ffa60a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a882968-pm4vv"
Oct 30 01:10:01.050171 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:10:01.050141    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:10:01.050311 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:10:01.050257    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5phsh\""
Oct 30 01:10:01.132789 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:10:01.132733    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5dab5c4c-0cf8-430f-b6ba-b81775ffa60a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6a882968-pm4vv\" (UID: \"5dab5c4c-0cf8-430f-b6ba-b81775ffa60a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a882968-pm4vv"
Oct 30 01:10:01.133006 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:10:01.132826    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbr4j\" (UniqueName: \"kubernetes.io/projected/5dab5c4c-0cf8-430f-b6ba-b81775ffa60a-kube-api-access-xbr4j\") pod \"maas-api-revoke-sub-6a882968-pm4vv\" (UID: \"5dab5c4c-0cf8-430f-b6ba-b81775ffa60a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a882968-pm4vv"
Oct 30 01:10:01.133081 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:10:01.133036    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5dab5c4c-0cf8-430f-b6ba-b81775ffa60a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6a882968-pm4vv\" (UID: \"5dab5c4c-0cf8-430f-b6ba-b81775ffa60a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a882968-pm4vv"
Oct 30 01:10:01.133897 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:10:01.133853    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5dab5c4c-0cf8-430f-b6ba-b81775ffa60a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6a882968-pm4vv\" (UID: \"5dab5c4c-0cf8-430f-b6ba-b81775ffa60a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a882968-pm4vv"
Oct 30 01:10:01.135439 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:10:01.135414    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5dab5c4c-0cf8-430f-b6ba-b81775ffa60a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6a882968-pm4vv\" (UID: \"5dab5c4c-0cf8-430f-b6ba-b81775ffa60a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a882968-pm4vv"
Oct 30 01:10:01.159587 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:10:01.159556    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbr4j\" (UniqueName: \"kubernetes.io/projected/5dab5c4c-0cf8-430f-b6ba-b81775ffa60a-kube-api-access-xbr4j\") pod \"maas-api-revoke-sub-6a882968-pm4vv\" (UID: \"5dab5c4c-0cf8-430f-b6ba-b81775ffa60a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a882968-pm4vv"
Oct 30 01:10:01.296481 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:10:01.296421    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a882968-pm4vv"
Oct 30 01:10:01.602338 ip-10-0-128-7 kubenswrapper[5002]: W1030 01:10:01.602305    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5dab5c4c_0cf8_430f_b6ba_b81775ffa60a.slice/crio-8e05fe15e6026683da1221c81b9f63251edc25875cd367db5dec6f9f2becf92d WatchSource:0}: Error finding container 8e05fe15e6026683da1221c81b9f63251edc25875cd367db5dec6f9f2becf92d: Status 404 returned error can't find the container with id 8e05fe15e6026683da1221c81b9f63251edc25875cd367db5dec6f9f2becf92d
Oct 30 01:10:01.603146 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:10:01.603114    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a882968-pm4vv"]
Oct 30 01:10:02.493236 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:10:02.493209    5002 generic.go:356] "Generic (PLEG): container finished" podID="5dab5c4c-0cf8-430f-b6ba-b81775ffa60a" containerID="675513e0f17c304163c14f327e739f3aacda9c002e7c4bb6aa84c2e23046433d" exitCode=0
Oct 30 01:10:02.495773 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:10:02.495747    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c72b5051-4287-4298-b179-4e39b2535eb5" path="/var/lib/kubelet/pods/c72b5051-4287-4298-b179-4e39b2535eb5/volumes"
Oct 30 01:10:02.496113 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:10:02.496085    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a882968-pm4vv" event={"ID":"5dab5c4c-0cf8-430f-b6ba-b81775ffa60a","Type":"ContainerDied","Data":"675513e0f17c304163c14f327e739f3aacda9c002e7c4bb6aa84c2e23046433d"}
Oct 30 01:10:02.496226 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:10:02.496144    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a882968-pm4vv" event={"ID":"5dab5c4c-0cf8-430f-b6ba-b81775ffa60a","Type":"ContainerStarted","Data":"8e05fe15e6026683da1221c81b9f63251edc25875cd367db5dec6f9f2becf92d"}
Oct 30 01:10:03.674338 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:10:03.674309    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a882968-pm4vv"
Oct 30 01:10:03.776273 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:10:03.776170    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5dab5c4c-0cf8-430f-b6ba-b81775ffa60a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5dab5c4c-0cf8-430f-b6ba-b81775ffa60a-maas-api-openshift-ca\") pod \"5dab5c4c-0cf8-430f-b6ba-b81775ffa60a\" (UID: \"5dab5c4c-0cf8-430f-b6ba-b81775ffa60a\") "
Oct 30 01:10:03.776481 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:10:03.776330    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5dab5c4c-0cf8-430f-b6ba-b81775ffa60a-kube-api-access-xbr4j\" (UniqueName: \"kubernetes.io/projected/5dab5c4c-0cf8-430f-b6ba-b81775ffa60a-kube-api-access-xbr4j\") pod \"5dab5c4c-0cf8-430f-b6ba-b81775ffa60a\" (UID: \"5dab5c4c-0cf8-430f-b6ba-b81775ffa60a\") "
Oct 30 01:10:03.776481 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:10:03.776351    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5dab5c4c-0cf8-430f-b6ba-b81775ffa60a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5dab5c4c-0cf8-430f-b6ba-b81775ffa60a-maas-api-xks-ca\") pod \"5dab5c4c-0cf8-430f-b6ba-b81775ffa60a\" (UID: \"5dab5c4c-0cf8-430f-b6ba-b81775ffa60a\") "
Oct 30 01:10:03.776646 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:10:03.776609    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5dab5c4c-0cf8-430f-b6ba-b81775ffa60a-maas-api-openshift-ca" pod "5dab5c4c-0cf8-430f-b6ba-b81775ffa60a" (UID: "5dab5c4c-0cf8-430f-b6ba-b81775ffa60a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:10:03.776932 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:10:03.776912    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5dab5c4c-0cf8-430f-b6ba-b81775ffa60a-maas-api-openshift-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:10:03.779462 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:10:03.779439    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5dab5c4c-0cf8-430f-b6ba-b81775ffa60a-maas-api-xks-ca" pod "5dab5c4c-0cf8-430f-b6ba-b81775ffa60a" (UID: "5dab5c4c-0cf8-430f-b6ba-b81775ffa60a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:10:03.779566 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:10:03.779462    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5dab5c4c-0cf8-430f-b6ba-b81775ffa60a-kube-api-access-xbr4j" pod "5dab5c4c-0cf8-430f-b6ba-b81775ffa60a" (UID: "5dab5c4c-0cf8-430f-b6ba-b81775ffa60a"). InnerVolumeSpecName "kube-api-access-xbr4j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:10:03.878020 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:10:03.877987    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xbr4j\" (UniqueName: \"kubernetes.io/projected/5dab5c4c-0cf8-430f-b6ba-b81775ffa60a-kube-api-access-xbr4j\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:10:03.878020 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:10:03.878019    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5dab5c4c-0cf8-430f-b6ba-b81775ffa60a-maas-api-xks-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:10:04.511247 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:10:04.511216    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a882968-pm4vv"
Oct 30 01:10:04.511420 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:10:04.511247    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a882968-pm4vv" event={"ID":"5dab5c4c-0cf8-430f-b6ba-b81775ffa60a","Type":"ContainerDied","Data":"8e05fe15e6026683da1221c81b9f63251edc25875cd367db5dec6f9f2becf92d"}
Oct 30 01:10:04.511420 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:10:04.511297    5002 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8e05fe15e6026683da1221c81b9f63251edc25875cd367db5dec6f9f2becf92d"
Oct 30 01:10:11.484902 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:10:11.484857    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a13c059-746ns"]
Oct 30 01:10:11.485944 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:10:11.485923    5002 cpu_manager.go:414] "containerMap: removing container" podUID="5dab5c4c-0cf8-430f-b6ba-b81775ffa60a" containerName="revoke-keys"
Oct 30 01:10:11.486033 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:10:11.485947    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5dab5c4c-0cf8-430f-b6ba-b81775ffa60a" containerName="revoke-keys"
Oct 30 01:10:11.486093 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:10:11.486086    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="5dab5c4c-0cf8-430f-b6ba-b81775ffa60a" containerName="revoke-keys"
Oct 30 01:10:11.490517 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:10:11.490489    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a13c059-746ns"
Oct 30 01:10:11.507419 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:10:11.507340    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a13c059-746ns"]
Oct 30 01:10:11.516210 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:10:11.516180    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:10:11.516372 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:10:11.516250    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5phsh\""
Oct 30 01:10:11.540079 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:10:11.540032    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ml7g\" (UniqueName: \"kubernetes.io/projected/6d3c5627-d6b7-4703-afd1-3b965ae529c1-kube-api-access-8ml7g\") pod \"maas-api-revoke-sub-5a13c059-746ns\" (UID: \"6d3c5627-d6b7-4703-afd1-3b965ae529c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a13c059-746ns"
Oct 30 01:10:11.540277 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:10:11.540194    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d3c5627-d6b7-4703-afd1-3b965ae529c1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a13c059-746ns\" (UID: \"6d3c5627-d6b7-4703-afd1-3b965ae529c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a13c059-746ns"
Oct 30 01:10:11.540277 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:10:11.540261    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d3c5627-d6b7-4703-afd1-3b965ae529c1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a13c059-746ns\" (UID: \"6d3c5627-d6b7-4703-afd1-3b965ae529c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a13c059-746ns"
Oct 30 01:10:11.601357 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:10:11.601325    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:10:11.601357 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:10:11.601355    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:10:11.642037 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:10:11.642003    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ml7g\" (UniqueName: \"kubernetes.io/projected/6d3c5627-d6b7-4703-afd1-3b965ae529c1-kube-api-access-8ml7g\") pod \"maas-api-revoke-sub-5a13c059-746ns\" (UID: \"6d3c5627-d6b7-4703-afd1-3b965ae529c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a13c059-746ns"
Oct 30 01:10:11.642223 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:10:11.642059    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d3c5627-d6b7-4703-afd1-3b965ae529c1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a13c059-746ns\" (UID: \"6d3c5627-d6b7-4703-afd1-3b965ae529c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a13c059-746ns"
Oct 30 01:10:11.642223 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:10:11.642093    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d3c5627-d6b7-4703-afd1-3b965ae529c1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a13c059-746ns\" (UID: \"6d3c5627-d6b7-4703-afd1-3b965ae529c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a13c059-746ns"
Oct 30 01:10:11.642829 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:10:11.642803    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d3c5627-d6b7-4703-afd1-3b965ae529c1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a13c059-746ns\" (UID: \"6d3c5627-d6b7-4703-afd1-3b965ae529c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a13c059-746ns"
Oct 30 01:10:11.644660 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:10:11.644636    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d3c5627-d6b7-4703-afd1-3b965ae529c1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a13c059-746ns\" (UID: \"6d3c5627-d6b7-4703-afd1-3b965ae529c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a13c059-746ns"
Oct 30 01:10:11.667804 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:10:11.667762    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ml7g\" (UniqueName: \"kubernetes.io/projected/6d3c5627-d6b7-4703-afd1-3b965ae529c1-kube-api-access-8ml7g\") pod \"maas-api-revoke-sub-5a13c059-746ns\" (UID: \"6d3c5627-d6b7-4703-afd1-3b965ae529c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a13c059-746ns"
Oct 30 01:10:11.811239 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:10:11.811148    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a13c059-746ns"
Oct 30 01:10:12.116923 ip-10-0-128-7 kubenswrapper[5002]: W1030 01:10:12.116890    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d3c5627_d6b7_4703_afd1_3b965ae529c1.slice/crio-4c9bf75601c6ee491f256eb6f253d4c71f32886256bd3ba6b6ea2b2db69b1ba4 WatchSource:0}: Error finding container 4c9bf75601c6ee491f256eb6f253d4c71f32886256bd3ba6b6ea2b2db69b1ba4: Status 404 returned error can't find the container with id 4c9bf75601c6ee491f256eb6f253d4c71f32886256bd3ba6b6ea2b2db69b1ba4
Oct 30 01:10:12.120462 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:10:12.120433    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a13c059-746ns"]
Oct 30 01:10:12.586788 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:10:12.586748    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a13c059-746ns" event={"ID":"6d3c5627-d6b7-4703-afd1-3b965ae529c1","Type":"ContainerStarted","Data":"e2488766af84367d08385f61563c800c92c0aa3c5179a21d052706a5b4d7ed76"}
Oct 30 01:10:12.587157 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:10:12.586818    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a13c059-746ns" event={"ID":"6d3c5627-d6b7-4703-afd1-3b965ae529c1","Type":"ContainerStarted","Data":"4c9bf75601c6ee491f256eb6f253d4c71f32886256bd3ba6b6ea2b2db69b1ba4"}
Oct 30 01:10:12.633033 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:10:12.632926    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a13c059-746ns" podStartSLOduration=1.6329111090000001 podStartE2EDuration="1.632911109s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:10:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:10:12.630078373 +0000 UTC m=+2308.246511254" watchObservedRunningTime="2026-10-30 01:10:12.632911109 +0000 UTC m=+2308.249343987"
Oct 30 01:10:12.867259 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:10:12.867215    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbjdq" containerName="extractor"
Oct 30 01:10:13.599253 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:10:13.599212    5002 generic.go:356] "Generic (PLEG): container finished" podID="6d3c5627-d6b7-4703-afd1-3b965ae529c1" containerID="e2488766af84367d08385f61563c800c92c0aa3c5179a21d052706a5b4d7ed76" exitCode=0
Oct 30 01:10:13.599713 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:10:13.599302    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a13c059-746ns" event={"ID":"6d3c5627-d6b7-4703-afd1-3b965ae529c1","Type":"ContainerDied","Data":"e2488766af84367d08385f61563c800c92c0aa3c5179a21d052706a5b4d7ed76"}
Oct 30 01:10:14.812463 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:10:14.812435    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a13c059-746ns"
Oct 30 01:10:14.900049 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:10:14.899949    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6d3c5627-d6b7-4703-afd1-3b965ae529c1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d3c5627-d6b7-4703-afd1-3b965ae529c1-maas-api-xks-ca\") pod \"6d3c5627-d6b7-4703-afd1-3b965ae529c1\" (UID: \"6d3c5627-d6b7-4703-afd1-3b965ae529c1\") "
Oct 30 01:10:14.900236 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:10:14.900173    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d3c5627-d6b7-4703-afd1-3b965ae529c1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d3c5627-d6b7-4703-afd1-3b965ae529c1-maas-api-openshift-ca\") pod \"6d3c5627-d6b7-4703-afd1-3b965ae529c1\" (UID: \"6d3c5627-d6b7-4703-afd1-3b965ae529c1\") "
Oct 30 01:10:14.900343 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:10:14.900324    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d3c5627-d6b7-4703-afd1-3b965ae529c1-kube-api-access-8ml7g\" (UniqueName: \"kubernetes.io/projected/6d3c5627-d6b7-4703-afd1-3b965ae529c1-kube-api-access-8ml7g\") pod \"6d3c5627-d6b7-4703-afd1-3b965ae529c1\" (UID: \"6d3c5627-d6b7-4703-afd1-3b965ae529c1\") "
Oct 30 01:10:14.900641 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:10:14.900614    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d3c5627-d6b7-4703-afd1-3b965ae529c1-maas-api-openshift-ca" pod "6d3c5627-d6b7-4703-afd1-3b965ae529c1" (UID: "6d3c5627-d6b7-4703-afd1-3b965ae529c1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:10:14.901052 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:10:14.901020    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d3c5627-d6b7-4703-afd1-3b965ae529c1-maas-api-openshift-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:10:14.902475 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:10:14.902446    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d3c5627-d6b7-4703-afd1-3b965ae529c1-maas-api-xks-ca" pod "6d3c5627-d6b7-4703-afd1-3b965ae529c1" (UID: "6d3c5627-d6b7-4703-afd1-3b965ae529c1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:10:14.902748 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:10:14.902722    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d3c5627-d6b7-4703-afd1-3b965ae529c1-kube-api-access-8ml7g" pod "6d3c5627-d6b7-4703-afd1-3b965ae529c1" (UID: "6d3c5627-d6b7-4703-afd1-3b965ae529c1"). InnerVolumeSpecName "kube-api-access-8ml7g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:10:15.001978 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:10:15.001935    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8ml7g\" (UniqueName: \"kubernetes.io/projected/6d3c5627-d6b7-4703-afd1-3b965ae529c1-kube-api-access-8ml7g\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:10:15.001978 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:10:15.001980    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d3c5627-d6b7-4703-afd1-3b965ae529c1-maas-api-xks-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:10:15.616351 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:10:15.616317    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a13c059-746ns"
Oct 30 01:10:15.616351 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:10:15.616339    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a13c059-746ns" event={"ID":"6d3c5627-d6b7-4703-afd1-3b965ae529c1","Type":"ContainerDied","Data":"4c9bf75601c6ee491f256eb6f253d4c71f32886256bd3ba6b6ea2b2db69b1ba4"}
Oct 30 01:10:15.616631 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:10:15.616430    5002 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4c9bf75601c6ee491f256eb6f253d4c71f32886256bd3ba6b6ea2b2db69b1ba4"
Oct 30 01:10:19.374414 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:10:19.339310    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7d8z4"]
Oct 30 01:10:24.033464 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:10:24.033412    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4672e3a2-sfk88"]
Oct 30 01:10:24.044851 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:10:24.044815    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4672e3a2-sfk88"]
Oct 30 01:10:24.498883 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:10:24.498842    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05f79dc7-3c9c-491a-b288-a5ee57061e7a" path="/var/lib/kubelet/pods/05f79dc7-3c9c-491a-b288-a5ee57061e7a/volumes"
Oct 30 01:10:35.812743 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:10:35.812702    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-x49mt"]
Oct 30 01:10:35.813338 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:10:35.813323    5002 cpu_manager.go:414] "containerMap: removing container" podUID="6d3c5627-d6b7-4703-afd1-3b965ae529c1" containerName="revoke-keys"
Oct 30 01:10:35.813402 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:10:35.813338    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d3c5627-d6b7-4703-afd1-3b965ae529c1" containerName="revoke-keys"
Oct 30 01:10:35.813497 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:10:35.813487    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d3c5627-d6b7-4703-afd1-3b965ae529c1" containerName="revoke-keys"
Oct 30 01:10:35.818548 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:10:35.818522    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-x49mt"
Oct 30 01:10:35.865402 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:10:35.854510    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-beeba4-openshift-default-dockercfg-5vtrz\""
Oct 30 01:10:35.887359 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:10:35.887322    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-x49mt"]
Oct 30 01:10:35.900877 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:10:35.900837    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19abf60c-9f0e-47c8-9679-970496e6772c-istiod-ca-cert\") pod \"e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-x49mt\" (UID: \"19abf60c-9f0e-47c8-9679-970496e6772c\") " pod="openshift-ingress/e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-x49mt"
Oct 30 01:10:35.901062 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:10:35.900885    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19abf60c-9f0e-47c8-9679-970496e6772c-istio-envoy\") pod \"e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-x49mt\" (UID: \"19abf60c-9f0e-47c8-9679-970496e6772c\") " pod="openshift-ingress/e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-x49mt"
Oct 30 01:10:35.901062 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:10:35.901028    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19abf60c-9f0e-47c8-9679-970496e6772c-istio-data\") pod \"e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-x49mt\" (UID: \"19abf60c-9f0e-47c8-9679-970496e6772c\") " pod="openshift-ingress/e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-x49mt"
Oct 30 01:10:35.901163 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:10:35.901094    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19abf60c-9f0e-47c8-9679-970496e6772c-credential-socket\") pod \"e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-x49mt\" (UID: \"19abf60c-9f0e-47c8-9679-970496e6772c\") " pod="openshift-ingress/e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-x49mt"
Oct 30 01:10:35.901163 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:10:35.901155    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19abf60c-9f0e-47c8-9679-970496e6772c-istio-podinfo\") pod \"e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-x49mt\" (UID: \"19abf60c-9f0e-47c8-9679-970496e6772c\") " pod="openshift-ingress/e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-x49mt"
Oct 30 01:10:35.901243 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:10:35.901209    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19abf60c-9f0e-47c8-9679-970496e6772c-workload-certs\") pod \"e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-x49mt\" (UID: \"19abf60c-9f0e-47c8-9679-970496e6772c\") " pod="openshift-ingress/e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-x49mt"
Oct 30 01:10:35.901317 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:10:35.901295    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19abf60c-9f0e-47c8-9679-970496e6772c-workload-socket\") pod \"e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-x49mt\" (UID: \"19abf60c-9f0e-47c8-9679-970496e6772c\") " pod="openshift-ingress/e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-x49mt"
Oct 30 01:10:35.901371 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:10:35.901355    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19abf60c-9f0e-47c8-9679-970496e6772c-istio-token\") pod \"e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-x49mt\" (UID: \"19abf60c-9f0e-47c8-9679-970496e6772c\") " pod="openshift-ingress/e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-x49mt"
Oct 30 01:10:35.901522 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:10:35.901501    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlc9t\" (UniqueName: \"kubernetes.io/projected/19abf60c-9f0e-47c8-9679-970496e6772c-kube-api-access-dlc9t\") pod \"e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-x49mt\" (UID: \"19abf60c-9f0e-47c8-9679-970496e6772c\") " pod="openshift-ingress/e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-x49mt"
Oct 30 01:10:36.002574 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:10:36.002536    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19abf60c-9f0e-47c8-9679-970496e6772c-credential-socket\") pod \"e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-x49mt\" (UID: \"19abf60c-9f0e-47c8-9679-970496e6772c\") " pod="openshift-ingress/e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-x49mt"
Oct 30 01:10:36.002791 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:10:36.002598    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19abf60c-9f0e-47c8-9679-970496e6772c-istio-podinfo\") pod \"e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-x49mt\" (UID: \"19abf60c-9f0e-47c8-9679-970496e6772c\") " pod="openshift-ingress/e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-x49mt"
Oct 30 01:10:36.002791 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:10:36.002638    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19abf60c-9f0e-47c8-9679-970496e6772c-workload-certs\") pod \"e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-x49mt\" (UID: \"19abf60c-9f0e-47c8-9679-970496e6772c\") " pod="openshift-ingress/e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-x49mt"
Oct 30 01:10:36.002791 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:10:36.002679    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19abf60c-9f0e-47c8-9679-970496e6772c-workload-socket\") pod \"e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-x49mt\" (UID: \"19abf60c-9f0e-47c8-9679-970496e6772c\") " pod="openshift-ingress/e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-x49mt"
Oct 30 01:10:36.002791 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:10:36.002741    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19abf60c-9f0e-47c8-9679-970496e6772c-istio-token\") pod \"e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-x49mt\" (UID: \"19abf60c-9f0e-47c8-9679-970496e6772c\") " pod="openshift-ingress/e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-x49mt"
Oct 30 01:10:36.003023 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:10:36.002797    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlc9t\" (UniqueName: \"kubernetes.io/projected/19abf60c-9f0e-47c8-9679-970496e6772c-kube-api-access-dlc9t\") pod \"e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-x49mt\" (UID: \"19abf60c-9f0e-47c8-9679-970496e6772c\") " pod="openshift-ingress/e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-x49mt"
Oct 30 01:10:36.003023 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:10:36.002936    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19abf60c-9f0e-47c8-9679-970496e6772c-istiod-ca-cert\") pod \"e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-x49mt\" (UID: \"19abf60c-9f0e-47c8-9679-970496e6772c\") " pod="openshift-ingress/e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-x49mt"
Oct 30 01:10:36.003023 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:10:36.002964    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19abf60c-9f0e-47c8-9679-970496e6772c-istio-envoy\") pod \"e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-x49mt\" (UID: \"19abf60c-9f0e-47c8-9679-970496e6772c\") " pod="openshift-ingress/e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-x49mt"
Oct 30 01:10:36.003023 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:10:36.003008    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19abf60c-9f0e-47c8-9679-970496e6772c-credential-socket\") pod \"e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-x49mt\" (UID: \"19abf60c-9f0e-47c8-9679-970496e6772c\") " pod="openshift-ingress/e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-x49mt"
Oct 30 01:10:36.003235 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:10:36.003035    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19abf60c-9f0e-47c8-9679-970496e6772c-istio-data\") pod \"e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-x49mt\" (UID: \"19abf60c-9f0e-47c8-9679-970496e6772c\") " pod="openshift-ingress/e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-x49mt"
Oct 30 01:10:36.003235 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:10:36.003082    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19abf60c-9f0e-47c8-9679-970496e6772c-workload-socket\") pod \"e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-x49mt\" (UID: \"19abf60c-9f0e-47c8-9679-970496e6772c\") " pod="openshift-ingress/e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-x49mt"
Oct 30 01:10:36.003431 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:10:36.003400    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19abf60c-9f0e-47c8-9679-970496e6772c-istio-data\") pod \"e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-x49mt\" (UID: \"19abf60c-9f0e-47c8-9679-970496e6772c\") " pod="openshift-ingress/e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-x49mt"
Oct 30 01:10:36.003754 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:10:36.003728    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19abf60c-9f0e-47c8-9679-970496e6772c-istiod-ca-cert\") pod \"e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-x49mt\" (UID: \"19abf60c-9f0e-47c8-9679-970496e6772c\") " pod="openshift-ingress/e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-x49mt"
Oct 30 01:10:36.003754 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:10:36.003737    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19abf60c-9f0e-47c8-9679-970496e6772c-workload-certs\") pod \"e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-x49mt\" (UID: \"19abf60c-9f0e-47c8-9679-970496e6772c\") " pod="openshift-ingress/e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-x49mt"
Oct 30 01:10:36.005358 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:10:36.005333    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19abf60c-9f0e-47c8-9679-970496e6772c-istio-envoy\") pod \"e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-x49mt\" (UID: \"19abf60c-9f0e-47c8-9679-970496e6772c\") " pod="openshift-ingress/e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-x49mt"
Oct 30 01:10:36.005648 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:10:36.005632    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19abf60c-9f0e-47c8-9679-970496e6772c-istio-podinfo\") pod \"e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-x49mt\" (UID: \"19abf60c-9f0e-47c8-9679-970496e6772c\") " pod="openshift-ingress/e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-x49mt"
Oct 30 01:10:36.028112 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:10:36.028090    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19abf60c-9f0e-47c8-9679-970496e6772c-istio-token\") pod \"e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-x49mt\" (UID: \"19abf60c-9f0e-47c8-9679-970496e6772c\") " pod="openshift-ingress/e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-x49mt"
Oct 30 01:10:36.028302 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:10:36.028279    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlc9t\" (UniqueName: \"kubernetes.io/projected/19abf60c-9f0e-47c8-9679-970496e6772c-kube-api-access-dlc9t\") pod \"e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-x49mt\" (UID: \"19abf60c-9f0e-47c8-9679-970496e6772c\") " pod="openshift-ingress/e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-x49mt"
Oct 30 01:10:36.133375 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:10:36.133278    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-x49mt"
Oct 30 01:10:36.507298 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:10:36.507230    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-x49mt"]
Oct 30 01:10:36.510830 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:10:36.510793    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:10:36.510925 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:10:36.510872    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:10:36.510925 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:10:36.510909    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:10:37.123465 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:10:37.123268    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-x49mt" event={"ID":"19abf60c-9f0e-47c8-9679-970496e6772c","Type":"ContainerStarted","Data":"15f26cf4361f3eea12c3580685bc363574cce9f0653ff0122924c0552072b762"}
Oct 30 01:10:37.123465 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:10:37.123337    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-x49mt" event={"ID":"19abf60c-9f0e-47c8-9679-970496e6772c","Type":"ContainerStarted","Data":"6a0a417e628fd6686455d0688856fb7d1cff6213eaa53155eb30366f0a6c3eff"}
Oct 30 01:10:37.151410 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:10:37.135405    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-x49mt"
Oct 30 01:10:37.151410 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:10:37.139510    5002 patch_prober.go:29] interesting pod/e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-x49mt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" start-of-body=
Oct 30 01:10:37.151410 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:10:37.139568    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-x49mt" podUID="19abf60c-9f0e-47c8-9679-970496e6772c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused"
Oct 30 01:10:38.133678 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:10:38.133645    5002 patch_prober.go:29] interesting pod/e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-x49mt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" start-of-body=
Oct 30 01:10:38.134150 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:10:38.133708    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-x49mt" podUID="19abf60c-9f0e-47c8-9679-970496e6772c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused"
Oct 30 01:10:39.134600 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:10:39.134129    5002 patch_prober.go:29] interesting pod/e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-x49mt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" start-of-body=
Oct 30 01:10:39.134600 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:10:39.134192    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-x49mt" podUID="19abf60c-9f0e-47c8-9679-970496e6772c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused"
Oct 30 01:10:39.607687 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:10:39.607529    5002 patch_prober.go:29] interesting pod/e2e-ipp-a-73f667-openshift-default-7bcfd5c9b9-7fcm7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.196:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 01:10:39.607687 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:10:39.607604    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-73f667-openshift-default-7bcfd5c9b9-7fcm7" podUID="0eb54f67-9f54-468e-a88c-ba3fbcaa6d2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.196:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:10:40.134075 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:10:40.134030    5002 patch_prober.go:29] interesting pod/e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-x49mt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" start-of-body=
Oct 30 01:10:40.134294 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:10:40.134089    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-x49mt" podUID="19abf60c-9f0e-47c8-9679-970496e6772c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused"
Oct 30 01:10:41.136048 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:10:41.136006    5002 patch_prober.go:29] interesting pod/e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-x49mt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" start-of-body=
Oct 30 01:10:41.136597 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:10:41.136069    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-x49mt" podUID="19abf60c-9f0e-47c8-9679-970496e6772c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused"
Oct 30 01:10:42.133966 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:10:42.133921    5002 patch_prober.go:29] interesting pod/e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-x49mt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" start-of-body=
Oct 30 01:10:42.134250 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:10:42.134224    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-x49mt" podUID="19abf60c-9f0e-47c8-9679-970496e6772c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused"
Oct 30 01:10:42.925112 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:10:42.925075    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbjdq" containerName="extractor"
Oct 30 01:10:43.135317 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:10:43.135283    5002 patch_prober.go:29] interesting pod/e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-x49mt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" start-of-body=
Oct 30 01:10:43.135505 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:10:43.135336    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-x49mt" podUID="19abf60c-9f0e-47c8-9679-970496e6772c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused"
Oct 30 01:10:44.133909 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:10:44.133871    5002 patch_prober.go:29] interesting pod/e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-x49mt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" start-of-body=
Oct 30 01:10:44.134512 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:10:44.133935    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-x49mt" podUID="19abf60c-9f0e-47c8-9679-970496e6772c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused"
Oct 30 01:10:44.234600 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:10:44.234559    5002 generic.go:356] "Generic (PLEG): container finished" podID="0eb54f67-9f54-468e-a88c-ba3fbcaa6d2d" containerID="c7ea4c5c15a25ce02e96b7fcc3c1265f8042d3ca9174740c5341b3b07aa2fdd1" exitCode=0
Oct 30 01:10:44.234600 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:10:44.234591    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-73f667-openshift-default-7bcfd5c9b9-7fcm7" event={"ID":"0eb54f67-9f54-468e-a88c-ba3fbcaa6d2d","Type":"ContainerDied","Data":"c7ea4c5c15a25ce02e96b7fcc3c1265f8042d3ca9174740c5341b3b07aa2fdd1"}
Oct 30 01:10:44.235059 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:10:44.235031    5002 scope.go:122] "RemoveContainer" containerID="c7ea4c5c15a25ce02e96b7fcc3c1265f8042d3ca9174740c5341b3b07aa2fdd1"
Oct 30 01:10:44.240151 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:10:44.240105    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:10:44.240300 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:10:44.240276    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:10:44.240431 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:10:44.240411    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:10:44.300375 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:10:44.300105    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-x49mt" podStartSLOduration=9.300086237 podStartE2EDuration="9.300086237s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:10:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:10:37.200032652 +0000 UTC m=+2332.816465590" watchObservedRunningTime="2026-10-30 01:10:44.300086237 +0000 UTC m=+2339.916519117"
Oct 30 01:10:45.140566 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:10:45.140530    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-x49mt"
Oct 30 01:10:45.141049 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:10:45.140960    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-x49mt"
Oct 30 01:10:45.141851 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:10:45.141831    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-x49mt"
Oct 30 01:10:45.245081 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:10:45.245043    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-73f667-openshift-default-7bcfd5c9b9-7fcm7" event={"ID":"0eb54f67-9f54-468e-a88c-ba3fbcaa6d2d","Type":"ContainerStarted","Data":"02308566a5d06b2f98c4dcf4fd883478831cfc487f53021637bf30e0a6b094ac"}
Oct 30 01:10:45.645624 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:10:45.645515    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-73f667-openshift-default-7bcfd5c9b9-7fcm7"
Oct 30 01:10:45.646220 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:10:45.646195    5002 patch_prober.go:29] interesting pod/e2e-ipp-a-73f667-openshift-default-7bcfd5c9b9-7fcm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" start-of-body=
Oct 30 01:10:45.646418 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:10:45.646393    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-73f667-openshift-default-7bcfd5c9b9-7fcm7" podUID="0eb54f67-9f54-468e-a88c-ba3fbcaa6d2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused"
Oct 30 01:10:46.645110 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:10:46.645078    5002 patch_prober.go:29] interesting pod/e2e-ipp-a-73f667-openshift-default-7bcfd5c9b9-7fcm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" start-of-body=
Oct 30 01:10:46.645593 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:10:46.645566    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-73f667-openshift-default-7bcfd5c9b9-7fcm7" podUID="0eb54f67-9f54-468e-a88c-ba3fbcaa6d2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused"
Oct 30 01:10:47.644679 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:10:47.644640    5002 patch_prober.go:29] interesting pod/e2e-ipp-a-73f667-openshift-default-7bcfd5c9b9-7fcm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" start-of-body=
Oct 30 01:10:47.644892 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:10:47.644700    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-73f667-openshift-default-7bcfd5c9b9-7fcm7" podUID="0eb54f67-9f54-468e-a88c-ba3fbcaa6d2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused"
Oct 30 01:10:48.039311 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:10:48.039266    5002 scope.go:122] "RemoveContainer" containerID="dd192abbc69c3be96a2c91b78b7245eda34d0e5eba6a8f299080e86ae72a7c62"
Oct 30 01:10:48.068659 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:10:48.068615    5002 scope.go:122] "RemoveContainer" containerID="c45dd3879f6956061f289a74598e4b795875b85985bd04394cb62d73e3f7687e"
Oct 30 01:10:48.095004 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:10:48.094972    5002 scope.go:122] "RemoveContainer" containerID="6353abe176e6dd1b4348592b656844dcb228b9911c894f261b980df7f6cb1a10"
Oct 30 01:10:48.108457 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:10:48.108430    5002 scope.go:122] "RemoveContainer" containerID="3bc612d98ed2cfb7adb8e8ca53bf5bbad1f7cf1375bd9b656e839a2c9af965eb"
Oct 30 01:10:48.120206 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:10:48.120179    5002 scope.go:122] "RemoveContainer" containerID="3573151956bf14c3538772080527f3d46f8b4b787b1743248e4e500c1dbdb5ff"
Oct 30 01:10:48.679410 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:10:48.677530    5002 patch_prober.go:29] interesting pod/e2e-ipp-a-73f667-openshift-default-7bcfd5c9b9-7fcm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" start-of-body=
Oct 30 01:10:48.679410 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:10:48.677618    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-73f667-openshift-default-7bcfd5c9b9-7fcm7" podUID="0eb54f67-9f54-468e-a88c-ba3fbcaa6d2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused"
Oct 30 01:10:49.647409 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:10:49.644849    5002 patch_prober.go:29] interesting pod/e2e-ipp-a-73f667-openshift-default-7bcfd5c9b9-7fcm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" start-of-body=
Oct 30 01:10:49.647409 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:10:49.644937    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-73f667-openshift-default-7bcfd5c9b9-7fcm7" podUID="0eb54f67-9f54-468e-a88c-ba3fbcaa6d2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused"
Oct 30 01:10:50.644361 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:10:50.644312    5002 patch_prober.go:29] interesting pod/e2e-ipp-a-73f667-openshift-default-7bcfd5c9b9-7fcm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" start-of-body=
Oct 30 01:10:50.644577 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:10:50.644405    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-73f667-openshift-default-7bcfd5c9b9-7fcm7" podUID="0eb54f67-9f54-468e-a88c-ba3fbcaa6d2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused"
Oct 30 01:10:51.650155 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:10:51.650121    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-73f667-openshift-default-7bcfd5c9b9-7fcm7"
Oct 30 01:10:51.650650 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:10:51.650608    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-73f667-openshift-default-7bcfd5c9b9-7fcm7"
Oct 30 01:10:51.651495 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:10:51.651472    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-73f667-openshift-default-7bcfd5c9b9-7fcm7"
Oct 30 01:11:12.865540 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:11:12.865487    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbjdq" containerName="extractor"
Oct 30 01:11:12.936136 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:11:12.936098    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7d8z4"]
Oct 30 01:11:21.472054 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:11:21.472013    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0aabe7a-6qthl"]
Oct 30 01:11:21.477306 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:11:21.477280    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0aabe7a-6qthl"
Oct 30 01:11:21.495429 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:11:21.495367    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0aabe7a-6qthl"]
Oct 30 01:11:21.503107 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:11:21.503067    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:11:21.503107 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:11:21.503075    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:11:21.503107 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:11:21.503071    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:11:21.509611 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:11:21.509588    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5phsh\""
Oct 30 01:11:21.646919 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:11:21.646873    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jpm2\" (UniqueName: \"kubernetes.io/projected/d930a049-cee8-4106-a988-0032fcaf229e-kube-api-access-8jpm2\") pod \"maas-api-revoke-sub-e0aabe7a-6qthl\" (UID: \"d930a049-cee8-4106-a988-0032fcaf229e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0aabe7a-6qthl"
Oct 30 01:11:21.647104 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:11:21.646930    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d930a049-cee8-4106-a988-0032fcaf229e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e0aabe7a-6qthl\" (UID: \"d930a049-cee8-4106-a988-0032fcaf229e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0aabe7a-6qthl"
Oct 30 01:11:21.647104 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:11:21.647003    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d930a049-cee8-4106-a988-0032fcaf229e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e0aabe7a-6qthl\" (UID: \"d930a049-cee8-4106-a988-0032fcaf229e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0aabe7a-6qthl"
Oct 30 01:11:21.749264 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:11:21.749159    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jpm2\" (UniqueName: \"kubernetes.io/projected/d930a049-cee8-4106-a988-0032fcaf229e-kube-api-access-8jpm2\") pod \"maas-api-revoke-sub-e0aabe7a-6qthl\" (UID: \"d930a049-cee8-4106-a988-0032fcaf229e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0aabe7a-6qthl"
Oct 30 01:11:21.749264 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:11:21.749220    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d930a049-cee8-4106-a988-0032fcaf229e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e0aabe7a-6qthl\" (UID: \"d930a049-cee8-4106-a988-0032fcaf229e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0aabe7a-6qthl"
Oct 30 01:11:21.749264 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:11:21.749255    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d930a049-cee8-4106-a988-0032fcaf229e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e0aabe7a-6qthl\" (UID: \"d930a049-cee8-4106-a988-0032fcaf229e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0aabe7a-6qthl"
Oct 30 01:11:21.750218 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:11:21.750184    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d930a049-cee8-4106-a988-0032fcaf229e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e0aabe7a-6qthl\" (UID: \"d930a049-cee8-4106-a988-0032fcaf229e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0aabe7a-6qthl"
Oct 30 01:11:21.752618 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:11:21.752591    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d930a049-cee8-4106-a988-0032fcaf229e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e0aabe7a-6qthl\" (UID: \"d930a049-cee8-4106-a988-0032fcaf229e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0aabe7a-6qthl"
Oct 30 01:11:21.774389 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:11:21.774350    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jpm2\" (UniqueName: \"kubernetes.io/projected/d930a049-cee8-4106-a988-0032fcaf229e-kube-api-access-8jpm2\") pod \"maas-api-revoke-sub-e0aabe7a-6qthl\" (UID: \"d930a049-cee8-4106-a988-0032fcaf229e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0aabe7a-6qthl"
Oct 30 01:11:21.803133 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:11:21.803105    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0aabe7a-6qthl"
Oct 30 01:11:22.110979 ip-10-0-128-7 kubenswrapper[5002]: W1030 01:11:22.110932    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd930a049_cee8_4106_a988_0032fcaf229e.slice/crio-9223db137fc82179e45bf6d50f31d26d92af0f429c704b5bea754a106c2316a3 WatchSource:0}: Error finding container 9223db137fc82179e45bf6d50f31d26d92af0f429c704b5bea754a106c2316a3: Status 404 returned error can't find the container with id 9223db137fc82179e45bf6d50f31d26d92af0f429c704b5bea754a106c2316a3
Oct 30 01:11:22.112077 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:11:22.112049    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0aabe7a-6qthl"]
Oct 30 01:11:22.718257 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:11:22.718218    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0aabe7a-6qthl" event={"ID":"d930a049-cee8-4106-a988-0032fcaf229e","Type":"ContainerStarted","Data":"a184ce67447af89684194ee1ccc178000b0677ce2c3d1a77772a0a08e111be34"}
Oct 30 01:11:22.718743 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:11:22.718284    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0aabe7a-6qthl" event={"ID":"d930a049-cee8-4106-a988-0032fcaf229e","Type":"ContainerStarted","Data":"9223db137fc82179e45bf6d50f31d26d92af0f429c704b5bea754a106c2316a3"}
Oct 30 01:11:22.763808 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:11:22.763742    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0aabe7a-6qthl" podStartSLOduration=1.763722934 podStartE2EDuration="1.763722934s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:11:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:11:22.762025969 +0000 UTC m=+2378.378458849" watchObservedRunningTime="2026-10-30 01:11:22.763722934 +0000 UTC m=+2378.380155814"
Oct 30 01:11:23.728571 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:11:23.728525    5002 generic.go:356] "Generic (PLEG): container finished" podID="d930a049-cee8-4106-a988-0032fcaf229e" containerID="a184ce67447af89684194ee1ccc178000b0677ce2c3d1a77772a0a08e111be34" exitCode=0
Oct 30 01:11:23.729073 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:11:23.728596    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0aabe7a-6qthl" event={"ID":"d930a049-cee8-4106-a988-0032fcaf229e","Type":"ContainerDied","Data":"a184ce67447af89684194ee1ccc178000b0677ce2c3d1a77772a0a08e111be34"}
Oct 30 01:11:24.922069 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:11:24.922039    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0aabe7a-6qthl"
Oct 30 01:11:25.026573 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:11:25.026479    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d930a049-cee8-4106-a988-0032fcaf229e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d930a049-cee8-4106-a988-0032fcaf229e-maas-api-xks-ca\") pod \"d930a049-cee8-4106-a988-0032fcaf229e\" (UID: \"d930a049-cee8-4106-a988-0032fcaf229e\") "
Oct 30 01:11:25.026573 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:11:25.026560    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d930a049-cee8-4106-a988-0032fcaf229e-kube-api-access-8jpm2\" (UniqueName: \"kubernetes.io/projected/d930a049-cee8-4106-a988-0032fcaf229e-kube-api-access-8jpm2\") pod \"d930a049-cee8-4106-a988-0032fcaf229e\" (UID: \"d930a049-cee8-4106-a988-0032fcaf229e\") "
Oct 30 01:11:25.026769 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:11:25.026698    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d930a049-cee8-4106-a988-0032fcaf229e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d930a049-cee8-4106-a988-0032fcaf229e-maas-api-openshift-ca\") pod \"d930a049-cee8-4106-a988-0032fcaf229e\" (UID: \"d930a049-cee8-4106-a988-0032fcaf229e\") "
Oct 30 01:11:25.027057 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:11:25.027029    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d930a049-cee8-4106-a988-0032fcaf229e-maas-api-openshift-ca" pod "d930a049-cee8-4106-a988-0032fcaf229e" (UID: "d930a049-cee8-4106-a988-0032fcaf229e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:11:25.027295 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:11:25.027277    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d930a049-cee8-4106-a988-0032fcaf229e-maas-api-openshift-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:11:25.028761 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:11:25.028738    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d930a049-cee8-4106-a988-0032fcaf229e-maas-api-xks-ca" pod "d930a049-cee8-4106-a988-0032fcaf229e" (UID: "d930a049-cee8-4106-a988-0032fcaf229e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:11:25.028874 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:11:25.028793    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d930a049-cee8-4106-a988-0032fcaf229e-kube-api-access-8jpm2" pod "d930a049-cee8-4106-a988-0032fcaf229e" (UID: "d930a049-cee8-4106-a988-0032fcaf229e"). InnerVolumeSpecName "kube-api-access-8jpm2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:11:25.128307 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:11:25.128261    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d930a049-cee8-4106-a988-0032fcaf229e-maas-api-xks-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:11:25.128508 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:11:25.128323    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8jpm2\" (UniqueName: \"kubernetes.io/projected/d930a049-cee8-4106-a988-0032fcaf229e-kube-api-access-8jpm2\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:11:25.744969 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:11:25.744889    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0aabe7a-6qthl"
Oct 30 01:11:25.744969 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:11:25.744922    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0aabe7a-6qthl" event={"ID":"d930a049-cee8-4106-a988-0032fcaf229e","Type":"ContainerDied","Data":"9223db137fc82179e45bf6d50f31d26d92af0f429c704b5bea754a106c2316a3"}
Oct 30 01:11:25.745172 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:11:25.744983    5002 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9223db137fc82179e45bf6d50f31d26d92af0f429c704b5bea754a106c2316a3"
Oct 30 01:11:29.766997 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:11:29.766962    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-6d44c2-openshift-default-69fb7659f-snvvd"]
Oct 30 01:11:29.768552 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:11:29.768524    5002 cpu_manager.go:414] "containerMap: removing container" podUID="d930a049-cee8-4106-a988-0032fcaf229e" containerName="revoke-keys"
Oct 30 01:11:29.768724 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:11:29.768700    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d930a049-cee8-4106-a988-0032fcaf229e" containerName="revoke-keys"
Oct 30 01:11:29.768955 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:11:29.768938    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="d930a049-cee8-4106-a988-0032fcaf229e" containerName="revoke-keys"
Oct 30 01:11:29.773365 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:11:29.773343    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-6d44c2-openshift-default-69fb7659f-snvvd"
Oct 30 01:11:29.807588 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:11:29.807558    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-6d44c2-openshift-default-69fb7659f-snvvd"]
Oct 30 01:11:29.811724 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:11:29.811694    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-6d44c2-openshift-default-dockercfg-z4t8c\""
Oct 30 01:11:29.827100 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:11:29.827065    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3f9e906-5594-4cdf-a620-efc9cbc28a44-istio-token\") pod \"e2e-ipp-b-6d44c2-openshift-default-69fb7659f-snvvd\" (UID: \"f3f9e906-5594-4cdf-a620-efc9cbc28a44\") " pod="openshift-ingress/e2e-ipp-b-6d44c2-openshift-default-69fb7659f-snvvd"
Oct 30 01:11:29.827259 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:11:29.827240    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3f9e906-5594-4cdf-a620-efc9cbc28a44-istio-data\") pod \"e2e-ipp-b-6d44c2-openshift-default-69fb7659f-snvvd\" (UID: \"f3f9e906-5594-4cdf-a620-efc9cbc28a44\") " pod="openshift-ingress/e2e-ipp-b-6d44c2-openshift-default-69fb7659f-snvvd"
Oct 30 01:11:29.827315 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:11:29.827276    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3f9e906-5594-4cdf-a620-efc9cbc28a44-istiod-ca-cert\") pod \"e2e-ipp-b-6d44c2-openshift-default-69fb7659f-snvvd\" (UID: \"f3f9e906-5594-4cdf-a620-efc9cbc28a44\") " pod="openshift-ingress/e2e-ipp-b-6d44c2-openshift-default-69fb7659f-snvvd"
Oct 30 01:11:29.827444 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:11:29.827426    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3f9e906-5594-4cdf-a620-efc9cbc28a44-workload-certs\") pod \"e2e-ipp-b-6d44c2-openshift-default-69fb7659f-snvvd\" (UID: \"f3f9e906-5594-4cdf-a620-efc9cbc28a44\") " pod="openshift-ingress/e2e-ipp-b-6d44c2-openshift-default-69fb7659f-snvvd"
Oct 30 01:11:29.827509 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:11:29.827491    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3f9e906-5594-4cdf-a620-efc9cbc28a44-workload-socket\") pod \"e2e-ipp-b-6d44c2-openshift-default-69fb7659f-snvvd\" (UID: \"f3f9e906-5594-4cdf-a620-efc9cbc28a44\") " pod="openshift-ingress/e2e-ipp-b-6d44c2-openshift-default-69fb7659f-snvvd"
Oct 30 01:11:29.827589 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:11:29.827574    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3f9e906-5594-4cdf-a620-efc9cbc28a44-credential-socket\") pod \"e2e-ipp-b-6d44c2-openshift-default-69fb7659f-snvvd\" (UID: \"f3f9e906-5594-4cdf-a620-efc9cbc28a44\") " pod="openshift-ingress/e2e-ipp-b-6d44c2-openshift-default-69fb7659f-snvvd"
Oct 30 01:11:29.827632 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:11:29.827600    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3f9e906-5594-4cdf-a620-efc9cbc28a44-istio-envoy\") pod \"e2e-ipp-b-6d44c2-openshift-default-69fb7659f-snvvd\" (UID: \"f3f9e906-5594-4cdf-a620-efc9cbc28a44\") " pod="openshift-ingress/e2e-ipp-b-6d44c2-openshift-default-69fb7659f-snvvd"
Oct 30 01:11:29.827782 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:11:29.827764    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbhqz\" (UniqueName: \"kubernetes.io/projected/f3f9e906-5594-4cdf-a620-efc9cbc28a44-kube-api-access-qbhqz\") pod \"e2e-ipp-b-6d44c2-openshift-default-69fb7659f-snvvd\" (UID: \"f3f9e906-5594-4cdf-a620-efc9cbc28a44\") " pod="openshift-ingress/e2e-ipp-b-6d44c2-openshift-default-69fb7659f-snvvd"
Oct 30 01:11:29.827831 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:11:29.827818    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3f9e906-5594-4cdf-a620-efc9cbc28a44-istio-podinfo\") pod \"e2e-ipp-b-6d44c2-openshift-default-69fb7659f-snvvd\" (UID: \"f3f9e906-5594-4cdf-a620-efc9cbc28a44\") " pod="openshift-ingress/e2e-ipp-b-6d44c2-openshift-default-69fb7659f-snvvd"
Oct 30 01:11:29.965535 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:11:29.965495    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbhqz\" (UniqueName: \"kubernetes.io/projected/f3f9e906-5594-4cdf-a620-efc9cbc28a44-kube-api-access-qbhqz\") pod \"e2e-ipp-b-6d44c2-openshift-default-69fb7659f-snvvd\" (UID: \"f3f9e906-5594-4cdf-a620-efc9cbc28a44\") " pod="openshift-ingress/e2e-ipp-b-6d44c2-openshift-default-69fb7659f-snvvd"
Oct 30 01:11:29.966050 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:11:29.966022    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3f9e906-5594-4cdf-a620-efc9cbc28a44-istio-podinfo\") pod \"e2e-ipp-b-6d44c2-openshift-default-69fb7659f-snvvd\" (UID: \"f3f9e906-5594-4cdf-a620-efc9cbc28a44\") " pod="openshift-ingress/e2e-ipp-b-6d44c2-openshift-default-69fb7659f-snvvd"
Oct 30 01:11:29.966825 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:11:29.966804    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3f9e906-5594-4cdf-a620-efc9cbc28a44-istio-token\") pod \"e2e-ipp-b-6d44c2-openshift-default-69fb7659f-snvvd\" (UID: \"f3f9e906-5594-4cdf-a620-efc9cbc28a44\") " pod="openshift-ingress/e2e-ipp-b-6d44c2-openshift-default-69fb7659f-snvvd"
Oct 30 01:11:29.967636 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:11:29.967608    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3f9e906-5594-4cdf-a620-efc9cbc28a44-istio-data\") pod \"e2e-ipp-b-6d44c2-openshift-default-69fb7659f-snvvd\" (UID: \"f3f9e906-5594-4cdf-a620-efc9cbc28a44\") " pod="openshift-ingress/e2e-ipp-b-6d44c2-openshift-default-69fb7659f-snvvd"
Oct 30 01:11:29.967842 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:11:29.967795    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3f9e906-5594-4cdf-a620-efc9cbc28a44-istiod-ca-cert\") pod \"e2e-ipp-b-6d44c2-openshift-default-69fb7659f-snvvd\" (UID: \"f3f9e906-5594-4cdf-a620-efc9cbc28a44\") " pod="openshift-ingress/e2e-ipp-b-6d44c2-openshift-default-69fb7659f-snvvd"
Oct 30 01:11:29.969089 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:11:29.968613    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3f9e906-5594-4cdf-a620-efc9cbc28a44-istio-data\") pod \"e2e-ipp-b-6d44c2-openshift-default-69fb7659f-snvvd\" (UID: \"f3f9e906-5594-4cdf-a620-efc9cbc28a44\") " pod="openshift-ingress/e2e-ipp-b-6d44c2-openshift-default-69fb7659f-snvvd"
Oct 30 01:11:29.969089 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:11:29.968725    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3f9e906-5594-4cdf-a620-efc9cbc28a44-workload-certs\") pod \"e2e-ipp-b-6d44c2-openshift-default-69fb7659f-snvvd\" (UID: \"f3f9e906-5594-4cdf-a620-efc9cbc28a44\") " pod="openshift-ingress/e2e-ipp-b-6d44c2-openshift-default-69fb7659f-snvvd"
Oct 30 01:11:29.969089 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:11:29.968801    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3f9e906-5594-4cdf-a620-efc9cbc28a44-workload-socket\") pod \"e2e-ipp-b-6d44c2-openshift-default-69fb7659f-snvvd\" (UID: \"f3f9e906-5594-4cdf-a620-efc9cbc28a44\") " pod="openshift-ingress/e2e-ipp-b-6d44c2-openshift-default-69fb7659f-snvvd"
Oct 30 01:11:29.969089 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:11:29.968844    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3f9e906-5594-4cdf-a620-efc9cbc28a44-credential-socket\") pod \"e2e-ipp-b-6d44c2-openshift-default-69fb7659f-snvvd\" (UID: \"f3f9e906-5594-4cdf-a620-efc9cbc28a44\") " pod="openshift-ingress/e2e-ipp-b-6d44c2-openshift-default-69fb7659f-snvvd"
Oct 30 01:11:29.974401 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:11:29.969451    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3f9e906-5594-4cdf-a620-efc9cbc28a44-workload-socket\") pod \"e2e-ipp-b-6d44c2-openshift-default-69fb7659f-snvvd\" (UID: \"f3f9e906-5594-4cdf-a620-efc9cbc28a44\") " pod="openshift-ingress/e2e-ipp-b-6d44c2-openshift-default-69fb7659f-snvvd"
Oct 30 01:11:29.974401 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:11:29.969677    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3f9e906-5594-4cdf-a620-efc9cbc28a44-workload-certs\") pod \"e2e-ipp-b-6d44c2-openshift-default-69fb7659f-snvvd\" (UID: \"f3f9e906-5594-4cdf-a620-efc9cbc28a44\") " pod="openshift-ingress/e2e-ipp-b-6d44c2-openshift-default-69fb7659f-snvvd"
Oct 30 01:11:29.974401 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:11:29.969720    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3f9e906-5594-4cdf-a620-efc9cbc28a44-istio-envoy\") pod \"e2e-ipp-b-6d44c2-openshift-default-69fb7659f-snvvd\" (UID: \"f3f9e906-5594-4cdf-a620-efc9cbc28a44\") " pod="openshift-ingress/e2e-ipp-b-6d44c2-openshift-default-69fb7659f-snvvd"
Oct 30 01:11:29.974879 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:11:29.974855    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3f9e906-5594-4cdf-a620-efc9cbc28a44-credential-socket\") pod \"e2e-ipp-b-6d44c2-openshift-default-69fb7659f-snvvd\" (UID: \"f3f9e906-5594-4cdf-a620-efc9cbc28a44\") " pod="openshift-ingress/e2e-ipp-b-6d44c2-openshift-default-69fb7659f-snvvd"
Oct 30 01:11:29.987485 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:11:29.987438    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3f9e906-5594-4cdf-a620-efc9cbc28a44-istiod-ca-cert\") pod \"e2e-ipp-b-6d44c2-openshift-default-69fb7659f-snvvd\" (UID: \"f3f9e906-5594-4cdf-a620-efc9cbc28a44\") " pod="openshift-ingress/e2e-ipp-b-6d44c2-openshift-default-69fb7659f-snvvd"
Oct 30 01:11:30.023094 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:11:30.022979    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3f9e906-5594-4cdf-a620-efc9cbc28a44-istio-envoy\") pod \"e2e-ipp-b-6d44c2-openshift-default-69fb7659f-snvvd\" (UID: \"f3f9e906-5594-4cdf-a620-efc9cbc28a44\") " pod="openshift-ingress/e2e-ipp-b-6d44c2-openshift-default-69fb7659f-snvvd"
Oct 30 01:11:30.023932 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:11:30.023906    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3f9e906-5594-4cdf-a620-efc9cbc28a44-istio-podinfo\") pod \"e2e-ipp-b-6d44c2-openshift-default-69fb7659f-snvvd\" (UID: \"f3f9e906-5594-4cdf-a620-efc9cbc28a44\") " pod="openshift-ingress/e2e-ipp-b-6d44c2-openshift-default-69fb7659f-snvvd"
Oct 30 01:11:30.052499 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:11:30.052464    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3f9e906-5594-4cdf-a620-efc9cbc28a44-istio-token\") pod \"e2e-ipp-b-6d44c2-openshift-default-69fb7659f-snvvd\" (UID: \"f3f9e906-5594-4cdf-a620-efc9cbc28a44\") " pod="openshift-ingress/e2e-ipp-b-6d44c2-openshift-default-69fb7659f-snvvd"
Oct 30 01:11:30.053523 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:11:30.053495    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbhqz\" (UniqueName: \"kubernetes.io/projected/f3f9e906-5594-4cdf-a620-efc9cbc28a44-kube-api-access-qbhqz\") pod \"e2e-ipp-b-6d44c2-openshift-default-69fb7659f-snvvd\" (UID: \"f3f9e906-5594-4cdf-a620-efc9cbc28a44\") " pod="openshift-ingress/e2e-ipp-b-6d44c2-openshift-default-69fb7659f-snvvd"
Oct 30 01:11:30.117887 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:11:30.117847    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-6d44c2-openshift-default-69fb7659f-snvvd"
Oct 30 01:11:30.586110 ip-10-0-128-7 kubenswrapper[5002]: W1030 01:11:30.586076    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3f9e906_5594_4cdf_a620_efc9cbc28a44.slice/crio-ee9990e89d0b13463b6865740de3fc8535fd4812e1e7efc553a2dbbc90e74c19 WatchSource:0}: Error finding container ee9990e89d0b13463b6865740de3fc8535fd4812e1e7efc553a2dbbc90e74c19: Status 404 returned error can't find the container with id ee9990e89d0b13463b6865740de3fc8535fd4812e1e7efc553a2dbbc90e74c19
Oct 30 01:11:30.587722 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:11:30.587689    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-6d44c2-openshift-default-69fb7659f-snvvd"]
Oct 30 01:11:30.589232 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:11:30.589198    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:11:30.589335 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:11:30.589266    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:11:30.589335 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:11:30.589296    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:11:30.809638 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:11:30.809593    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-6d44c2-openshift-default-69fb7659f-snvvd" event={"ID":"f3f9e906-5594-4cdf-a620-efc9cbc28a44","Type":"ContainerStarted","Data":"670c9acd81e92e790db801790362f50310923577a102c2934548080f2d5fd45e"}
Oct 30 01:11:30.810100 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:11:30.809667    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-6d44c2-openshift-default-69fb7659f-snvvd" event={"ID":"f3f9e906-5594-4cdf-a620-efc9cbc28a44","Type":"ContainerStarted","Data":"ee9990e89d0b13463b6865740de3fc8535fd4812e1e7efc553a2dbbc90e74c19"}
Oct 30 01:11:30.860939 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:11:30.860833    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-6d44c2-openshift-default-69fb7659f-snvvd" podStartSLOduration=1.8608131060000002 podStartE2EDuration="1.860813106s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:11:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:11:30.858086745 +0000 UTC m=+2386.474519624" watchObservedRunningTime="2026-10-30 01:11:30.860813106 +0000 UTC m=+2386.477245987"
Oct 30 01:11:31.134052 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:11:31.133961    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-6d44c2-openshift-default-69fb7659f-snvvd"
Oct 30 01:11:31.136368 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:11:31.136335    5002 patch_prober.go:29] interesting pod/e2e-ipp-b-6d44c2-openshift-default-69fb7659f-snvvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" start-of-body=
Oct 30 01:11:31.136560 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:11:31.136451    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-6d44c2-openshift-default-69fb7659f-snvvd" podUID="f3f9e906-5594-4cdf-a620-efc9cbc28a44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused"
Oct 30 01:11:31.807410 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:11:31.807333    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4b8027ba-bz2kg"]
Oct 30 01:11:31.813111 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:11:31.813079    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b8027ba-bz2kg"
Oct 30 01:11:31.830742 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:11:31.830708    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4b8027ba-bz2kg"]
Oct 30 01:11:31.837883 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:11:31.837852    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:11:31.838078 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:11:31.837858    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:11:31.838193 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:11:31.838016    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:11:31.843300 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:11:31.843265    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfdzq\" (UniqueName: \"kubernetes.io/projected/bf6bfe10-0128-4cb2-93f6-1eb9eed7f8ad-kube-api-access-kfdzq\") pod \"maas-api-revoke-sub-4b8027ba-bz2kg\" (UID: \"bf6bfe10-0128-4cb2-93f6-1eb9eed7f8ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b8027ba-bz2kg"
Oct 30 01:11:31.843513 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:11:31.843346    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf6bfe10-0128-4cb2-93f6-1eb9eed7f8ad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4b8027ba-bz2kg\" (UID: \"bf6bfe10-0128-4cb2-93f6-1eb9eed7f8ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b8027ba-bz2kg"
Oct 30 01:11:31.843605 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:11:31.843568    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf6bfe10-0128-4cb2-93f6-1eb9eed7f8ad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4b8027ba-bz2kg\" (UID: \"bf6bfe10-0128-4cb2-93f6-1eb9eed7f8ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b8027ba-bz2kg"
Oct 30 01:11:31.843835 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:11:31.843802    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5phsh\""
Oct 30 01:11:31.945484 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:11:31.945439    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfdzq\" (UniqueName: \"kubernetes.io/projected/bf6bfe10-0128-4cb2-93f6-1eb9eed7f8ad-kube-api-access-kfdzq\") pod \"maas-api-revoke-sub-4b8027ba-bz2kg\" (UID: \"bf6bfe10-0128-4cb2-93f6-1eb9eed7f8ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b8027ba-bz2kg"
Oct 30 01:11:31.945664 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:11:31.945536    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf6bfe10-0128-4cb2-93f6-1eb9eed7f8ad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4b8027ba-bz2kg\" (UID: \"bf6bfe10-0128-4cb2-93f6-1eb9eed7f8ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b8027ba-bz2kg"
Oct 30 01:11:31.945746 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:11:31.945685    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf6bfe10-0128-4cb2-93f6-1eb9eed7f8ad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4b8027ba-bz2kg\" (UID: \"bf6bfe10-0128-4cb2-93f6-1eb9eed7f8ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b8027ba-bz2kg"
Oct 30 01:11:31.946300 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:11:31.946272    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf6bfe10-0128-4cb2-93f6-1eb9eed7f8ad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4b8027ba-bz2kg\" (UID: \"bf6bfe10-0128-4cb2-93f6-1eb9eed7f8ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b8027ba-bz2kg"
Oct 30 01:11:31.948124 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:11:31.948103    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf6bfe10-0128-4cb2-93f6-1eb9eed7f8ad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4b8027ba-bz2kg\" (UID: \"bf6bfe10-0128-4cb2-93f6-1eb9eed7f8ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b8027ba-bz2kg"
Oct 30 01:11:31.969962 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:11:31.969935    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfdzq\" (UniqueName: \"kubernetes.io/projected/bf6bfe10-0128-4cb2-93f6-1eb9eed7f8ad-kube-api-access-kfdzq\") pod \"maas-api-revoke-sub-4b8027ba-bz2kg\" (UID: \"bf6bfe10-0128-4cb2-93f6-1eb9eed7f8ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b8027ba-bz2kg"
Oct 30 01:11:32.132723 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:11:32.132635    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b8027ba-bz2kg"
Oct 30 01:11:32.144434 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:11:32.144361    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-6d44c2-openshift-default-69fb7659f-snvvd"
Oct 30 01:11:32.451050 ip-10-0-128-7 kubenswrapper[5002]: W1030 01:11:32.451003    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf6bfe10_0128_4cb2_93f6_1eb9eed7f8ad.slice/crio-1a7a1ce78136f5923e1b5f107466a06138fb89bc8413c81589c38cf5b0e7301d WatchSource:0}: Error finding container 1a7a1ce78136f5923e1b5f107466a06138fb89bc8413c81589c38cf5b0e7301d: Status 404 returned error can't find the container with id 1a7a1ce78136f5923e1b5f107466a06138fb89bc8413c81589c38cf5b0e7301d
Oct 30 01:11:32.451342 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:11:32.451310    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4b8027ba-bz2kg"]
Oct 30 01:11:32.832328 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:11:32.832238    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b8027ba-bz2kg" event={"ID":"bf6bfe10-0128-4cb2-93f6-1eb9eed7f8ad","Type":"ContainerStarted","Data":"1a7a1ce78136f5923e1b5f107466a06138fb89bc8413c81589c38cf5b0e7301d"}
Oct 30 01:11:32.832785 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:11:32.832664    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-6d44c2-openshift-default-69fb7659f-snvvd"
Oct 30 01:11:32.833533 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:11:32.833513    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-6d44c2-openshift-default-69fb7659f-snvvd"
Oct 30 01:11:33.842741 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:11:33.842651    5002 generic.go:356] "Generic (PLEG): container finished" podID="bf6bfe10-0128-4cb2-93f6-1eb9eed7f8ad" containerID="458905a468f2867fa4f492d43c839631ef4cdb302bad5886770c56fb71efba89" exitCode=0
Oct 30 01:11:33.842741 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:11:33.842722    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b8027ba-bz2kg" event={"ID":"bf6bfe10-0128-4cb2-93f6-1eb9eed7f8ad","Type":"ContainerDied","Data":"458905a468f2867fa4f492d43c839631ef4cdb302bad5886770c56fb71efba89"}
Oct 30 01:11:35.040037 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:11:35.040009    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b8027ba-bz2kg"
Oct 30 01:11:35.134474 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:11:35.134367    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf6bfe10-0128-4cb2-93f6-1eb9eed7f8ad-kube-api-access-kfdzq\" (UniqueName: \"kubernetes.io/projected/bf6bfe10-0128-4cb2-93f6-1eb9eed7f8ad-kube-api-access-kfdzq\") pod \"bf6bfe10-0128-4cb2-93f6-1eb9eed7f8ad\" (UID: \"bf6bfe10-0128-4cb2-93f6-1eb9eed7f8ad\") "
Oct 30 01:11:35.134635 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:11:35.134510    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf6bfe10-0128-4cb2-93f6-1eb9eed7f8ad-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf6bfe10-0128-4cb2-93f6-1eb9eed7f8ad-maas-api-openshift-ca\") pod \"bf6bfe10-0128-4cb2-93f6-1eb9eed7f8ad\" (UID: \"bf6bfe10-0128-4cb2-93f6-1eb9eed7f8ad\") "
Oct 30 01:11:35.134719 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:11:35.134698    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bf6bfe10-0128-4cb2-93f6-1eb9eed7f8ad-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf6bfe10-0128-4cb2-93f6-1eb9eed7f8ad-maas-api-xks-ca\") pod \"bf6bfe10-0128-4cb2-93f6-1eb9eed7f8ad\" (UID: \"bf6bfe10-0128-4cb2-93f6-1eb9eed7f8ad\") "
Oct 30 01:11:35.134996 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:11:35.134963    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf6bfe10-0128-4cb2-93f6-1eb9eed7f8ad-maas-api-openshift-ca" pod "bf6bfe10-0128-4cb2-93f6-1eb9eed7f8ad" (UID: "bf6bfe10-0128-4cb2-93f6-1eb9eed7f8ad"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:11:35.135338 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:11:35.135312    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf6bfe10-0128-4cb2-93f6-1eb9eed7f8ad-maas-api-openshift-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:11:35.137027 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:11:35.136998    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf6bfe10-0128-4cb2-93f6-1eb9eed7f8ad-kube-api-access-kfdzq" pod "bf6bfe10-0128-4cb2-93f6-1eb9eed7f8ad" (UID: "bf6bfe10-0128-4cb2-93f6-1eb9eed7f8ad"). InnerVolumeSpecName "kube-api-access-kfdzq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:11:35.137532 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:11:35.137510    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf6bfe10-0128-4cb2-93f6-1eb9eed7f8ad-maas-api-xks-ca" pod "bf6bfe10-0128-4cb2-93f6-1eb9eed7f8ad" (UID: "bf6bfe10-0128-4cb2-93f6-1eb9eed7f8ad"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:11:35.236979 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:11:35.236938    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kfdzq\" (UniqueName: \"kubernetes.io/projected/bf6bfe10-0128-4cb2-93f6-1eb9eed7f8ad-kube-api-access-kfdzq\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:11:35.236979 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:11:35.236982    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf6bfe10-0128-4cb2-93f6-1eb9eed7f8ad-maas-api-xks-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:11:36.067466 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:11:36.022862    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b8027ba-bz2kg" event={"ID":"bf6bfe10-0128-4cb2-93f6-1eb9eed7f8ad","Type":"ContainerDied","Data":"1a7a1ce78136f5923e1b5f107466a06138fb89bc8413c81589c38cf5b0e7301d"}
Oct 30 01:11:36.067466 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:11:36.023161    5002 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1a7a1ce78136f5923e1b5f107466a06138fb89bc8413c81589c38cf5b0e7301d"
Oct 30 01:11:36.067466 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:11:36.023343    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b8027ba-bz2kg"
Oct 30 01:11:42.862414 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:11:42.862344    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbjdq" containerName="extractor"
Oct 30 01:11:43.119694 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:11:43.119610    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w5-a5e9ab-openshift-default-8498d5b8b9-vr4jt"]
Oct 30 01:11:43.120648 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:11:43.120627    5002 cpu_manager.go:414] "containerMap: removing container" podUID="bf6bfe10-0128-4cb2-93f6-1eb9eed7f8ad" containerName="revoke-keys"
Oct 30 01:11:43.120698 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:11:43.120650    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf6bfe10-0128-4cb2-93f6-1eb9eed7f8ad" containerName="revoke-keys"
Oct 30 01:11:43.120802 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:11:43.120777    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf6bfe10-0128-4cb2-93f6-1eb9eed7f8ad" containerName="revoke-keys"
Oct 30 01:11:43.124443 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:11:43.124422    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-a5e9ab-openshift-default-8498d5b8b9-vr4jt"
Oct 30 01:11:43.157511 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:11:43.155847    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w5-a5e9ab-openshift-default-dockercfg-f6wp2\""
Oct 30 01:11:43.163188 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:11:43.163155    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-a5e9ab-openshift-default-8498d5b8b9-vr4jt"]
Oct 30 01:11:43.225963 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:11:43.225926    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a61107c1-4b26-4cb7-85d3-eb83cad66da6-istio-token\") pod \"e2e-shared-b-w5-a5e9ab-openshift-default-8498d5b8b9-vr4jt\" (UID: \"a61107c1-4b26-4cb7-85d3-eb83cad66da6\") " pod="openshift-ingress/e2e-shared-b-w5-a5e9ab-openshift-default-8498d5b8b9-vr4jt"
Oct 30 01:11:43.225963 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:11:43.225972    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a61107c1-4b26-4cb7-85d3-eb83cad66da6-istiod-ca-cert\") pod \"e2e-shared-b-w5-a5e9ab-openshift-default-8498d5b8b9-vr4jt\" (UID: \"a61107c1-4b26-4cb7-85d3-eb83cad66da6\") " pod="openshift-ingress/e2e-shared-b-w5-a5e9ab-openshift-default-8498d5b8b9-vr4jt"
Oct 30 01:11:43.226195 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:11:43.226089    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4dq9\" (UniqueName: \"kubernetes.io/projected/a61107c1-4b26-4cb7-85d3-eb83cad66da6-kube-api-access-m4dq9\") pod \"e2e-shared-b-w5-a5e9ab-openshift-default-8498d5b8b9-vr4jt\" (UID: \"a61107c1-4b26-4cb7-85d3-eb83cad66da6\") " pod="openshift-ingress/e2e-shared-b-w5-a5e9ab-openshift-default-8498d5b8b9-vr4jt"
Oct 30 01:11:43.226195 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:11:43.226175    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a61107c1-4b26-4cb7-85d3-eb83cad66da6-istio-data\") pod \"e2e-shared-b-w5-a5e9ab-openshift-default-8498d5b8b9-vr4jt\" (UID: \"a61107c1-4b26-4cb7-85d3-eb83cad66da6\") " pod="openshift-ingress/e2e-shared-b-w5-a5e9ab-openshift-default-8498d5b8b9-vr4jt"
Oct 30 01:11:43.226282 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:11:43.226224    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a61107c1-4b26-4cb7-85d3-eb83cad66da6-workload-socket\") pod \"e2e-shared-b-w5-a5e9ab-openshift-default-8498d5b8b9-vr4jt\" (UID: \"a61107c1-4b26-4cb7-85d3-eb83cad66da6\") " pod="openshift-ingress/e2e-shared-b-w5-a5e9ab-openshift-default-8498d5b8b9-vr4jt"
Oct 30 01:11:43.226282 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:11:43.226278    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a61107c1-4b26-4cb7-85d3-eb83cad66da6-istio-podinfo\") pod \"e2e-shared-b-w5-a5e9ab-openshift-default-8498d5b8b9-vr4jt\" (UID: \"a61107c1-4b26-4cb7-85d3-eb83cad66da6\") " pod="openshift-ingress/e2e-shared-b-w5-a5e9ab-openshift-default-8498d5b8b9-vr4jt"
Oct 30 01:11:43.226374 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:11:43.226331    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a61107c1-4b26-4cb7-85d3-eb83cad66da6-istio-envoy\") pod \"e2e-shared-b-w5-a5e9ab-openshift-default-8498d5b8b9-vr4jt\" (UID: \"a61107c1-4b26-4cb7-85d3-eb83cad66da6\") " pod="openshift-ingress/e2e-shared-b-w5-a5e9ab-openshift-default-8498d5b8b9-vr4jt"
Oct 30 01:11:43.226561 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:11:43.226537    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a61107c1-4b26-4cb7-85d3-eb83cad66da6-credential-socket\") pod \"e2e-shared-b-w5-a5e9ab-openshift-default-8498d5b8b9-vr4jt\" (UID: \"a61107c1-4b26-4cb7-85d3-eb83cad66da6\") " pod="openshift-ingress/e2e-shared-b-w5-a5e9ab-openshift-default-8498d5b8b9-vr4jt"
Oct 30 01:11:43.226631 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:11:43.226577    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a61107c1-4b26-4cb7-85d3-eb83cad66da6-workload-certs\") pod \"e2e-shared-b-w5-a5e9ab-openshift-default-8498d5b8b9-vr4jt\" (UID: \"a61107c1-4b26-4cb7-85d3-eb83cad66da6\") " pod="openshift-ingress/e2e-shared-b-w5-a5e9ab-openshift-default-8498d5b8b9-vr4jt"
Oct 30 01:11:43.328658 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:11:43.328618    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a61107c1-4b26-4cb7-85d3-eb83cad66da6-credential-socket\") pod \"e2e-shared-b-w5-a5e9ab-openshift-default-8498d5b8b9-vr4jt\" (UID: \"a61107c1-4b26-4cb7-85d3-eb83cad66da6\") " pod="openshift-ingress/e2e-shared-b-w5-a5e9ab-openshift-default-8498d5b8b9-vr4jt"
Oct 30 01:11:43.328658 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:11:43.328661    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a61107c1-4b26-4cb7-85d3-eb83cad66da6-workload-certs\") pod \"e2e-shared-b-w5-a5e9ab-openshift-default-8498d5b8b9-vr4jt\" (UID: \"a61107c1-4b26-4cb7-85d3-eb83cad66da6\") " pod="openshift-ingress/e2e-shared-b-w5-a5e9ab-openshift-default-8498d5b8b9-vr4jt"
Oct 30 01:11:43.328933 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:11:43.328737    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a61107c1-4b26-4cb7-85d3-eb83cad66da6-istio-token\") pod \"e2e-shared-b-w5-a5e9ab-openshift-default-8498d5b8b9-vr4jt\" (UID: \"a61107c1-4b26-4cb7-85d3-eb83cad66da6\") " pod="openshift-ingress/e2e-shared-b-w5-a5e9ab-openshift-default-8498d5b8b9-vr4jt"
Oct 30 01:11:43.328933 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:11:43.328764    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a61107c1-4b26-4cb7-85d3-eb83cad66da6-istiod-ca-cert\") pod \"e2e-shared-b-w5-a5e9ab-openshift-default-8498d5b8b9-vr4jt\" (UID: \"a61107c1-4b26-4cb7-85d3-eb83cad66da6\") " pod="openshift-ingress/e2e-shared-b-w5-a5e9ab-openshift-default-8498d5b8b9-vr4jt"
Oct 30 01:11:43.328933 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:11:43.328820    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4dq9\" (UniqueName: \"kubernetes.io/projected/a61107c1-4b26-4cb7-85d3-eb83cad66da6-kube-api-access-m4dq9\") pod \"e2e-shared-b-w5-a5e9ab-openshift-default-8498d5b8b9-vr4jt\" (UID: \"a61107c1-4b26-4cb7-85d3-eb83cad66da6\") " pod="openshift-ingress/e2e-shared-b-w5-a5e9ab-openshift-default-8498d5b8b9-vr4jt"
Oct 30 01:11:43.328933 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:11:43.328889    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a61107c1-4b26-4cb7-85d3-eb83cad66da6-istio-data\") pod \"e2e-shared-b-w5-a5e9ab-openshift-default-8498d5b8b9-vr4jt\" (UID: \"a61107c1-4b26-4cb7-85d3-eb83cad66da6\") " pod="openshift-ingress/e2e-shared-b-w5-a5e9ab-openshift-default-8498d5b8b9-vr4jt"
Oct 30 01:11:43.329191 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:11:43.328938    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a61107c1-4b26-4cb7-85d3-eb83cad66da6-workload-socket\") pod \"e2e-shared-b-w5-a5e9ab-openshift-default-8498d5b8b9-vr4jt\" (UID: \"a61107c1-4b26-4cb7-85d3-eb83cad66da6\") " pod="openshift-ingress/e2e-shared-b-w5-a5e9ab-openshift-default-8498d5b8b9-vr4jt"
Oct 30 01:11:43.329191 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:11:43.328983    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a61107c1-4b26-4cb7-85d3-eb83cad66da6-istio-podinfo\") pod \"e2e-shared-b-w5-a5e9ab-openshift-default-8498d5b8b9-vr4jt\" (UID: \"a61107c1-4b26-4cb7-85d3-eb83cad66da6\") " pod="openshift-ingress/e2e-shared-b-w5-a5e9ab-openshift-default-8498d5b8b9-vr4jt"
Oct 30 01:11:43.329191 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:11:43.329058    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a61107c1-4b26-4cb7-85d3-eb83cad66da6-istio-envoy\") pod \"e2e-shared-b-w5-a5e9ab-openshift-default-8498d5b8b9-vr4jt\" (UID: \"a61107c1-4b26-4cb7-85d3-eb83cad66da6\") " pod="openshift-ingress/e2e-shared-b-w5-a5e9ab-openshift-default-8498d5b8b9-vr4jt"
Oct 30 01:11:43.329191 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:11:43.329095    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a61107c1-4b26-4cb7-85d3-eb83cad66da6-credential-socket\") pod \"e2e-shared-b-w5-a5e9ab-openshift-default-8498d5b8b9-vr4jt\" (UID: \"a61107c1-4b26-4cb7-85d3-eb83cad66da6\") " pod="openshift-ingress/e2e-shared-b-w5-a5e9ab-openshift-default-8498d5b8b9-vr4jt"
Oct 30 01:11:43.329406 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:11:43.329210    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a61107c1-4b26-4cb7-85d3-eb83cad66da6-workload-certs\") pod \"e2e-shared-b-w5-a5e9ab-openshift-default-8498d5b8b9-vr4jt\" (UID: \"a61107c1-4b26-4cb7-85d3-eb83cad66da6\") " pod="openshift-ingress/e2e-shared-b-w5-a5e9ab-openshift-default-8498d5b8b9-vr4jt"
Oct 30 01:11:43.329406 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:11:43.329275    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a61107c1-4b26-4cb7-85d3-eb83cad66da6-istio-data\") pod \"e2e-shared-b-w5-a5e9ab-openshift-default-8498d5b8b9-vr4jt\" (UID: \"a61107c1-4b26-4cb7-85d3-eb83cad66da6\") " pod="openshift-ingress/e2e-shared-b-w5-a5e9ab-openshift-default-8498d5b8b9-vr4jt"
Oct 30 01:11:43.329406 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:11:43.329353    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a61107c1-4b26-4cb7-85d3-eb83cad66da6-workload-socket\") pod \"e2e-shared-b-w5-a5e9ab-openshift-default-8498d5b8b9-vr4jt\" (UID: \"a61107c1-4b26-4cb7-85d3-eb83cad66da6\") " pod="openshift-ingress/e2e-shared-b-w5-a5e9ab-openshift-default-8498d5b8b9-vr4jt"
Oct 30 01:11:43.329749 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:11:43.329722    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a61107c1-4b26-4cb7-85d3-eb83cad66da6-istiod-ca-cert\") pod \"e2e-shared-b-w5-a5e9ab-openshift-default-8498d5b8b9-vr4jt\" (UID: \"a61107c1-4b26-4cb7-85d3-eb83cad66da6\") " pod="openshift-ingress/e2e-shared-b-w5-a5e9ab-openshift-default-8498d5b8b9-vr4jt"
Oct 30 01:11:43.332051 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:11:43.332020    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a61107c1-4b26-4cb7-85d3-eb83cad66da6-istio-envoy\") pod \"e2e-shared-b-w5-a5e9ab-openshift-default-8498d5b8b9-vr4jt\" (UID: \"a61107c1-4b26-4cb7-85d3-eb83cad66da6\") " pod="openshift-ingress/e2e-shared-b-w5-a5e9ab-openshift-default-8498d5b8b9-vr4jt"
Oct 30 01:11:43.332496 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:11:43.332476    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a61107c1-4b26-4cb7-85d3-eb83cad66da6-istio-podinfo\") pod \"e2e-shared-b-w5-a5e9ab-openshift-default-8498d5b8b9-vr4jt\" (UID: \"a61107c1-4b26-4cb7-85d3-eb83cad66da6\") " pod="openshift-ingress/e2e-shared-b-w5-a5e9ab-openshift-default-8498d5b8b9-vr4jt"
Oct 30 01:11:43.354107 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:11:43.354075    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a61107c1-4b26-4cb7-85d3-eb83cad66da6-istio-token\") pod \"e2e-shared-b-w5-a5e9ab-openshift-default-8498d5b8b9-vr4jt\" (UID: \"a61107c1-4b26-4cb7-85d3-eb83cad66da6\") " pod="openshift-ingress/e2e-shared-b-w5-a5e9ab-openshift-default-8498d5b8b9-vr4jt"
Oct 30 01:11:43.354240 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:11:43.354122    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4dq9\" (UniqueName: \"kubernetes.io/projected/a61107c1-4b26-4cb7-85d3-eb83cad66da6-kube-api-access-m4dq9\") pod \"e2e-shared-b-w5-a5e9ab-openshift-default-8498d5b8b9-vr4jt\" (UID: \"a61107c1-4b26-4cb7-85d3-eb83cad66da6\") " pod="openshift-ingress/e2e-shared-b-w5-a5e9ab-openshift-default-8498d5b8b9-vr4jt"
Oct 30 01:11:43.440014 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:11:43.439980    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-a5e9ab-openshift-default-8498d5b8b9-vr4jt"
Oct 30 01:11:43.778302 ip-10-0-128-7 kubenswrapper[5002]: W1030 01:11:43.778255    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda61107c1_4b26_4cb7_85d3_eb83cad66da6.slice/crio-56cc6c5936988295e798157d244ac83ae4a6b72b7fe41ae9ba4ecf5e81ef3216 WatchSource:0}: Error finding container 56cc6c5936988295e798157d244ac83ae4a6b72b7fe41ae9ba4ecf5e81ef3216: Status 404 returned error can't find the container with id 56cc6c5936988295e798157d244ac83ae4a6b72b7fe41ae9ba4ecf5e81ef3216
Oct 30 01:11:43.779122 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:11:43.779093    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-a5e9ab-openshift-default-8498d5b8b9-vr4jt"]
Oct 30 01:11:43.781264 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:11:43.781225    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:11:43.781368 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:11:43.781307    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:11:43.781368 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:11:43.781352    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:11:44.094542 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:11:44.094422    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-a5e9ab-openshift-default-8498d5b8b9-vr4jt" event={"ID":"a61107c1-4b26-4cb7-85d3-eb83cad66da6","Type":"ContainerStarted","Data":"bade7345ee202ad59779741a0d89b828943e1db291e90b257a79d3e8a207cfae"}
Oct 30 01:11:44.094542 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:11:44.094490    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-a5e9ab-openshift-default-8498d5b8b9-vr4jt" event={"ID":"a61107c1-4b26-4cb7-85d3-eb83cad66da6","Type":"ContainerStarted","Data":"56cc6c5936988295e798157d244ac83ae4a6b72b7fe41ae9ba4ecf5e81ef3216"}
Oct 30 01:11:44.145479 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:11:44.145409    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w5-a5e9ab-openshift-default-8498d5b8b9-vr4jt" podStartSLOduration=1.145372274 podStartE2EDuration="1.145372274s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:11:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:11:44.144440491 +0000 UTC m=+2399.760873394" watchObservedRunningTime="2026-10-30 01:11:44.145372274 +0000 UTC m=+2399.761805153"
Oct 30 01:11:44.447174 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:11:44.446209    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w5-a5e9ab-openshift-default-8498d5b8b9-vr4jt"
Oct 30 01:11:44.449246 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:11:44.447971    5002 patch_prober.go:29] interesting pod/e2e-shared-b-w5-a5e9ab-openshift-default-8498d5b8b9-vr4jt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" start-of-body=
Oct 30 01:11:44.449246 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:11:44.448016    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-a5e9ab-openshift-default-8498d5b8b9-vr4jt" podUID="a61107c1-4b26-4cb7-85d3-eb83cad66da6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused"
Oct 30 01:11:45.126473 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:11:45.126441    5002 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-g7c2w_5dc56e7e-f3c1-43eb-a21b-8be6bb4706cf/console-operator/1.log"
Oct 30 01:11:45.132250 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:11:45.132214    5002 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-g7c2w_5dc56e7e-f3c1-43eb-a21b-8be6bb4706cf/console-operator/1.log"
Oct 30 01:11:45.440935 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:11:45.440898    5002 patch_prober.go:29] interesting pod/e2e-shared-b-w5-a5e9ab-openshift-default-8498d5b8b9-vr4jt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" start-of-body=
Oct 30 01:11:45.441251 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:11:45.441215    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-a5e9ab-openshift-default-8498d5b8b9-vr4jt" podUID="a61107c1-4b26-4cb7-85d3-eb83cad66da6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused"
Oct 30 01:11:46.440483 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:11:46.440372    5002 patch_prober.go:29] interesting pod/e2e-shared-b-w5-a5e9ab-openshift-default-8498d5b8b9-vr4jt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" start-of-body=
Oct 30 01:11:46.440483 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:11:46.440472    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-a5e9ab-openshift-default-8498d5b8b9-vr4jt" podUID="a61107c1-4b26-4cb7-85d3-eb83cad66da6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused"
Oct 30 01:11:47.441140 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:11:47.441104    5002 patch_prober.go:29] interesting pod/e2e-shared-b-w5-a5e9ab-openshift-default-8498d5b8b9-vr4jt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" start-of-body=
Oct 30 01:11:47.441565 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:11:47.441158    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-a5e9ab-openshift-default-8498d5b8b9-vr4jt" podUID="a61107c1-4b26-4cb7-85d3-eb83cad66da6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused"
Oct 30 01:11:48.120333 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:11:48.120292    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w6-009e78/auth-test-model-009e78-kserve-75bf4b99d5-gxw7n"]
Oct 30 01:11:48.273470 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:11:48.266212    5002 scope.go:122] "RemoveContainer" containerID="8e15f8d3e362db3d9581ee7215dde9279ce0b379e21ffcc218dac3072895be88"
Oct 30 01:11:48.349050 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:11:48.349024    5002 scope.go:122] "RemoveContainer" containerID="822f817f9f3ed6f587314779dde3b475e21ce478e78104238649d77d56da2eb7"
Oct 30 01:11:48.428051 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:11:48.427874    5002 scope.go:122] "RemoveContainer" containerID="91009f3671528fd5344084e5034c38e3d4724d598f75411aae25f069d7bccaea"
Oct 30 01:11:48.440966 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:11:48.440937    5002 patch_prober.go:29] interesting pod/e2e-shared-b-w5-a5e9ab-openshift-default-8498d5b8b9-vr4jt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" start-of-body=
Oct 30 01:11:48.441097 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:11:48.440984    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-a5e9ab-openshift-default-8498d5b8b9-vr4jt" podUID="a61107c1-4b26-4cb7-85d3-eb83cad66da6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused"
Oct 30 01:11:48.441579 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:11:48.441545    5002 scope.go:122] "RemoveContainer" containerID="96a9f524441ea093b606df9abbdc0df63ccfb654ebf95c844dc87f348387d3b9"
Oct 30 01:11:48.498574 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:11:48.495546    5002 scope.go:122] "RemoveContainer" containerID="e92c3c6ceacc638f96136b0ceb8bb52aee9679f5602773934393242c2093a2fe"
Oct 30 01:11:49.441099 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:11:49.441064    5002 patch_prober.go:29] interesting pod/e2e-shared-b-w5-a5e9ab-openshift-default-8498d5b8b9-vr4jt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" start-of-body=
Oct 30 01:11:49.441263 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:11:49.441131    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-a5e9ab-openshift-default-8498d5b8b9-vr4jt" podUID="a61107c1-4b26-4cb7-85d3-eb83cad66da6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused"
Oct 30 01:11:50.440628 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:11:50.440581    5002 patch_prober.go:29] interesting pod/e2e-shared-b-w5-a5e9ab-openshift-default-8498d5b8b9-vr4jt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" start-of-body=
Oct 30 01:11:50.441064 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:11:50.440657    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-a5e9ab-openshift-default-8498d5b8b9-vr4jt" podUID="a61107c1-4b26-4cb7-85d3-eb83cad66da6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused"
Oct 30 01:11:51.451087 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:11:51.451052    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w5-a5e9ab-openshift-default-8498d5b8b9-vr4jt"
Oct 30 01:11:51.454617 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:11:51.454585    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-a5e9ab-openshift-default-8498d5b8b9-vr4jt"
Oct 30 01:11:51.455637 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:11:51.455595    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w5-a5e9ab-openshift-default-8498d5b8b9-vr4jt"
Oct 30 01:12:03.151198 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:12:03.151146    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w6-009e78/auth-test-model-009e78-kserve-75bf4b99d5-gxw7n" podUID="0df627c9-5e19-4e18-b250-a57bda637815" containerName="main" containerID="cri-o://ee11bf850384ae4fe0fa6e6a7ca73d678b6a914fa221b4580fc3a0d7a00aab6c" gracePeriod=45
Oct 30 01:12:03.434459 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:12:03.424955    5002 generic.go:356] "Generic (PLEG): container finished" podID="0df627c9-5e19-4e18-b250-a57bda637815" containerID="ee11bf850384ae4fe0fa6e6a7ca73d678b6a914fa221b4580fc3a0d7a00aab6c" exitCode=0
Oct 30 01:12:03.434459 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:12:03.425144    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w6-009e78/auth-test-model-009e78-kserve-75bf4b99d5-gxw7n" event={"ID":"0df627c9-5e19-4e18-b250-a57bda637815","Type":"ContainerDied","Data":"ee11bf850384ae4fe0fa6e6a7ca73d678b6a914fa221b4580fc3a0d7a00aab6c"}
Oct 30 01:12:03.665998 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:12:03.665931    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w6-009e78/auth-test-model-009e78-kserve-75bf4b99d5-gxw7n" podUID="0df627c9-5e19-4e18-b250-a57bda637815" containerName="main" probeResult="failure" output="Get \"https://10.132.0.190:8000/ready\": dial tcp 10.132.0.190:8000: connect: connection refused"
Oct 30 01:12:04.123447 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:12:04.122510    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf59ea86-db8wc"]
Oct 30 01:12:04.183942 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:12:04.183851    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf59ea86-db8wc"]
Oct 30 01:12:04.222424 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:12:04.221845    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w6-009e78/auth-test-model-009e78-kserve-75bf4b99d5-gxw7n"
Oct 30 01:12:04.317514 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:12:04.316286    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0df627c9-5e19-4e18-b250-a57bda637815-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0df627c9-5e19-4e18-b250-a57bda637815-tmp-dir\") pod \"0df627c9-5e19-4e18-b250-a57bda637815\" (UID: \"0df627c9-5e19-4e18-b250-a57bda637815\") "
Oct 30 01:12:04.317514 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:12:04.316349    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0df627c9-5e19-4e18-b250-a57bda637815-home\" (UniqueName: \"kubernetes.io/empty-dir/0df627c9-5e19-4e18-b250-a57bda637815-home\") pod \"0df627c9-5e19-4e18-b250-a57bda637815\" (UID: \"0df627c9-5e19-4e18-b250-a57bda637815\") "
Oct 30 01:12:04.317514 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:12:04.316452    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0df627c9-5e19-4e18-b250-a57bda637815-tls-certs\" (UniqueName: \"kubernetes.io/secret/0df627c9-5e19-4e18-b250-a57bda637815-tls-certs\") pod \"0df627c9-5e19-4e18-b250-a57bda637815\" (UID: \"0df627c9-5e19-4e18-b250-a57bda637815\") "
Oct 30 01:12:04.317514 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:12:04.316527    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0df627c9-5e19-4e18-b250-a57bda637815-kube-api-access-7lnfs\" (UniqueName: \"kubernetes.io/projected/0df627c9-5e19-4e18-b250-a57bda637815-kube-api-access-7lnfs\") pod \"0df627c9-5e19-4e18-b250-a57bda637815\" (UID: \"0df627c9-5e19-4e18-b250-a57bda637815\") "
Oct 30 01:12:04.317514 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:12:04.316579    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0df627c9-5e19-4e18-b250-a57bda637815-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0df627c9-5e19-4e18-b250-a57bda637815-dshm\") pod \"0df627c9-5e19-4e18-b250-a57bda637815\" (UID: \"0df627c9-5e19-4e18-b250-a57bda637815\") "
Oct 30 01:12:04.317514 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:12:04.316689    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0df627c9-5e19-4e18-b250-a57bda637815-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0df627c9-5e19-4e18-b250-a57bda637815-model-cache\") pod \"0df627c9-5e19-4e18-b250-a57bda637815\" (UID: \"0df627c9-5e19-4e18-b250-a57bda637815\") "
Oct 30 01:12:04.317514 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:12:04.316686    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0df627c9-5e19-4e18-b250-a57bda637815-tmp-dir" pod "0df627c9-5e19-4e18-b250-a57bda637815" (UID: "0df627c9-5e19-4e18-b250-a57bda637815"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:12:04.317514 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:12:04.317327    5002 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0df627c9-5e19-4e18-b250-a57bda637815-tmp-dir\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:12:04.318102 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:12:04.317584    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0df627c9-5e19-4e18-b250-a57bda637815-model-cache" pod "0df627c9-5e19-4e18-b250-a57bda637815" (UID: "0df627c9-5e19-4e18-b250-a57bda637815"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:12:04.318777 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:12:04.318739    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0df627c9-5e19-4e18-b250-a57bda637815-home" pod "0df627c9-5e19-4e18-b250-a57bda637815" (UID: "0df627c9-5e19-4e18-b250-a57bda637815"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:12:04.362405 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:12:04.342541    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0df627c9-5e19-4e18-b250-a57bda637815-dshm" pod "0df627c9-5e19-4e18-b250-a57bda637815" (UID: "0df627c9-5e19-4e18-b250-a57bda637815"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:12:04.371602 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:12:04.369976    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0df627c9-5e19-4e18-b250-a57bda637815-kube-api-access-7lnfs" pod "0df627c9-5e19-4e18-b250-a57bda637815" (UID: "0df627c9-5e19-4e18-b250-a57bda637815"). InnerVolumeSpecName "kube-api-access-7lnfs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:12:04.379578 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:12:04.379532    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0df627c9-5e19-4e18-b250-a57bda637815-tls-certs" pod "0df627c9-5e19-4e18-b250-a57bda637815" (UID: "0df627c9-5e19-4e18-b250-a57bda637815"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:12:04.547539 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:12:04.437735    5002 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0df627c9-5e19-4e18-b250-a57bda637815-model-cache\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:12:04.547539 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:12:04.437772    5002 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0df627c9-5e19-4e18-b250-a57bda637815-home\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:12:04.547539 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:12:04.437786    5002 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0df627c9-5e19-4e18-b250-a57bda637815-tls-certs\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:12:04.547539 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:12:04.437801    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7lnfs\" (UniqueName: \"kubernetes.io/projected/0df627c9-5e19-4e18-b250-a57bda637815-kube-api-access-7lnfs\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:12:04.547539 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:12:04.437823    5002 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0df627c9-5e19-4e18-b250-a57bda637815-dshm\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:12:05.040418 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:12:05.027592    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w6-009e78/auth-test-model-009e78-kserve-75bf4b99d5-gxw7n"
Oct 30 01:12:05.042034 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:12:05.041997    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12301847-7ec2-4d14-87c9-b0a9743b8e2d" path="/var/lib/kubelet/pods/12301847-7ec2-4d14-87c9-b0a9743b8e2d/volumes"
Oct 30 01:12:05.060500 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:12:05.042947    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w6-009e78/auth-test-model-009e78-kserve-75bf4b99d5-gxw7n" event={"ID":"0df627c9-5e19-4e18-b250-a57bda637815","Type":"ContainerDied","Data":"f1f0f28ffc13f35fecd1421acff931ce5602343702948d3a4446dbfd1d2c4c95"}
Oct 30 01:12:05.060500 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:12:05.049470    5002 scope.go:122] "RemoveContainer" containerID="ee11bf850384ae4fe0fa6e6a7ca73d678b6a914fa221b4580fc3a0d7a00aab6c"
Oct 30 01:12:05.118118 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:12:05.118072    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w6-009e78/auth-test-model-009e78-kserve-75bf4b99d5-gxw7n"]
Oct 30 01:12:05.132585 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:12:05.132539    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w6-009e78/auth-test-model-009e78-kserve-75bf4b99d5-gxw7n"]
Oct 30 01:12:06.502174 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:12:06.502133    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0df627c9-5e19-4e18-b250-a57bda637815" path="/var/lib/kubelet/pods/0df627c9-5e19-4e18-b250-a57bda637815/volumes"
Oct 30 01:12:12.864283 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:12:12.864239    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbjdq" containerName="extractor"
Oct 30 01:12:40.031950 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:12:40.031913    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d7720eca-fk52z"]
Oct 30 01:12:40.045682 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:12:40.045652    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d7720eca-fk52z"]
Oct 30 01:12:40.498549 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:12:40.498508    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33b7d06d-4835-41b8-aa1f-aa3b68232ff6" path="/var/lib/kubelet/pods/33b7d06d-4835-41b8-aa1f-aa3b68232ff6/volumes"
Oct 30 01:12:42.865445 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:12:42.865409    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbjdq" containerName="extractor"
Oct 30 01:12:45.034053 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:12:45.034014    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w6-009e78/rate-test-model-009e78-kserve-65478f8b7c-4z5ns"]
Oct 30 01:12:45.034839 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:12:45.034817    5002 cpu_manager.go:414] "containerMap: removing container" podUID="0df627c9-5e19-4e18-b250-a57bda637815" containerName="main"
Oct 30 01:12:45.034910 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:12:45.034841    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0df627c9-5e19-4e18-b250-a57bda637815" containerName="main"
Oct 30 01:12:45.034949 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:12:45.034936    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="0df627c9-5e19-4e18-b250-a57bda637815" containerName="main"
Oct 30 01:12:45.038287 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:12:45.038268    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w6-009e78/rate-test-model-009e78-kserve-65478f8b7c-4z5ns"
Oct 30 01:12:45.064955 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:12:45.061949    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w6-009e78/rate-test-model-009e78-kserve-65478f8b7c-4z5ns"]
Oct 30 01:12:45.067366 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:12:45.067338    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w6-009e78\"/\"default-dockercfg-v6wcl\""
Oct 30 01:12:45.067366 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:12:45.067338    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w6-009e78\"/\"openshift-service-ca.crt\""
Oct 30 01:12:45.067594 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:12:45.067541    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w6-009e78\"/\"kube-root-ca.crt\""
Oct 30 01:12:45.067733 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:12:45.067700    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w6-009e78\"/\"rate-test-model-009e78-kserve-self-signed-certs\""
Oct 30 01:12:45.199038 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:12:45.199003    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gl55x\" (UniqueName: \"kubernetes.io/projected/c8fc10d8-e03d-4a46-bc04-410e1af5ffbf-kube-api-access-gl55x\") pod \"rate-test-model-009e78-kserve-65478f8b7c-4z5ns\" (UID: \"c8fc10d8-e03d-4a46-bc04-410e1af5ffbf\") " pod="ai-tenant-e2e-shared-a-w6-009e78/rate-test-model-009e78-kserve-65478f8b7c-4z5ns"
Oct 30 01:12:45.199225 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:12:45.199070    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8fc10d8-e03d-4a46-bc04-410e1af5ffbf-tls-certs\") pod \"rate-test-model-009e78-kserve-65478f8b7c-4z5ns\" (UID: \"c8fc10d8-e03d-4a46-bc04-410e1af5ffbf\") " pod="ai-tenant-e2e-shared-a-w6-009e78/rate-test-model-009e78-kserve-65478f8b7c-4z5ns"
Oct 30 01:12:45.199225 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:12:45.199173    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c8fc10d8-e03d-4a46-bc04-410e1af5ffbf-tmp-dir\") pod \"rate-test-model-009e78-kserve-65478f8b7c-4z5ns\" (UID: \"c8fc10d8-e03d-4a46-bc04-410e1af5ffbf\") " pod="ai-tenant-e2e-shared-a-w6-009e78/rate-test-model-009e78-kserve-65478f8b7c-4z5ns"
Oct 30 01:12:45.199305 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:12:45.199244    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c8fc10d8-e03d-4a46-bc04-410e1af5ffbf-model-cache\") pod \"rate-test-model-009e78-kserve-65478f8b7c-4z5ns\" (UID: \"c8fc10d8-e03d-4a46-bc04-410e1af5ffbf\") " pod="ai-tenant-e2e-shared-a-w6-009e78/rate-test-model-009e78-kserve-65478f8b7c-4z5ns"
Oct 30 01:12:45.199351 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:12:45.199317    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c8fc10d8-e03d-4a46-bc04-410e1af5ffbf-dshm\") pod \"rate-test-model-009e78-kserve-65478f8b7c-4z5ns\" (UID: \"c8fc10d8-e03d-4a46-bc04-410e1af5ffbf\") " pod="ai-tenant-e2e-shared-a-w6-009e78/rate-test-model-009e78-kserve-65478f8b7c-4z5ns"
Oct 30 01:12:45.199470 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:12:45.199422    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c8fc10d8-e03d-4a46-bc04-410e1af5ffbf-home\") pod \"rate-test-model-009e78-kserve-65478f8b7c-4z5ns\" (UID: \"c8fc10d8-e03d-4a46-bc04-410e1af5ffbf\") " pod="ai-tenant-e2e-shared-a-w6-009e78/rate-test-model-009e78-kserve-65478f8b7c-4z5ns"
Oct 30 01:12:45.304510 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:12:45.301459    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c8fc10d8-e03d-4a46-bc04-410e1af5ffbf-home\") pod \"rate-test-model-009e78-kserve-65478f8b7c-4z5ns\" (UID: \"c8fc10d8-e03d-4a46-bc04-410e1af5ffbf\") " pod="ai-tenant-e2e-shared-a-w6-009e78/rate-test-model-009e78-kserve-65478f8b7c-4z5ns"
Oct 30 01:12:45.304510 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:12:45.301688    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gl55x\" (UniqueName: \"kubernetes.io/projected/c8fc10d8-e03d-4a46-bc04-410e1af5ffbf-kube-api-access-gl55x\") pod \"rate-test-model-009e78-kserve-65478f8b7c-4z5ns\" (UID: \"c8fc10d8-e03d-4a46-bc04-410e1af5ffbf\") " pod="ai-tenant-e2e-shared-a-w6-009e78/rate-test-model-009e78-kserve-65478f8b7c-4z5ns"
Oct 30 01:12:45.304510 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:12:45.301746    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8fc10d8-e03d-4a46-bc04-410e1af5ffbf-tls-certs\") pod \"rate-test-model-009e78-kserve-65478f8b7c-4z5ns\" (UID: \"c8fc10d8-e03d-4a46-bc04-410e1af5ffbf\") " pod="ai-tenant-e2e-shared-a-w6-009e78/rate-test-model-009e78-kserve-65478f8b7c-4z5ns"
Oct 30 01:12:45.304510 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:12:45.301787    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c8fc10d8-e03d-4a46-bc04-410e1af5ffbf-tmp-dir\") pod \"rate-test-model-009e78-kserve-65478f8b7c-4z5ns\" (UID: \"c8fc10d8-e03d-4a46-bc04-410e1af5ffbf\") " pod="ai-tenant-e2e-shared-a-w6-009e78/rate-test-model-009e78-kserve-65478f8b7c-4z5ns"
Oct 30 01:12:45.304510 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:12:45.301842    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c8fc10d8-e03d-4a46-bc04-410e1af5ffbf-model-cache\") pod \"rate-test-model-009e78-kserve-65478f8b7c-4z5ns\" (UID: \"c8fc10d8-e03d-4a46-bc04-410e1af5ffbf\") " pod="ai-tenant-e2e-shared-a-w6-009e78/rate-test-model-009e78-kserve-65478f8b7c-4z5ns"
Oct 30 01:12:45.304510 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:12:45.301915    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c8fc10d8-e03d-4a46-bc04-410e1af5ffbf-dshm\") pod \"rate-test-model-009e78-kserve-65478f8b7c-4z5ns\" (UID: \"c8fc10d8-e03d-4a46-bc04-410e1af5ffbf\") " pod="ai-tenant-e2e-shared-a-w6-009e78/rate-test-model-009e78-kserve-65478f8b7c-4z5ns"
Oct 30 01:12:45.304510 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:12:45.302756    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c8fc10d8-e03d-4a46-bc04-410e1af5ffbf-tmp-dir\") pod \"rate-test-model-009e78-kserve-65478f8b7c-4z5ns\" (UID: \"c8fc10d8-e03d-4a46-bc04-410e1af5ffbf\") " pod="ai-tenant-e2e-shared-a-w6-009e78/rate-test-model-009e78-kserve-65478f8b7c-4z5ns"
Oct 30 01:12:45.304510 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:12:45.303024    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c8fc10d8-e03d-4a46-bc04-410e1af5ffbf-model-cache\") pod \"rate-test-model-009e78-kserve-65478f8b7c-4z5ns\" (UID: \"c8fc10d8-e03d-4a46-bc04-410e1af5ffbf\") " pod="ai-tenant-e2e-shared-a-w6-009e78/rate-test-model-009e78-kserve-65478f8b7c-4z5ns"
Oct 30 01:12:45.304510 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:12:45.303253    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c8fc10d8-e03d-4a46-bc04-410e1af5ffbf-home\") pod \"rate-test-model-009e78-kserve-65478f8b7c-4z5ns\" (UID: \"c8fc10d8-e03d-4a46-bc04-410e1af5ffbf\") " pod="ai-tenant-e2e-shared-a-w6-009e78/rate-test-model-009e78-kserve-65478f8b7c-4z5ns"
Oct 30 01:12:45.343505 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:12:45.331127    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8fc10d8-e03d-4a46-bc04-410e1af5ffbf-tls-certs\") pod \"rate-test-model-009e78-kserve-65478f8b7c-4z5ns\" (UID: \"c8fc10d8-e03d-4a46-bc04-410e1af5ffbf\") " pod="ai-tenant-e2e-shared-a-w6-009e78/rate-test-model-009e78-kserve-65478f8b7c-4z5ns"
Oct 30 01:12:45.343799 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:12:45.342718    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c8fc10d8-e03d-4a46-bc04-410e1af5ffbf-dshm\") pod \"rate-test-model-009e78-kserve-65478f8b7c-4z5ns\" (UID: \"c8fc10d8-e03d-4a46-bc04-410e1af5ffbf\") " pod="ai-tenant-e2e-shared-a-w6-009e78/rate-test-model-009e78-kserve-65478f8b7c-4z5ns"
Oct 30 01:12:45.363198 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:12:45.363132    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gl55x\" (UniqueName: \"kubernetes.io/projected/c8fc10d8-e03d-4a46-bc04-410e1af5ffbf-kube-api-access-gl55x\") pod \"rate-test-model-009e78-kserve-65478f8b7c-4z5ns\" (UID: \"c8fc10d8-e03d-4a46-bc04-410e1af5ffbf\") " pod="ai-tenant-e2e-shared-a-w6-009e78/rate-test-model-009e78-kserve-65478f8b7c-4z5ns"
Oct 30 01:12:45.372938 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:12:45.372663    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w6-009e78/rate-test-model-009e78-kserve-65478f8b7c-4z5ns"
Oct 30 01:12:45.960704 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:12:45.960562    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w6-009e78/rate-test-model-009e78-kserve-65478f8b7c-4z5ns"]
Oct 30 01:12:45.964920 ip-10-0-128-7 kubenswrapper[5002]: W1030 01:12:45.964890    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc8fc10d8_e03d_4a46_bc04_410e1af5ffbf.slice/crio-54568528d138d94f43ca26a4e40a9cf2d07b3d5c80eb8d842371cbd211f3933c WatchSource:0}: Error finding container 54568528d138d94f43ca26a4e40a9cf2d07b3d5c80eb8d842371cbd211f3933c: Status 404 returned error can't find the container with id 54568528d138d94f43ca26a4e40a9cf2d07b3d5c80eb8d842371cbd211f3933c
Oct 30 01:12:46.485847 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:12:46.485806    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w6-009e78/rate-test-model-009e78-kserve-65478f8b7c-4z5ns" event={"ID":"c8fc10d8-e03d-4a46-bc04-410e1af5ffbf","Type":"ContainerStarted","Data":"8a7443383c0442c5c03ca3e166c4ffa57deaaeeee009aa77bd4846118a1bd9bd"}
Oct 30 01:12:46.486353 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:12:46.485861    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w6-009e78/rate-test-model-009e78-kserve-65478f8b7c-4z5ns" event={"ID":"c8fc10d8-e03d-4a46-bc04-410e1af5ffbf","Type":"ContainerStarted","Data":"54568528d138d94f43ca26a4e40a9cf2d07b3d5c80eb8d842371cbd211f3933c"}
Oct 30 01:12:46.533258 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:12:46.533205    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w6-009e78/rate-test-model-009e78-kserve-65478f8b7c-4z5ns" podStartSLOduration=1.533191048 podStartE2EDuration="1.533191048s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:12:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:12:46.529782251 +0000 UTC m=+2462.146215150" watchObservedRunningTime="2026-10-30 01:12:46.533191048 +0000 UTC m=+2462.149623927"
Oct 30 01:12:47.407408 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:12:47.403993    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w6-009e78/rate-test-model-009e78-kserve-65478f8b7c-4z5ns"
Oct 30 01:12:48.569669 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:12:48.569625    5002 scope.go:122] "RemoveContainer" containerID="6bade2f59c598a7ce37e2ef462b40fcac82048901e72be8273836c88efeaabdf"
Oct 30 01:12:49.079004 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:12:49.078974    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w2-f64dc0/auth-test-model-f64dc0-kserve-7d44468c6f-xf5xp"]
Oct 30 01:12:49.085978 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:12:49.085926    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-f64dc0/auth-test-model-f64dc0-kserve-7d44468c6f-xf5xp"
Oct 30 01:12:49.104569 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:12:49.104539    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-f64dc0\"/\"openshift-service-ca.crt\""
Oct 30 01:12:49.104881 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:12:49.104860    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-f64dc0\"/\"kube-root-ca.crt\""
Oct 30 01:12:49.105278 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:12:49.105225    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-f64dc0/auth-test-model-f64dc0-kserve-7d44468c6f-xf5xp"]
Oct 30 01:12:49.113457 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:12:49.113433    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-f64dc0\"/\"default-dockercfg-sxfs9\""
Oct 30 01:12:49.113598 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:12:49.113586    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-f64dc0\"/\"auth-test-model-f64dc0-kserve-self-signed-certs\""
Oct 30 01:12:49.201464 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:12:49.201421    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54606b50-cba4-46d5-8bf4-12843a359919-tls-certs\") pod \"auth-test-model-f64dc0-kserve-7d44468c6f-xf5xp\" (UID: \"54606b50-cba4-46d5-8bf4-12843a359919\") " pod="ai-tenant-e2e-shared-a-w2-f64dc0/auth-test-model-f64dc0-kserve-7d44468c6f-xf5xp"
Oct 30 01:12:49.201650 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:12:49.201504    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54606b50-cba4-46d5-8bf4-12843a359919-tmp-dir\") pod \"auth-test-model-f64dc0-kserve-7d44468c6f-xf5xp\" (UID: \"54606b50-cba4-46d5-8bf4-12843a359919\") " pod="ai-tenant-e2e-shared-a-w2-f64dc0/auth-test-model-f64dc0-kserve-7d44468c6f-xf5xp"
Oct 30 01:12:49.201734 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:12:49.201719    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54606b50-cba4-46d5-8bf4-12843a359919-home\") pod \"auth-test-model-f64dc0-kserve-7d44468c6f-xf5xp\" (UID: \"54606b50-cba4-46d5-8bf4-12843a359919\") " pod="ai-tenant-e2e-shared-a-w2-f64dc0/auth-test-model-f64dc0-kserve-7d44468c6f-xf5xp"
Oct 30 01:12:49.201790 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:12:49.201773    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgzxc\" (UniqueName: \"kubernetes.io/projected/54606b50-cba4-46d5-8bf4-12843a359919-kube-api-access-dgzxc\") pod \"auth-test-model-f64dc0-kserve-7d44468c6f-xf5xp\" (UID: \"54606b50-cba4-46d5-8bf4-12843a359919\") " pod="ai-tenant-e2e-shared-a-w2-f64dc0/auth-test-model-f64dc0-kserve-7d44468c6f-xf5xp"
Oct 30 01:12:49.201863 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:12:49.201847    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54606b50-cba4-46d5-8bf4-12843a359919-dshm\") pod \"auth-test-model-f64dc0-kserve-7d44468c6f-xf5xp\" (UID: \"54606b50-cba4-46d5-8bf4-12843a359919\") " pod="ai-tenant-e2e-shared-a-w2-f64dc0/auth-test-model-f64dc0-kserve-7d44468c6f-xf5xp"
Oct 30 01:12:49.201912 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:12:49.201874    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54606b50-cba4-46d5-8bf4-12843a359919-model-cache\") pod \"auth-test-model-f64dc0-kserve-7d44468c6f-xf5xp\" (UID: \"54606b50-cba4-46d5-8bf4-12843a359919\") " pod="ai-tenant-e2e-shared-a-w2-f64dc0/auth-test-model-f64dc0-kserve-7d44468c6f-xf5xp"
Oct 30 01:12:49.304402 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:12:49.304134    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54606b50-cba4-46d5-8bf4-12843a359919-home\") pod \"auth-test-model-f64dc0-kserve-7d44468c6f-xf5xp\" (UID: \"54606b50-cba4-46d5-8bf4-12843a359919\") " pod="ai-tenant-e2e-shared-a-w2-f64dc0/auth-test-model-f64dc0-kserve-7d44468c6f-xf5xp"
Oct 30 01:12:49.304402 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:12:49.304235    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgzxc\" (UniqueName: \"kubernetes.io/projected/54606b50-cba4-46d5-8bf4-12843a359919-kube-api-access-dgzxc\") pod \"auth-test-model-f64dc0-kserve-7d44468c6f-xf5xp\" (UID: \"54606b50-cba4-46d5-8bf4-12843a359919\") " pod="ai-tenant-e2e-shared-a-w2-f64dc0/auth-test-model-f64dc0-kserve-7d44468c6f-xf5xp"
Oct 30 01:12:49.304402 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:12:49.304291    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54606b50-cba4-46d5-8bf4-12843a359919-dshm\") pod \"auth-test-model-f64dc0-kserve-7d44468c6f-xf5xp\" (UID: \"54606b50-cba4-46d5-8bf4-12843a359919\") " pod="ai-tenant-e2e-shared-a-w2-f64dc0/auth-test-model-f64dc0-kserve-7d44468c6f-xf5xp"
Oct 30 01:12:49.304402 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:12:49.304325    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54606b50-cba4-46d5-8bf4-12843a359919-model-cache\") pod \"auth-test-model-f64dc0-kserve-7d44468c6f-xf5xp\" (UID: \"54606b50-cba4-46d5-8bf4-12843a359919\") " pod="ai-tenant-e2e-shared-a-w2-f64dc0/auth-test-model-f64dc0-kserve-7d44468c6f-xf5xp"
Oct 30 01:12:49.304402 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:12:49.304409    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54606b50-cba4-46d5-8bf4-12843a359919-tls-certs\") pod \"auth-test-model-f64dc0-kserve-7d44468c6f-xf5xp\" (UID: \"54606b50-cba4-46d5-8bf4-12843a359919\") " pod="ai-tenant-e2e-shared-a-w2-f64dc0/auth-test-model-f64dc0-kserve-7d44468c6f-xf5xp"
Oct 30 01:12:49.304802 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:12:49.304461    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54606b50-cba4-46d5-8bf4-12843a359919-tmp-dir\") pod \"auth-test-model-f64dc0-kserve-7d44468c6f-xf5xp\" (UID: \"54606b50-cba4-46d5-8bf4-12843a359919\") " pod="ai-tenant-e2e-shared-a-w2-f64dc0/auth-test-model-f64dc0-kserve-7d44468c6f-xf5xp"
Oct 30 01:12:49.305126 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:12:49.305094    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54606b50-cba4-46d5-8bf4-12843a359919-tmp-dir\") pod \"auth-test-model-f64dc0-kserve-7d44468c6f-xf5xp\" (UID: \"54606b50-cba4-46d5-8bf4-12843a359919\") " pod="ai-tenant-e2e-shared-a-w2-f64dc0/auth-test-model-f64dc0-kserve-7d44468c6f-xf5xp"
Oct 30 01:12:49.305287 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:12:49.305267    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54606b50-cba4-46d5-8bf4-12843a359919-model-cache\") pod \"auth-test-model-f64dc0-kserve-7d44468c6f-xf5xp\" (UID: \"54606b50-cba4-46d5-8bf4-12843a359919\") " pod="ai-tenant-e2e-shared-a-w2-f64dc0/auth-test-model-f64dc0-kserve-7d44468c6f-xf5xp"
Oct 30 01:12:49.309741 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:12:49.309714    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54606b50-cba4-46d5-8bf4-12843a359919-home\") pod \"auth-test-model-f64dc0-kserve-7d44468c6f-xf5xp\" (UID: \"54606b50-cba4-46d5-8bf4-12843a359919\") " pod="ai-tenant-e2e-shared-a-w2-f64dc0/auth-test-model-f64dc0-kserve-7d44468c6f-xf5xp"
Oct 30 01:12:49.403327 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:12:49.403158    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgzxc\" (UniqueName: \"kubernetes.io/projected/54606b50-cba4-46d5-8bf4-12843a359919-kube-api-access-dgzxc\") pod \"auth-test-model-f64dc0-kserve-7d44468c6f-xf5xp\" (UID: \"54606b50-cba4-46d5-8bf4-12843a359919\") " pod="ai-tenant-e2e-shared-a-w2-f64dc0/auth-test-model-f64dc0-kserve-7d44468c6f-xf5xp"
Oct 30 01:12:49.411499 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:12:49.408703    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54606b50-cba4-46d5-8bf4-12843a359919-dshm\") pod \"auth-test-model-f64dc0-kserve-7d44468c6f-xf5xp\" (UID: \"54606b50-cba4-46d5-8bf4-12843a359919\") " pod="ai-tenant-e2e-shared-a-w2-f64dc0/auth-test-model-f64dc0-kserve-7d44468c6f-xf5xp"
Oct 30 01:12:49.411499 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:12:49.409903    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54606b50-cba4-46d5-8bf4-12843a359919-tls-certs\") pod \"auth-test-model-f64dc0-kserve-7d44468c6f-xf5xp\" (UID: \"54606b50-cba4-46d5-8bf4-12843a359919\") " pod="ai-tenant-e2e-shared-a-w2-f64dc0/auth-test-model-f64dc0-kserve-7d44468c6f-xf5xp"
Oct 30 01:12:49.425028 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:12:49.424703    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-f64dc0/auth-test-model-f64dc0-kserve-7d44468c6f-xf5xp"
Oct 30 01:12:49.889451 ip-10-0-128-7 kubenswrapper[5002]: W1030 01:12:49.889414    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod54606b50_cba4_46d5_8bf4_12843a359919.slice/crio-34caac5a9366015785342f3bc2965fd68e5cab8461a9e0742a63b3f04acab8e1 WatchSource:0}: Error finding container 34caac5a9366015785342f3bc2965fd68e5cab8461a9e0742a63b3f04acab8e1: Status 404 returned error can't find the container with id 34caac5a9366015785342f3bc2965fd68e5cab8461a9e0742a63b3f04acab8e1
Oct 30 01:12:49.889987 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:12:49.889959    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-f64dc0/auth-test-model-f64dc0-kserve-7d44468c6f-xf5xp"]
Oct 30 01:12:50.565700 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:12:50.565663    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-f64dc0/auth-test-model-f64dc0-kserve-7d44468c6f-xf5xp" event={"ID":"54606b50-cba4-46d5-8bf4-12843a359919","Type":"ContainerStarted","Data":"1f9cca611b5036823b7d809cfd1aca9bea737a048d756202a7774fbd149453ce"}
Oct 30 01:12:50.565700 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:12:50.565708    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-f64dc0/auth-test-model-f64dc0-kserve-7d44468c6f-xf5xp" event={"ID":"54606b50-cba4-46d5-8bf4-12843a359919","Type":"ContainerStarted","Data":"34caac5a9366015785342f3bc2965fd68e5cab8461a9e0742a63b3f04acab8e1"}
Oct 30 01:12:50.625941 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:12:50.625876    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w2-f64dc0/auth-test-model-f64dc0-kserve-7d44468c6f-xf5xp" podStartSLOduration=1.625861278 podStartE2EDuration="1.625861278s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:12:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:12:50.621453896 +0000 UTC m=+2466.237886798" watchObservedRunningTime="2026-10-30 01:12:50.625861278 +0000 UTC m=+2466.242294357"
Oct 30 01:12:50.642738 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:12:50.642706    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7d8z4"]
Oct 30 01:12:51.428965 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:12:51.428925    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w2-f64dc0/auth-test-model-f64dc0-kserve-7d44468c6f-xf5xp"
Oct 30 01:12:55.373791 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:12:55.373749    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w6-009e78/rate-test-model-009e78-kserve-65478f8b7c-4z5ns"
Oct 30 01:12:55.395761 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:12:55.395729    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w6-009e78/rate-test-model-009e78-kserve-65478f8b7c-4z5ns"
Oct 30 01:12:55.699121 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:12:55.699087    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w6-009e78/rate-test-model-009e78-kserve-65478f8b7c-4z5ns"
Oct 30 01:12:56.188465 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:12:56.188432    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e31412e-t2bcw"]
Oct 30 01:12:56.193235 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:12:56.193215    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e31412e-t2bcw"
Oct 30 01:12:56.203800 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:12:56.203775    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25f70018-d36a-4d1a-933a-1b7db5d8614d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9e31412e-t2bcw\" (UID: \"25f70018-d36a-4d1a-933a-1b7db5d8614d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e31412e-t2bcw"
Oct 30 01:12:56.203932 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:12:56.203836    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25f70018-d36a-4d1a-933a-1b7db5d8614d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9e31412e-t2bcw\" (UID: \"25f70018-d36a-4d1a-933a-1b7db5d8614d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e31412e-t2bcw"
Oct 30 01:12:56.204044 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:12:56.204018    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xp95f\" (UniqueName: \"kubernetes.io/projected/25f70018-d36a-4d1a-933a-1b7db5d8614d-kube-api-access-xp95f\") pod \"maas-api-revoke-sub-9e31412e-t2bcw\" (UID: \"25f70018-d36a-4d1a-933a-1b7db5d8614d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e31412e-t2bcw"
Oct 30 01:12:56.212243 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:12:56.212204    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:12:56.215098 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:12:56.215072    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e31412e-t2bcw"]
Oct 30 01:12:56.222514 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:12:56.222491    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:12:56.222644 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:12:56.222542    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:12:56.222644 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:12:56.222597    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5phsh\""
Oct 30 01:12:56.304995 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:12:56.304964    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xp95f\" (UniqueName: \"kubernetes.io/projected/25f70018-d36a-4d1a-933a-1b7db5d8614d-kube-api-access-xp95f\") pod \"maas-api-revoke-sub-9e31412e-t2bcw\" (UID: \"25f70018-d36a-4d1a-933a-1b7db5d8614d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e31412e-t2bcw"
Oct 30 01:12:56.305180 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:12:56.305164    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25f70018-d36a-4d1a-933a-1b7db5d8614d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9e31412e-t2bcw\" (UID: \"25f70018-d36a-4d1a-933a-1b7db5d8614d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e31412e-t2bcw"
Oct 30 01:12:56.305289 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:12:56.305207    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25f70018-d36a-4d1a-933a-1b7db5d8614d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9e31412e-t2bcw\" (UID: \"25f70018-d36a-4d1a-933a-1b7db5d8614d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e31412e-t2bcw"
Oct 30 01:12:56.306030 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:12:56.306006    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25f70018-d36a-4d1a-933a-1b7db5d8614d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9e31412e-t2bcw\" (UID: \"25f70018-d36a-4d1a-933a-1b7db5d8614d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e31412e-t2bcw"
Oct 30 01:12:56.308340 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:12:56.308312    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25f70018-d36a-4d1a-933a-1b7db5d8614d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9e31412e-t2bcw\" (UID: \"25f70018-d36a-4d1a-933a-1b7db5d8614d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e31412e-t2bcw"
Oct 30 01:12:56.330196 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:12:56.330171    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xp95f\" (UniqueName: \"kubernetes.io/projected/25f70018-d36a-4d1a-933a-1b7db5d8614d-kube-api-access-xp95f\") pod \"maas-api-revoke-sub-9e31412e-t2bcw\" (UID: \"25f70018-d36a-4d1a-933a-1b7db5d8614d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e31412e-t2bcw"
Oct 30 01:12:56.508438 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:12:56.508321    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e31412e-t2bcw"
Oct 30 01:12:56.847628 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:12:56.847588    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e31412e-t2bcw"]
Oct 30 01:12:57.699834 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:12:57.699788    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e31412e-t2bcw" event={"ID":"25f70018-d36a-4d1a-933a-1b7db5d8614d","Type":"ContainerStarted","Data":"fdbfd27f271bb4bb7f162253facf1a62f4aa110d21f1edef991f48d517e602e1"}
Oct 30 01:12:57.700282 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:12:57.699892    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e31412e-t2bcw" event={"ID":"25f70018-d36a-4d1a-933a-1b7db5d8614d","Type":"ContainerStarted","Data":"84ff841f31c1c70ebea685373d2d25c95b3bf9134289a0699c29b88e88f340d8"}
Oct 30 01:12:57.748175 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:12:57.748108    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e31412e-t2bcw" podStartSLOduration=1.748089313 podStartE2EDuration="1.748089313s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:12:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:12:57.74266786 +0000 UTC m=+2473.359100737" watchObservedRunningTime="2026-10-30 01:12:57.748089313 +0000 UTC m=+2473.364522195"
Oct 30 01:12:58.710037 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:12:58.709998    5002 generic.go:356] "Generic (PLEG): container finished" podID="25f70018-d36a-4d1a-933a-1b7db5d8614d" containerID="fdbfd27f271bb4bb7f162253facf1a62f4aa110d21f1edef991f48d517e602e1" exitCode=0
Oct 30 01:12:58.710489 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:12:58.710079    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e31412e-t2bcw" event={"ID":"25f70018-d36a-4d1a-933a-1b7db5d8614d","Type":"ContainerDied","Data":"fdbfd27f271bb4bb7f162253facf1a62f4aa110d21f1edef991f48d517e602e1"}
Oct 30 01:12:59.428617 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:12:59.428559    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w2-f64dc0/auth-test-model-f64dc0-kserve-7d44468c6f-xf5xp"
Oct 30 01:12:59.441980 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:12:59.441937    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w2-f64dc0/auth-test-model-f64dc0-kserve-7d44468c6f-xf5xp"
Oct 30 01:12:59.733824 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:12:59.733789    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w2-f64dc0/auth-test-model-f64dc0-kserve-7d44468c6f-xf5xp"
Oct 30 01:12:59.973521 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:12:59.973449    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e31412e-t2bcw"
Oct 30 01:13:00.074929 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:00.074887    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25f70018-d36a-4d1a-933a-1b7db5d8614d-kube-api-access-xp95f\" (UniqueName: \"kubernetes.io/projected/25f70018-d36a-4d1a-933a-1b7db5d8614d-kube-api-access-xp95f\") pod \"25f70018-d36a-4d1a-933a-1b7db5d8614d\" (UID: \"25f70018-d36a-4d1a-933a-1b7db5d8614d\") "
Oct 30 01:13:00.075183 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:00.074998    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/25f70018-d36a-4d1a-933a-1b7db5d8614d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25f70018-d36a-4d1a-933a-1b7db5d8614d-maas-api-xks-ca\") pod \"25f70018-d36a-4d1a-933a-1b7db5d8614d\" (UID: \"25f70018-d36a-4d1a-933a-1b7db5d8614d\") "
Oct 30 01:13:00.075272 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:00.075182    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25f70018-d36a-4d1a-933a-1b7db5d8614d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25f70018-d36a-4d1a-933a-1b7db5d8614d-maas-api-openshift-ca\") pod \"25f70018-d36a-4d1a-933a-1b7db5d8614d\" (UID: \"25f70018-d36a-4d1a-933a-1b7db5d8614d\") "
Oct 30 01:13:00.079083 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:00.077741    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25f70018-d36a-4d1a-933a-1b7db5d8614d-maas-api-openshift-ca" pod "25f70018-d36a-4d1a-933a-1b7db5d8614d" (UID: "25f70018-d36a-4d1a-933a-1b7db5d8614d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:13:00.082527 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:00.080981    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25f70018-d36a-4d1a-933a-1b7db5d8614d-maas-api-xks-ca" pod "25f70018-d36a-4d1a-933a-1b7db5d8614d" (UID: "25f70018-d36a-4d1a-933a-1b7db5d8614d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:13:00.082527 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:00.081463    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25f70018-d36a-4d1a-933a-1b7db5d8614d-kube-api-access-xp95f" pod "25f70018-d36a-4d1a-933a-1b7db5d8614d" (UID: "25f70018-d36a-4d1a-933a-1b7db5d8614d"). InnerVolumeSpecName "kube-api-access-xp95f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:13:00.177091 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:00.177051    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25f70018-d36a-4d1a-933a-1b7db5d8614d-maas-api-xks-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:13:00.177091 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:00.177092    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25f70018-d36a-4d1a-933a-1b7db5d8614d-maas-api-openshift-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:13:00.177334 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:00.177107    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xp95f\" (UniqueName: \"kubernetes.io/projected/25f70018-d36a-4d1a-933a-1b7db5d8614d-kube-api-access-xp95f\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:13:00.730920 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:00.730885    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e31412e-t2bcw"
Oct 30 01:13:00.730920 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:00.730902    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e31412e-t2bcw" event={"ID":"25f70018-d36a-4d1a-933a-1b7db5d8614d","Type":"ContainerDied","Data":"84ff841f31c1c70ebea685373d2d25c95b3bf9134289a0699c29b88e88f340d8"}
Oct 30 01:13:00.731126 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:00.730953    5002 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="84ff841f31c1c70ebea685373d2d25c95b3bf9134289a0699c29b88e88f340d8"
Oct 30 01:13:02.784430 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:02.784058    5002 generic.go:356] "Generic (PLEG): container finished" podID="f3f9e906-5594-4cdf-a620-efc9cbc28a44" containerID="670c9acd81e92e790db801790362f50310923577a102c2934548080f2d5fd45e" exitCode=0
Oct 30 01:13:02.784430 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:02.784124    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-6d44c2-openshift-default-69fb7659f-snvvd" event={"ID":"f3f9e906-5594-4cdf-a620-efc9cbc28a44","Type":"ContainerDied","Data":"670c9acd81e92e790db801790362f50310923577a102c2934548080f2d5fd45e"}
Oct 30 01:13:02.785025 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:02.784680    5002 scope.go:122] "RemoveContainer" containerID="670c9acd81e92e790db801790362f50310923577a102c2934548080f2d5fd45e"
Oct 30 01:13:02.787145 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:02.787103    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:13:02.787286 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:02.787201    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:13:02.787286 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:02.787250    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:13:02.833125 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:02.833084    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-6d44c2-openshift-default-69fb7659f-snvvd"
Oct 30 01:13:03.794626 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:03.794586    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-6d44c2-openshift-default-69fb7659f-snvvd" event={"ID":"f3f9e906-5594-4cdf-a620-efc9cbc28a44","Type":"ContainerStarted","Data":"61d2d21bb52878d7b3ccbd7517bca54723c0ec9066044bf1b77313d20d46cd63"}
Oct 30 01:13:04.133778 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:04.133688    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-6d44c2-openshift-default-69fb7659f-snvvd"
Oct 30 01:13:04.135151 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:04.135120    5002 patch_prober.go:29] interesting pod/e2e-ipp-b-6d44c2-openshift-default-69fb7659f-snvvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" start-of-body=
Oct 30 01:13:04.135293 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:04.135183    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-6d44c2-openshift-default-69fb7659f-snvvd" podUID="f3f9e906-5594-4cdf-a620-efc9cbc28a44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused"
Oct 30 01:13:05.133985 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:05.133947    5002 patch_prober.go:29] interesting pod/e2e-ipp-b-6d44c2-openshift-default-69fb7659f-snvvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" start-of-body=
Oct 30 01:13:05.134444 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:05.134015    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-6d44c2-openshift-default-69fb7659f-snvvd" podUID="f3f9e906-5594-4cdf-a620-efc9cbc28a44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused"
Oct 30 01:13:06.134011 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:06.133974    5002 patch_prober.go:29] interesting pod/e2e-ipp-b-6d44c2-openshift-default-69fb7659f-snvvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" start-of-body=
Oct 30 01:13:06.134482 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:06.134031    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-6d44c2-openshift-default-69fb7659f-snvvd" podUID="f3f9e906-5594-4cdf-a620-efc9cbc28a44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused"
Oct 30 01:13:06.538372 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:06.538332    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e7e87d1-db4wf"]
Oct 30 01:13:06.539358 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:06.539335    5002 cpu_manager.go:414] "containerMap: removing container" podUID="25f70018-d36a-4d1a-933a-1b7db5d8614d" containerName="revoke-keys"
Oct 30 01:13:06.539458 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:06.539362    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25f70018-d36a-4d1a-933a-1b7db5d8614d" containerName="revoke-keys"
Oct 30 01:13:06.539622 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:06.539606    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="25f70018-d36a-4d1a-933a-1b7db5d8614d" containerName="revoke-keys"
Oct 30 01:13:06.543389 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:06.543358    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e7e87d1-db4wf"
Oct 30 01:13:06.565726 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:06.565690    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:13:06.603748 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:06.567992    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e7e87d1-db4wf"]
Oct 30 01:13:06.603748 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:06.581715    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5phsh\""
Oct 30 01:13:06.603748 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:06.581778    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:13:06.603748 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:06.581847    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:13:06.716273 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:06.716229    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d1761e3-ec85-4fbe-9b17-323cd7b220d3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9e7e87d1-db4wf\" (UID: \"6d1761e3-ec85-4fbe-9b17-323cd7b220d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e7e87d1-db4wf"
Oct 30 01:13:06.716525 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:06.716356    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vg266\" (UniqueName: \"kubernetes.io/projected/6d1761e3-ec85-4fbe-9b17-323cd7b220d3-kube-api-access-vg266\") pod \"maas-api-revoke-sub-9e7e87d1-db4wf\" (UID: \"6d1761e3-ec85-4fbe-9b17-323cd7b220d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e7e87d1-db4wf"
Oct 30 01:13:06.716766 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:06.716737    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d1761e3-ec85-4fbe-9b17-323cd7b220d3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9e7e87d1-db4wf\" (UID: \"6d1761e3-ec85-4fbe-9b17-323cd7b220d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e7e87d1-db4wf"
Oct 30 01:13:06.818209 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:06.818117    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d1761e3-ec85-4fbe-9b17-323cd7b220d3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9e7e87d1-db4wf\" (UID: \"6d1761e3-ec85-4fbe-9b17-323cd7b220d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e7e87d1-db4wf"
Oct 30 01:13:06.818369 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:06.818210    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d1761e3-ec85-4fbe-9b17-323cd7b220d3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9e7e87d1-db4wf\" (UID: \"6d1761e3-ec85-4fbe-9b17-323cd7b220d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e7e87d1-db4wf"
Oct 30 01:13:06.818369 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:06.818325    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vg266\" (UniqueName: \"kubernetes.io/projected/6d1761e3-ec85-4fbe-9b17-323cd7b220d3-kube-api-access-vg266\") pod \"maas-api-revoke-sub-9e7e87d1-db4wf\" (UID: \"6d1761e3-ec85-4fbe-9b17-323cd7b220d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e7e87d1-db4wf"
Oct 30 01:13:06.819290 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:06.819257    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d1761e3-ec85-4fbe-9b17-323cd7b220d3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9e7e87d1-db4wf\" (UID: \"6d1761e3-ec85-4fbe-9b17-323cd7b220d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e7e87d1-db4wf"
Oct 30 01:13:06.820937 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:06.820893    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d1761e3-ec85-4fbe-9b17-323cd7b220d3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9e7e87d1-db4wf\" (UID: \"6d1761e3-ec85-4fbe-9b17-323cd7b220d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e7e87d1-db4wf"
Oct 30 01:13:06.843667 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:06.843639    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vg266\" (UniqueName: \"kubernetes.io/projected/6d1761e3-ec85-4fbe-9b17-323cd7b220d3-kube-api-access-vg266\") pod \"maas-api-revoke-sub-9e7e87d1-db4wf\" (UID: \"6d1761e3-ec85-4fbe-9b17-323cd7b220d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e7e87d1-db4wf"
Oct 30 01:13:06.908153 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:06.908114    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e7e87d1-db4wf"
Oct 30 01:13:07.134755 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:07.134100    5002 patch_prober.go:29] interesting pod/e2e-ipp-b-6d44c2-openshift-default-69fb7659f-snvvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" start-of-body=
Oct 30 01:13:07.134755 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:07.134164    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-6d44c2-openshift-default-69fb7659f-snvvd" podUID="f3f9e906-5594-4cdf-a620-efc9cbc28a44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused"
Oct 30 01:13:07.291666 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:07.291433    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e7e87d1-db4wf"]
Oct 30 01:13:07.833251 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:07.833160    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e7e87d1-db4wf" event={"ID":"6d1761e3-ec85-4fbe-9b17-323cd7b220d3","Type":"ContainerStarted","Data":"9b58c57b966194ffbd9e312be31029a8102154ea337b61d8a2935a137b09b0cb"}
Oct 30 01:13:07.833251 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:07.833213    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e7e87d1-db4wf" event={"ID":"6d1761e3-ec85-4fbe-9b17-323cd7b220d3","Type":"ContainerStarted","Data":"a9dfde67de3ad9088fe18f58135ca1f28b5f969ffae8bf6e610e0b7d91607dde"}
Oct 30 01:13:07.893401 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:07.892870    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e7e87d1-db4wf" podStartSLOduration=1.892850369 podStartE2EDuration="1.892850369s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:13:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:13:07.89206347 +0000 UTC m=+2483.508496355" watchObservedRunningTime="2026-10-30 01:13:07.892850369 +0000 UTC m=+2483.509283249"
Oct 30 01:13:08.134563 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:08.134468    5002 patch_prober.go:29] interesting pod/e2e-ipp-b-6d44c2-openshift-default-69fb7659f-snvvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" start-of-body=
Oct 30 01:13:08.134563 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:08.134524    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-6d44c2-openshift-default-69fb7659f-snvvd" podUID="f3f9e906-5594-4cdf-a620-efc9cbc28a44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused"
Oct 30 01:13:08.842156 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:08.842059    5002 generic.go:356] "Generic (PLEG): container finished" podID="6d1761e3-ec85-4fbe-9b17-323cd7b220d3" containerID="9b58c57b966194ffbd9e312be31029a8102154ea337b61d8a2935a137b09b0cb" exitCode=0
Oct 30 01:13:08.842156 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:08.842141    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e7e87d1-db4wf" event={"ID":"6d1761e3-ec85-4fbe-9b17-323cd7b220d3","Type":"ContainerDied","Data":"9b58c57b966194ffbd9e312be31029a8102154ea337b61d8a2935a137b09b0cb"}
Oct 30 01:13:09.140168 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:09.140082    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-6d44c2-openshift-default-69fb7659f-snvvd"
Oct 30 01:13:09.140676 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:09.140654    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-6d44c2-openshift-default-69fb7659f-snvvd"
Oct 30 01:13:09.141510 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:09.141488    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-6d44c2-openshift-default-69fb7659f-snvvd"
Oct 30 01:13:10.044952 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:10.044920    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e7e87d1-db4wf"
Oct 30 01:13:10.116296 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:10.116202    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6d1761e3-ec85-4fbe-9b17-323cd7b220d3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d1761e3-ec85-4fbe-9b17-323cd7b220d3-maas-api-xks-ca\") pod \"6d1761e3-ec85-4fbe-9b17-323cd7b220d3\" (UID: \"6d1761e3-ec85-4fbe-9b17-323cd7b220d3\") "
Oct 30 01:13:10.116521 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:10.116335    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d1761e3-ec85-4fbe-9b17-323cd7b220d3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d1761e3-ec85-4fbe-9b17-323cd7b220d3-maas-api-openshift-ca\") pod \"6d1761e3-ec85-4fbe-9b17-323cd7b220d3\" (UID: \"6d1761e3-ec85-4fbe-9b17-323cd7b220d3\") "
Oct 30 01:13:10.116521 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:10.116401    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d1761e3-ec85-4fbe-9b17-323cd7b220d3-kube-api-access-vg266\" (UniqueName: \"kubernetes.io/projected/6d1761e3-ec85-4fbe-9b17-323cd7b220d3-kube-api-access-vg266\") pod \"6d1761e3-ec85-4fbe-9b17-323cd7b220d3\" (UID: \"6d1761e3-ec85-4fbe-9b17-323cd7b220d3\") "
Oct 30 01:13:10.117077 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:10.117049    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d1761e3-ec85-4fbe-9b17-323cd7b220d3-maas-api-openshift-ca" pod "6d1761e3-ec85-4fbe-9b17-323cd7b220d3" (UID: "6d1761e3-ec85-4fbe-9b17-323cd7b220d3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:13:10.117427 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:10.117405    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d1761e3-ec85-4fbe-9b17-323cd7b220d3-maas-api-openshift-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:13:10.118976 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:10.118945    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d1761e3-ec85-4fbe-9b17-323cd7b220d3-kube-api-access-vg266" pod "6d1761e3-ec85-4fbe-9b17-323cd7b220d3" (UID: "6d1761e3-ec85-4fbe-9b17-323cd7b220d3"). InnerVolumeSpecName "kube-api-access-vg266". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:13:10.119112 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:10.119098    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d1761e3-ec85-4fbe-9b17-323cd7b220d3-maas-api-xks-ca" pod "6d1761e3-ec85-4fbe-9b17-323cd7b220d3" (UID: "6d1761e3-ec85-4fbe-9b17-323cd7b220d3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:13:10.219471 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:10.219433    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vg266\" (UniqueName: \"kubernetes.io/projected/6d1761e3-ec85-4fbe-9b17-323cd7b220d3-kube-api-access-vg266\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:13:10.219471 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:10.219464    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d1761e3-ec85-4fbe-9b17-323cd7b220d3-maas-api-xks-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:13:10.861144 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:10.861050    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e7e87d1-db4wf"
Oct 30 01:13:10.861144 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:10.861082    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e7e87d1-db4wf" event={"ID":"6d1761e3-ec85-4fbe-9b17-323cd7b220d3","Type":"ContainerDied","Data":"a9dfde67de3ad9088fe18f58135ca1f28b5f969ffae8bf6e610e0b7d91607dde"}
Oct 30 01:13:10.861400 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:10.861163    5002 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a9dfde67de3ad9088fe18f58135ca1f28b5f969ffae8bf6e610e0b7d91607dde"
Oct 30 01:13:12.867364 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:13:12.867316    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbjdq" containerName="extractor"
Oct 30 01:13:22.030962 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:22.030925    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f51a915-9g29d"]
Oct 30 01:13:22.044227 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:22.044189    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f51a915-9g29d"]
Oct 30 01:13:22.499700 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:22.499656    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ecc17fe-4d17-452c-9856-5ac462ffde03" path="/var/lib/kubelet/pods/1ecc17fe-4d17-452c-9856-5ac462ffde03/volumes"
Oct 30 01:13:29.670037 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:29.669969    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-01215f-openshift-default-95bfd666f-rkm78"]
Oct 30 01:13:29.671526 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:29.671497    5002 cpu_manager.go:414] "containerMap: removing container" podUID="6d1761e3-ec85-4fbe-9b17-323cd7b220d3" containerName="revoke-keys"
Oct 30 01:13:29.671675 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:29.671661    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d1761e3-ec85-4fbe-9b17-323cd7b220d3" containerName="revoke-keys"
Oct 30 01:13:29.671995 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:29.671981    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d1761e3-ec85-4fbe-9b17-323cd7b220d3" containerName="revoke-keys"
Oct 30 01:13:29.677454 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:29.677432    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-01215f-openshift-default-95bfd666f-rkm78"
Oct 30 01:13:29.715533 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:29.715495    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-01215f-openshift-default-95bfd666f-rkm78"]
Oct 30 01:13:29.724047 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:29.724013    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-01215f-openshift-default-dockercfg-d4787\""
Oct 30 01:13:29.780373 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:29.780313    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c140dcb-40bd-4cd2-ac16-df81ed774c16-istio-token\") pod \"e2e-ipp-a-01215f-openshift-default-95bfd666f-rkm78\" (UID: \"4c140dcb-40bd-4cd2-ac16-df81ed774c16\") " pod="openshift-ingress/e2e-ipp-a-01215f-openshift-default-95bfd666f-rkm78"
Oct 30 01:13:29.781169 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:29.781137    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c140dcb-40bd-4cd2-ac16-df81ed774c16-istio-podinfo\") pod \"e2e-ipp-a-01215f-openshift-default-95bfd666f-rkm78\" (UID: \"4c140dcb-40bd-4cd2-ac16-df81ed774c16\") " pod="openshift-ingress/e2e-ipp-a-01215f-openshift-default-95bfd666f-rkm78"
Oct 30 01:13:29.781317 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:29.781234    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txhw8\" (UniqueName: \"kubernetes.io/projected/4c140dcb-40bd-4cd2-ac16-df81ed774c16-kube-api-access-txhw8\") pod \"e2e-ipp-a-01215f-openshift-default-95bfd666f-rkm78\" (UID: \"4c140dcb-40bd-4cd2-ac16-df81ed774c16\") " pod="openshift-ingress/e2e-ipp-a-01215f-openshift-default-95bfd666f-rkm78"
Oct 30 01:13:29.781450 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:29.781336    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c140dcb-40bd-4cd2-ac16-df81ed774c16-credential-socket\") pod \"e2e-ipp-a-01215f-openshift-default-95bfd666f-rkm78\" (UID: \"4c140dcb-40bd-4cd2-ac16-df81ed774c16\") " pod="openshift-ingress/e2e-ipp-a-01215f-openshift-default-95bfd666f-rkm78"
Oct 30 01:13:29.781450 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:29.781364    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c140dcb-40bd-4cd2-ac16-df81ed774c16-istio-data\") pod \"e2e-ipp-a-01215f-openshift-default-95bfd666f-rkm78\" (UID: \"4c140dcb-40bd-4cd2-ac16-df81ed774c16\") " pod="openshift-ingress/e2e-ipp-a-01215f-openshift-default-95bfd666f-rkm78"
Oct 30 01:13:29.781603 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:29.781550    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c140dcb-40bd-4cd2-ac16-df81ed774c16-istio-envoy\") pod \"e2e-ipp-a-01215f-openshift-default-95bfd666f-rkm78\" (UID: \"4c140dcb-40bd-4cd2-ac16-df81ed774c16\") " pod="openshift-ingress/e2e-ipp-a-01215f-openshift-default-95bfd666f-rkm78"
Oct 30 01:13:29.781669 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:29.781630    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c140dcb-40bd-4cd2-ac16-df81ed774c16-workload-certs\") pod \"e2e-ipp-a-01215f-openshift-default-95bfd666f-rkm78\" (UID: \"4c140dcb-40bd-4cd2-ac16-df81ed774c16\") " pod="openshift-ingress/e2e-ipp-a-01215f-openshift-default-95bfd666f-rkm78"
Oct 30 01:13:29.781772 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:29.781749    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c140dcb-40bd-4cd2-ac16-df81ed774c16-istiod-ca-cert\") pod \"e2e-ipp-a-01215f-openshift-default-95bfd666f-rkm78\" (UID: \"4c140dcb-40bd-4cd2-ac16-df81ed774c16\") " pod="openshift-ingress/e2e-ipp-a-01215f-openshift-default-95bfd666f-rkm78"
Oct 30 01:13:29.781906 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:29.781888    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c140dcb-40bd-4cd2-ac16-df81ed774c16-workload-socket\") pod \"e2e-ipp-a-01215f-openshift-default-95bfd666f-rkm78\" (UID: \"4c140dcb-40bd-4cd2-ac16-df81ed774c16\") " pod="openshift-ingress/e2e-ipp-a-01215f-openshift-default-95bfd666f-rkm78"
Oct 30 01:13:29.896457 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:29.886574    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c140dcb-40bd-4cd2-ac16-df81ed774c16-workload-socket\") pod \"e2e-ipp-a-01215f-openshift-default-95bfd666f-rkm78\" (UID: \"4c140dcb-40bd-4cd2-ac16-df81ed774c16\") " pod="openshift-ingress/e2e-ipp-a-01215f-openshift-default-95bfd666f-rkm78"
Oct 30 01:13:29.896457 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:29.886646    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c140dcb-40bd-4cd2-ac16-df81ed774c16-istio-token\") pod \"e2e-ipp-a-01215f-openshift-default-95bfd666f-rkm78\" (UID: \"4c140dcb-40bd-4cd2-ac16-df81ed774c16\") " pod="openshift-ingress/e2e-ipp-a-01215f-openshift-default-95bfd666f-rkm78"
Oct 30 01:13:29.896457 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:29.886727    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c140dcb-40bd-4cd2-ac16-df81ed774c16-istio-podinfo\") pod \"e2e-ipp-a-01215f-openshift-default-95bfd666f-rkm78\" (UID: \"4c140dcb-40bd-4cd2-ac16-df81ed774c16\") " pod="openshift-ingress/e2e-ipp-a-01215f-openshift-default-95bfd666f-rkm78"
Oct 30 01:13:29.896457 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:29.886846    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txhw8\" (UniqueName: \"kubernetes.io/projected/4c140dcb-40bd-4cd2-ac16-df81ed774c16-kube-api-access-txhw8\") pod \"e2e-ipp-a-01215f-openshift-default-95bfd666f-rkm78\" (UID: \"4c140dcb-40bd-4cd2-ac16-df81ed774c16\") " pod="openshift-ingress/e2e-ipp-a-01215f-openshift-default-95bfd666f-rkm78"
Oct 30 01:13:29.896457 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:29.886901    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c140dcb-40bd-4cd2-ac16-df81ed774c16-credential-socket\") pod \"e2e-ipp-a-01215f-openshift-default-95bfd666f-rkm78\" (UID: \"4c140dcb-40bd-4cd2-ac16-df81ed774c16\") " pod="openshift-ingress/e2e-ipp-a-01215f-openshift-default-95bfd666f-rkm78"
Oct 30 01:13:29.896457 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:29.886938    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c140dcb-40bd-4cd2-ac16-df81ed774c16-istio-data\") pod \"e2e-ipp-a-01215f-openshift-default-95bfd666f-rkm78\" (UID: \"4c140dcb-40bd-4cd2-ac16-df81ed774c16\") " pod="openshift-ingress/e2e-ipp-a-01215f-openshift-default-95bfd666f-rkm78"
Oct 30 01:13:29.896457 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:29.887037    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c140dcb-40bd-4cd2-ac16-df81ed774c16-istio-envoy\") pod \"e2e-ipp-a-01215f-openshift-default-95bfd666f-rkm78\" (UID: \"4c140dcb-40bd-4cd2-ac16-df81ed774c16\") " pod="openshift-ingress/e2e-ipp-a-01215f-openshift-default-95bfd666f-rkm78"
Oct 30 01:13:29.896457 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:29.887086    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c140dcb-40bd-4cd2-ac16-df81ed774c16-workload-certs\") pod \"e2e-ipp-a-01215f-openshift-default-95bfd666f-rkm78\" (UID: \"4c140dcb-40bd-4cd2-ac16-df81ed774c16\") " pod="openshift-ingress/e2e-ipp-a-01215f-openshift-default-95bfd666f-rkm78"
Oct 30 01:13:29.896457 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:29.887165    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c140dcb-40bd-4cd2-ac16-df81ed774c16-istiod-ca-cert\") pod \"e2e-ipp-a-01215f-openshift-default-95bfd666f-rkm78\" (UID: \"4c140dcb-40bd-4cd2-ac16-df81ed774c16\") " pod="openshift-ingress/e2e-ipp-a-01215f-openshift-default-95bfd666f-rkm78"
Oct 30 01:13:29.896457 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:29.888199    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c140dcb-40bd-4cd2-ac16-df81ed774c16-istiod-ca-cert\") pod \"e2e-ipp-a-01215f-openshift-default-95bfd666f-rkm78\" (UID: \"4c140dcb-40bd-4cd2-ac16-df81ed774c16\") " pod="openshift-ingress/e2e-ipp-a-01215f-openshift-default-95bfd666f-rkm78"
Oct 30 01:13:29.896457 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:29.888532    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c140dcb-40bd-4cd2-ac16-df81ed774c16-workload-socket\") pod \"e2e-ipp-a-01215f-openshift-default-95bfd666f-rkm78\" (UID: \"4c140dcb-40bd-4cd2-ac16-df81ed774c16\") " pod="openshift-ingress/e2e-ipp-a-01215f-openshift-default-95bfd666f-rkm78"
Oct 30 01:13:29.896457 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:29.890646    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c140dcb-40bd-4cd2-ac16-df81ed774c16-istio-data\") pod \"e2e-ipp-a-01215f-openshift-default-95bfd666f-rkm78\" (UID: \"4c140dcb-40bd-4cd2-ac16-df81ed774c16\") " pod="openshift-ingress/e2e-ipp-a-01215f-openshift-default-95bfd666f-rkm78"
Oct 30 01:13:29.896457 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:29.891197    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c140dcb-40bd-4cd2-ac16-df81ed774c16-credential-socket\") pod \"e2e-ipp-a-01215f-openshift-default-95bfd666f-rkm78\" (UID: \"4c140dcb-40bd-4cd2-ac16-df81ed774c16\") " pod="openshift-ingress/e2e-ipp-a-01215f-openshift-default-95bfd666f-rkm78"
Oct 30 01:13:29.923355 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:29.901465    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c140dcb-40bd-4cd2-ac16-df81ed774c16-istio-podinfo\") pod \"e2e-ipp-a-01215f-openshift-default-95bfd666f-rkm78\" (UID: \"4c140dcb-40bd-4cd2-ac16-df81ed774c16\") " pod="openshift-ingress/e2e-ipp-a-01215f-openshift-default-95bfd666f-rkm78"
Oct 30 01:13:29.923355 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:29.901815    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c140dcb-40bd-4cd2-ac16-df81ed774c16-workload-certs\") pod \"e2e-ipp-a-01215f-openshift-default-95bfd666f-rkm78\" (UID: \"4c140dcb-40bd-4cd2-ac16-df81ed774c16\") " pod="openshift-ingress/e2e-ipp-a-01215f-openshift-default-95bfd666f-rkm78"
Oct 30 01:13:29.935272 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:29.933719    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c140dcb-40bd-4cd2-ac16-df81ed774c16-istio-envoy\") pod \"e2e-ipp-a-01215f-openshift-default-95bfd666f-rkm78\" (UID: \"4c140dcb-40bd-4cd2-ac16-df81ed774c16\") " pod="openshift-ingress/e2e-ipp-a-01215f-openshift-default-95bfd666f-rkm78"
Oct 30 01:13:29.946191 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:29.944022    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c140dcb-40bd-4cd2-ac16-df81ed774c16-istio-token\") pod \"e2e-ipp-a-01215f-openshift-default-95bfd666f-rkm78\" (UID: \"4c140dcb-40bd-4cd2-ac16-df81ed774c16\") " pod="openshift-ingress/e2e-ipp-a-01215f-openshift-default-95bfd666f-rkm78"
Oct 30 01:13:29.958404 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:29.947054    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-txhw8\" (UniqueName: \"kubernetes.io/projected/4c140dcb-40bd-4cd2-ac16-df81ed774c16-kube-api-access-txhw8\") pod \"e2e-ipp-a-01215f-openshift-default-95bfd666f-rkm78\" (UID: \"4c140dcb-40bd-4cd2-ac16-df81ed774c16\") " pod="openshift-ingress/e2e-ipp-a-01215f-openshift-default-95bfd666f-rkm78"
Oct 30 01:13:30.017282 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:30.017230    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-01215f-openshift-default-95bfd666f-rkm78"
Oct 30 01:13:30.598361 ip-10-0-128-7 kubenswrapper[5002]: W1030 01:13:30.598328    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c140dcb_40bd_4cd2_ac16_df81ed774c16.slice/crio-c2be7eab5e8eb8158fe1f0994652da997ce772600055919c694ddbe01d8404e6 WatchSource:0}: Error finding container c2be7eab5e8eb8158fe1f0994652da997ce772600055919c694ddbe01d8404e6: Status 404 returned error can't find the container with id c2be7eab5e8eb8158fe1f0994652da997ce772600055919c694ddbe01d8404e6
Oct 30 01:13:30.601520 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:30.601489    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-01215f-openshift-default-95bfd666f-rkm78"]
Oct 30 01:13:30.601855 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:30.601822    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:13:30.601961 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:30.601942    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:13:30.602035 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:30.602005    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:13:31.211844 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:31.211795    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-01215f-openshift-default-95bfd666f-rkm78" event={"ID":"4c140dcb-40bd-4cd2-ac16-df81ed774c16","Type":"ContainerStarted","Data":"d8efe79ab9d9d2cee14f023376facf07f6bf3d680db7bd4489893d8432f8bc3d"}
Oct 30 01:13:31.212343 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:31.211888    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-01215f-openshift-default-95bfd666f-rkm78" event={"ID":"4c140dcb-40bd-4cd2-ac16-df81ed774c16","Type":"ContainerStarted","Data":"c2be7eab5e8eb8158fe1f0994652da997ce772600055919c694ddbe01d8404e6"}
Oct 30 01:13:31.278758 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:31.278452    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-01215f-openshift-default-95bfd666f-rkm78" podStartSLOduration=2.27842946 podStartE2EDuration="2.27842946s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:13:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:13:31.273632991 +0000 UTC m=+2506.890065890" watchObservedRunningTime="2026-10-30 01:13:31.27842946 +0000 UTC m=+2506.894862340"
Oct 30 01:13:32.017844 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:32.017810    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-01215f-openshift-default-95bfd666f-rkm78"
Oct 30 01:13:32.024461 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:32.024436    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-01215f-openshift-default-95bfd666f-rkm78"
Oct 30 01:13:32.220244 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:32.220204    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-01215f-openshift-default-95bfd666f-rkm78"
Oct 30 01:13:32.221212 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:32.221188    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-01215f-openshift-default-95bfd666f-rkm78"
Oct 30 01:13:33.029747 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:33.029710    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0265ca5-5rxfz"]
Oct 30 01:13:33.043190 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:33.043162    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0265ca5-5rxfz"]
Oct 30 01:13:33.106684 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:33.106650    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-4nbkc"]
Oct 30 01:13:33.106871 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:33.106848    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-4nbkc" podUID="6259aaf4-6f22-4781-8265-5ed5724b0f41" containerName="istio-proxy" containerID="cri-o://19e6cbf9915af52ce172cbc672957520edc7465be4c5830b184fa03720f63d6d" gracePeriod=30
Oct 30 01:13:34.496607 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:34.496573    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="822f4720-123e-47f1-b83e-da2c509d6e41" path="/var/lib/kubelet/pods/822f4720-123e-47f1-b83e-da2c509d6e41/volumes"
Oct 30 01:13:38.299462 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:38.299419    5002 generic.go:356] "Generic (PLEG): container finished" podID="6259aaf4-6f22-4781-8265-5ed5724b0f41" containerID="19e6cbf9915af52ce172cbc672957520edc7465be4c5830b184fa03720f63d6d" exitCode=0
Oct 30 01:13:38.299462 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:38.299440    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-4nbkc" event={"ID":"6259aaf4-6f22-4781-8265-5ed5724b0f41","Type":"ContainerDied","Data":"19e6cbf9915af52ce172cbc672957520edc7465be4c5830b184fa03720f63d6d"}
Oct 30 01:13:38.434657 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:38.434627    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-4nbkc"
Oct 30 01:13:38.512492 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:38.512452    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6259aaf4-6f22-4781-8265-5ed5724b0f41-istio-token\" (UniqueName: \"kubernetes.io/projected/6259aaf4-6f22-4781-8265-5ed5724b0f41-istio-token\") pod \"6259aaf4-6f22-4781-8265-5ed5724b0f41\" (UID: \"6259aaf4-6f22-4781-8265-5ed5724b0f41\") "
Oct 30 01:13:38.512674 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:38.512575    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6259aaf4-6f22-4781-8265-5ed5724b0f41-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6259aaf4-6f22-4781-8265-5ed5724b0f41-istio-data\") pod \"6259aaf4-6f22-4781-8265-5ed5724b0f41\" (UID: \"6259aaf4-6f22-4781-8265-5ed5724b0f41\") "
Oct 30 01:13:38.512674 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:38.512617    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6259aaf4-6f22-4781-8265-5ed5724b0f41-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6259aaf4-6f22-4781-8265-5ed5724b0f41-istio-envoy\") pod \"6259aaf4-6f22-4781-8265-5ed5724b0f41\" (UID: \"6259aaf4-6f22-4781-8265-5ed5724b0f41\") "
Oct 30 01:13:38.512801 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:38.512740    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6259aaf4-6f22-4781-8265-5ed5724b0f41-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6259aaf4-6f22-4781-8265-5ed5724b0f41-istio-podinfo\") pod \"6259aaf4-6f22-4781-8265-5ed5724b0f41\" (UID: \"6259aaf4-6f22-4781-8265-5ed5724b0f41\") "
Oct 30 01:13:38.512855 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:38.512839    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6259aaf4-6f22-4781-8265-5ed5724b0f41-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6259aaf4-6f22-4781-8265-5ed5724b0f41-workload-socket\") pod \"6259aaf4-6f22-4781-8265-5ed5724b0f41\" (UID: \"6259aaf4-6f22-4781-8265-5ed5724b0f41\") "
Oct 30 01:13:38.512906 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:38.512897    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6259aaf4-6f22-4781-8265-5ed5724b0f41-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6259aaf4-6f22-4781-8265-5ed5724b0f41-workload-certs\") pod \"6259aaf4-6f22-4781-8265-5ed5724b0f41\" (UID: \"6259aaf4-6f22-4781-8265-5ed5724b0f41\") "
Oct 30 01:13:38.512962 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:38.512915    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6259aaf4-6f22-4781-8265-5ed5724b0f41-istio-data" pod "6259aaf4-6f22-4781-8265-5ed5724b0f41" (UID: "6259aaf4-6f22-4781-8265-5ed5724b0f41"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:13:38.512962 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:38.512947    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6259aaf4-6f22-4781-8265-5ed5724b0f41-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6259aaf4-6f22-4781-8265-5ed5724b0f41-istiod-ca-cert\") pod \"6259aaf4-6f22-4781-8265-5ed5724b0f41\" (UID: \"6259aaf4-6f22-4781-8265-5ed5724b0f41\") "
Oct 30 01:13:38.513062 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:38.512992    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6259aaf4-6f22-4781-8265-5ed5724b0f41-kube-api-access-qvq7c\" (UniqueName: \"kubernetes.io/projected/6259aaf4-6f22-4781-8265-5ed5724b0f41-kube-api-access-qvq7c\") pod \"6259aaf4-6f22-4781-8265-5ed5724b0f41\" (UID: \"6259aaf4-6f22-4781-8265-5ed5724b0f41\") "
Oct 30 01:13:38.513117 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:38.513073    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6259aaf4-6f22-4781-8265-5ed5724b0f41-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6259aaf4-6f22-4781-8265-5ed5724b0f41-credential-socket\") pod \"6259aaf4-6f22-4781-8265-5ed5724b0f41\" (UID: \"6259aaf4-6f22-4781-8265-5ed5724b0f41\") "
Oct 30 01:13:38.513179 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:38.513127    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6259aaf4-6f22-4781-8265-5ed5724b0f41-workload-socket" pod "6259aaf4-6f22-4781-8265-5ed5724b0f41" (UID: "6259aaf4-6f22-4781-8265-5ed5724b0f41"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:13:38.513620 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:38.513590    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6259aaf4-6f22-4781-8265-5ed5724b0f41-istiod-ca-cert" pod "6259aaf4-6f22-4781-8265-5ed5724b0f41" (UID: "6259aaf4-6f22-4781-8265-5ed5724b0f41"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:13:38.513913 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:38.513847    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6259aaf4-6f22-4781-8265-5ed5724b0f41-credential-socket" pod "6259aaf4-6f22-4781-8265-5ed5724b0f41" (UID: "6259aaf4-6f22-4781-8265-5ed5724b0f41"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:13:38.513913 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:38.513733    5002 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6259aaf4-6f22-4781-8265-5ed5724b0f41-istio-data\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:13:38.514106 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:38.513938    5002 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6259aaf4-6f22-4781-8265-5ed5724b0f41-workload-socket\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:13:38.514106 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:38.514006    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6259aaf4-6f22-4781-8265-5ed5724b0f41-workload-certs" pod "6259aaf4-6f22-4781-8265-5ed5724b0f41" (UID: "6259aaf4-6f22-4781-8265-5ed5724b0f41"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:13:38.515466 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:38.515444    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6259aaf4-6f22-4781-8265-5ed5724b0f41-kube-api-access-qvq7c" pod "6259aaf4-6f22-4781-8265-5ed5724b0f41" (UID: "6259aaf4-6f22-4781-8265-5ed5724b0f41"). InnerVolumeSpecName "kube-api-access-qvq7c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:13:38.516781 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:38.516757    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6259aaf4-6f22-4781-8265-5ed5724b0f41-istio-envoy" pod "6259aaf4-6f22-4781-8265-5ed5724b0f41" (UID: "6259aaf4-6f22-4781-8265-5ed5724b0f41"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:13:38.516896 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:38.516839    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6259aaf4-6f22-4781-8265-5ed5724b0f41-istio-token" pod "6259aaf4-6f22-4781-8265-5ed5724b0f41" (UID: "6259aaf4-6f22-4781-8265-5ed5724b0f41"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:13:38.518003 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:38.517976    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6259aaf4-6f22-4781-8265-5ed5724b0f41-istio-podinfo" pod "6259aaf4-6f22-4781-8265-5ed5724b0f41" (UID: "6259aaf4-6f22-4781-8265-5ed5724b0f41"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:13:38.620612 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:38.619459    5002 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6259aaf4-6f22-4781-8265-5ed5724b0f41-workload-certs\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:13:38.620612 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:38.619501    5002 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6259aaf4-6f22-4781-8265-5ed5724b0f41-istiod-ca-cert\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:13:38.620612 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:38.619515    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qvq7c\" (UniqueName: \"kubernetes.io/projected/6259aaf4-6f22-4781-8265-5ed5724b0f41-kube-api-access-qvq7c\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:13:38.620612 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:38.619530    5002 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6259aaf4-6f22-4781-8265-5ed5724b0f41-credential-socket\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:13:38.620612 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:38.619543    5002 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6259aaf4-6f22-4781-8265-5ed5724b0f41-istio-token\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:13:38.620612 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:38.619557    5002 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6259aaf4-6f22-4781-8265-5ed5724b0f41-istio-envoy\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:13:38.620612 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:38.619568    5002 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6259aaf4-6f22-4781-8265-5ed5724b0f41-istio-podinfo\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:13:39.310829 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:39.310790    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-4nbkc" event={"ID":"6259aaf4-6f22-4781-8265-5ed5724b0f41","Type":"ContainerDied","Data":"e3abaed88e13bb32688003b7e123f99daa1358206701083028fa6a5bf4ff3d89"}
Oct 30 01:13:39.311268 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:39.310859    5002 scope.go:122] "RemoveContainer" containerID="19e6cbf9915af52ce172cbc672957520edc7465be4c5830b184fa03720f63d6d"
Oct 30 01:13:39.311268 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:39.310812    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-4nbkc"
Oct 30 01:13:39.401289 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:39.401254    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-4nbkc"]
Oct 30 01:13:39.445375 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:39.445341    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-4nbkc"]
Oct 30 01:13:40.395723 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:40.395683    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-cf4567-8cf8db99b-rw6qt"]
Oct 30 01:13:40.396697 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:40.396673    5002 cpu_manager.go:414] "containerMap: removing container" podUID="6259aaf4-6f22-4781-8265-5ed5724b0f41" containerName="istio-proxy"
Oct 30 01:13:40.396794 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:40.396698    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6259aaf4-6f22-4781-8265-5ed5724b0f41" containerName="istio-proxy"
Oct 30 01:13:40.396961 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:40.396947    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="6259aaf4-6f22-4781-8265-5ed5724b0f41" containerName="istio-proxy"
Oct 30 01:13:40.402215 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:40.402186    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-cf4567-8cf8db99b-rw6qt"
Oct 30 01:13:40.420912 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:40.420879    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-cf4567-8cf8db99b-rw6qt"]
Oct 30 01:13:40.429433 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:40.429345    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-cf4567-dockercfg-x66cd\""
Oct 30 01:13:40.429628 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:40.429493    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-cf4567\""
Oct 30 01:13:40.449195 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:40.449165    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkdd9\" (UniqueName: \"kubernetes.io/projected/6ad27632-6ab9-4b4f-bf4f-3afa10eed73b-kube-api-access-nkdd9\") pod \"payload-processing-e2e-ipp-b-cf4567-8cf8db99b-rw6qt\" (UID: \"6ad27632-6ab9-4b4f-bf4f-3afa10eed73b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-cf4567-8cf8db99b-rw6qt"
Oct 30 01:13:40.449333 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:40.449316    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ad27632-6ab9-4b4f-bf4f-3afa10eed73b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-cf4567-8cf8db99b-rw6qt\" (UID: \"6ad27632-6ab9-4b4f-bf4f-3afa10eed73b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-cf4567-8cf8db99b-rw6qt"
Oct 30 01:13:40.475921 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:40.475888    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-cf4567-9f6f5f475-6wcvd"]
Oct 30 01:13:40.480601 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:40.480570    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cf4567-9f6f5f475-6wcvd"
Oct 30 01:13:40.571080 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:40.567553    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/466ca974-a672-4f29-81c0-5e90b552b583-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-cf4567-9f6f5f475-6wcvd\" (UID: \"466ca974-a672-4f29-81c0-5e90b552b583\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cf4567-9f6f5f475-6wcvd"
Oct 30 01:13:40.571080 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:40.567628    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dgst\" (UniqueName: \"kubernetes.io/projected/466ca974-a672-4f29-81c0-5e90b552b583-kube-api-access-8dgst\") pod \"payload-pre-processing-e2e-ipp-b-cf4567-9f6f5f475-6wcvd\" (UID: \"466ca974-a672-4f29-81c0-5e90b552b583\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cf4567-9f6f5f475-6wcvd"
Oct 30 01:13:40.571080 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:40.567754    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ad27632-6ab9-4b4f-bf4f-3afa10eed73b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-cf4567-8cf8db99b-rw6qt\" (UID: \"6ad27632-6ab9-4b4f-bf4f-3afa10eed73b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-cf4567-8cf8db99b-rw6qt"
Oct 30 01:13:40.571080 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:40.568017    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkdd9\" (UniqueName: \"kubernetes.io/projected/6ad27632-6ab9-4b4f-bf4f-3afa10eed73b-kube-api-access-nkdd9\") pod \"payload-processing-e2e-ipp-b-cf4567-8cf8db99b-rw6qt\" (UID: \"6ad27632-6ab9-4b4f-bf4f-3afa10eed73b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-cf4567-8cf8db99b-rw6qt"
Oct 30 01:13:40.571080 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:40.569309    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ad27632-6ab9-4b4f-bf4f-3afa10eed73b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-cf4567-8cf8db99b-rw6qt\" (UID: \"6ad27632-6ab9-4b4f-bf4f-3afa10eed73b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-cf4567-8cf8db99b-rw6qt"
Oct 30 01:13:40.672453 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:40.642131    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkdd9\" (UniqueName: \"kubernetes.io/projected/6ad27632-6ab9-4b4f-bf4f-3afa10eed73b-kube-api-access-nkdd9\") pod \"payload-processing-e2e-ipp-b-cf4567-8cf8db99b-rw6qt\" (UID: \"6ad27632-6ab9-4b4f-bf4f-3afa10eed73b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-cf4567-8cf8db99b-rw6qt"
Oct 30 01:13:40.845263 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:40.810401    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-cf4567-8cf8db99b-rw6qt"
Oct 30 01:13:40.845263 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:40.841675    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6259aaf4-6f22-4781-8265-5ed5724b0f41" path="/var/lib/kubelet/pods/6259aaf4-6f22-4781-8265-5ed5724b0f41/volumes"
Oct 30 01:13:40.845263 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:40.842191    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-55a4c7-dxkzn"]
Oct 30 01:13:40.848039 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:40.826935    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/466ca974-a672-4f29-81c0-5e90b552b583-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-cf4567-9f6f5f475-6wcvd\" (UID: \"466ca974-a672-4f29-81c0-5e90b552b583\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cf4567-9f6f5f475-6wcvd"
Oct 30 01:13:40.848039 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:40.846675    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dgst\" (UniqueName: \"kubernetes.io/projected/466ca974-a672-4f29-81c0-5e90b552b583-kube-api-access-8dgst\") pod \"payload-pre-processing-e2e-ipp-b-cf4567-9f6f5f475-6wcvd\" (UID: \"466ca974-a672-4f29-81c0-5e90b552b583\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cf4567-9f6f5f475-6wcvd"
Oct 30 01:13:40.848786 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:40.827889    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/466ca974-a672-4f29-81c0-5e90b552b583-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-cf4567-9f6f5f475-6wcvd\" (UID: \"466ca974-a672-4f29-81c0-5e90b552b583\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cf4567-9f6f5f475-6wcvd"
Oct 30 01:13:40.867866 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:40.867837    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-cf4567-9f6f5f475-6wcvd"]
Oct 30 01:13:40.868029 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:40.867893    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-55a4c7-dxkzn"]
Oct 30 01:13:40.868029 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:40.867996    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-55a4c7-dxkzn"
Oct 30 01:13:40.875247 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:40.875028    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dgst\" (UniqueName: \"kubernetes.io/projected/466ca974-a672-4f29-81c0-5e90b552b583-kube-api-access-8dgst\") pod \"payload-pre-processing-e2e-ipp-b-cf4567-9f6f5f475-6wcvd\" (UID: \"466ca974-a672-4f29-81c0-5e90b552b583\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cf4567-9f6f5f475-6wcvd"
Oct 30 01:13:40.884468 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:40.884066    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cf4567-9f6f5f475-6wcvd"
Oct 30 01:13:40.885310 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:40.885285    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:13:40.886297 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:40.885595    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5phsh\""
Oct 30 01:13:40.886297 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:40.885682    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:13:40.886297 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:40.885900    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:13:40.951684 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:40.948654    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xcm6c\" (UniqueName: \"kubernetes.io/projected/1e58a0f6-c2b4-4ee8-8e2e-c03030352ec3-kube-api-access-xcm6c\") pod \"maas-api-revoke-keys-e2e-ipp-a-55a4c7-dxkzn\" (UID: \"1e58a0f6-c2b4-4ee8-8e2e-c03030352ec3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-55a4c7-dxkzn"
Oct 30 01:13:40.951684 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:40.948811    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e58a0f6-c2b4-4ee8-8e2e-c03030352ec3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-55a4c7-dxkzn\" (UID: \"1e58a0f6-c2b4-4ee8-8e2e-c03030352ec3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-55a4c7-dxkzn"
Oct 30 01:13:40.951684 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:40.948905    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e58a0f6-c2b4-4ee8-8e2e-c03030352ec3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-55a4c7-dxkzn\" (UID: \"1e58a0f6-c2b4-4ee8-8e2e-c03030352ec3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-55a4c7-dxkzn"
Oct 30 01:13:41.049912 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:41.049871    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e58a0f6-c2b4-4ee8-8e2e-c03030352ec3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-55a4c7-dxkzn\" (UID: \"1e58a0f6-c2b4-4ee8-8e2e-c03030352ec3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-55a4c7-dxkzn"
Oct 30 01:13:41.050135 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:41.050120    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xcm6c\" (UniqueName: \"kubernetes.io/projected/1e58a0f6-c2b4-4ee8-8e2e-c03030352ec3-kube-api-access-xcm6c\") pod \"maas-api-revoke-keys-e2e-ipp-a-55a4c7-dxkzn\" (UID: \"1e58a0f6-c2b4-4ee8-8e2e-c03030352ec3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-55a4c7-dxkzn"
Oct 30 01:13:41.050265 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:41.050249    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e58a0f6-c2b4-4ee8-8e2e-c03030352ec3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-55a4c7-dxkzn\" (UID: \"1e58a0f6-c2b4-4ee8-8e2e-c03030352ec3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-55a4c7-dxkzn"
Oct 30 01:13:41.050829 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:41.050747    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e58a0f6-c2b4-4ee8-8e2e-c03030352ec3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-55a4c7-dxkzn\" (UID: \"1e58a0f6-c2b4-4ee8-8e2e-c03030352ec3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-55a4c7-dxkzn"
Oct 30 01:13:41.053393 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:41.053354    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e58a0f6-c2b4-4ee8-8e2e-c03030352ec3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-55a4c7-dxkzn\" (UID: \"1e58a0f6-c2b4-4ee8-8e2e-c03030352ec3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-55a4c7-dxkzn"
Oct 30 01:13:41.076094 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:41.076064    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xcm6c\" (UniqueName: \"kubernetes.io/projected/1e58a0f6-c2b4-4ee8-8e2e-c03030352ec3-kube-api-access-xcm6c\") pod \"maas-api-revoke-keys-e2e-ipp-a-55a4c7-dxkzn\" (UID: \"1e58a0f6-c2b4-4ee8-8e2e-c03030352ec3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-55a4c7-dxkzn"
Oct 30 01:13:41.192506 ip-10-0-128-7 kubenswrapper[5002]: W1030 01:13:41.189892    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ad27632_6ab9_4b4f_bf4f_3afa10eed73b.slice/crio-7b200552bc6edfab0fb07909a8591ce57fc6559036a727dd14b27b2458a079b5 WatchSource:0}: Error finding container 7b200552bc6edfab0fb07909a8591ce57fc6559036a727dd14b27b2458a079b5: Status 404 returned error can't find the container with id 7b200552bc6edfab0fb07909a8591ce57fc6559036a727dd14b27b2458a079b5
Oct 30 01:13:41.207653 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:41.207619    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-cf4567-8cf8db99b-rw6qt"]
Oct 30 01:13:41.223296 ip-10-0-128-7 kubenswrapper[5002]: W1030 01:13:41.223262    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod466ca974_a672_4f29_81c0_5e90b552b583.slice/crio-99ea81244f35423aef4e45cb663afafdf7ff47437f4fd7cd6e0f5f18cb15c9f1 WatchSource:0}: Error finding container 99ea81244f35423aef4e45cb663afafdf7ff47437f4fd7cd6e0f5f18cb15c9f1: Status 404 returned error can't find the container with id 99ea81244f35423aef4e45cb663afafdf7ff47437f4fd7cd6e0f5f18cb15c9f1
Oct 30 01:13:41.225009 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:41.224984    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-cf4567-9f6f5f475-6wcvd"]
Oct 30 01:13:41.269563 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:41.269539    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-55a4c7-dxkzn"
Oct 30 01:13:41.336955 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:41.336876    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cf4567-9f6f5f475-6wcvd" event={"ID":"466ca974-a672-4f29-81c0-5e90b552b583","Type":"ContainerStarted","Data":"99ea81244f35423aef4e45cb663afafdf7ff47437f4fd7cd6e0f5f18cb15c9f1"}
Oct 30 01:13:41.339061 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:41.339010    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-cf4567-8cf8db99b-rw6qt" event={"ID":"6ad27632-6ab9-4b4f-bf4f-3afa10eed73b","Type":"ContainerStarted","Data":"1e68e98c9f174b88b569aaafc457d9be0f3cfb05c29e0b22e5019d0c745f7db6"}
Oct 30 01:13:41.339200 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:41.339062    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-cf4567-8cf8db99b-rw6qt" event={"ID":"6ad27632-6ab9-4b4f-bf4f-3afa10eed73b","Type":"ContainerStarted","Data":"7b200552bc6edfab0fb07909a8591ce57fc6559036a727dd14b27b2458a079b5"}
Oct 30 01:13:41.339200 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:41.339187    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-cf4567-8cf8db99b-rw6qt"
Oct 30 01:13:41.386426 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:41.385488    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-cf4567-8cf8db99b-rw6qt" podStartSLOduration=1.385467382 podStartE2EDuration="1.385467382s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:13:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:13:41.38372643 +0000 UTC m=+2517.000159334" watchObservedRunningTime="2026-10-30 01:13:41.385467382 +0000 UTC m=+2517.001900262"
Oct 30 01:13:41.605736 ip-10-0-128-7 kubenswrapper[5002]: W1030 01:13:41.605696    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e58a0f6_c2b4_4ee8_8e2e_c03030352ec3.slice/crio-734d60b0a3d648bbfa87328a63ccb42de3cfa4dae5982cad6af7374d715b7d91 WatchSource:0}: Error finding container 734d60b0a3d648bbfa87328a63ccb42de3cfa4dae5982cad6af7374d715b7d91: Status 404 returned error can't find the container with id 734d60b0a3d648bbfa87328a63ccb42de3cfa4dae5982cad6af7374d715b7d91
Oct 30 01:13:41.606901 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:41.606875    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-55a4c7-dxkzn"]
Oct 30 01:13:42.349696 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:42.349649    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cf4567-9f6f5f475-6wcvd" event={"ID":"466ca974-a672-4f29-81c0-5e90b552b583","Type":"ContainerStarted","Data":"0d549f036e5aafd0e87361b3a9d9f96d62bd8ed3d27fbfca33df93a58919fd26"}
Oct 30 01:13:42.349909 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:42.349731    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cf4567-9f6f5f475-6wcvd"
Oct 30 01:13:42.352868 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:42.352833    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-55a4c7-dxkzn" event={"ID":"1e58a0f6-c2b4-4ee8-8e2e-c03030352ec3","Type":"ContainerStarted","Data":"0c6df505545d38121ca68480bdea497c91715e6386d580ab1b30103e32e654c8"}
Oct 30 01:13:42.353044 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:42.352883    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-55a4c7-dxkzn" event={"ID":"1e58a0f6-c2b4-4ee8-8e2e-c03030352ec3","Type":"ContainerStarted","Data":"734d60b0a3d648bbfa87328a63ccb42de3cfa4dae5982cad6af7374d715b7d91"}
Oct 30 01:13:42.394923 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:42.394860    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cf4567-9f6f5f475-6wcvd" podStartSLOduration=2.394840594 podStartE2EDuration="2.394840594s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:13:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:13:42.393570444 +0000 UTC m=+2518.010003326" watchObservedRunningTime="2026-10-30 01:13:42.394840594 +0000 UTC m=+2518.011273475"
Oct 30 01:13:42.438243 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:42.438189    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-55a4c7-dxkzn" podStartSLOduration=2.438176234 podStartE2EDuration="2.438176234s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:13:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:13:42.437148095 +0000 UTC m=+2518.053581007" watchObservedRunningTime="2026-10-30 01:13:42.438176234 +0000 UTC m=+2518.054609114"
Oct 30 01:13:42.867847 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:13:42.867806    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbjdq" containerName="extractor"
Oct 30 01:13:43.363539 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:43.363492    5002 generic.go:356] "Generic (PLEG): container finished" podID="1e58a0f6-c2b4-4ee8-8e2e-c03030352ec3" containerID="0c6df505545d38121ca68480bdea497c91715e6386d580ab1b30103e32e654c8" exitCode=0
Oct 30 01:13:43.363746 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:43.363561    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-55a4c7-dxkzn" event={"ID":"1e58a0f6-c2b4-4ee8-8e2e-c03030352ec3","Type":"ContainerDied","Data":"0c6df505545d38121ca68480bdea497c91715e6386d580ab1b30103e32e654c8"}
Oct 30 01:13:44.768427 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:44.765412    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-55a4c7-dxkzn"
Oct 30 01:13:44.827410 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:44.822162    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e58a0f6-c2b4-4ee8-8e2e-c03030352ec3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e58a0f6-c2b4-4ee8-8e2e-c03030352ec3-maas-api-openshift-ca\") pod \"1e58a0f6-c2b4-4ee8-8e2e-c03030352ec3\" (UID: \"1e58a0f6-c2b4-4ee8-8e2e-c03030352ec3\") "
Oct 30 01:13:44.827410 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:44.822255    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e58a0f6-c2b4-4ee8-8e2e-c03030352ec3-kube-api-access-xcm6c\" (UniqueName: \"kubernetes.io/projected/1e58a0f6-c2b4-4ee8-8e2e-c03030352ec3-kube-api-access-xcm6c\") pod \"1e58a0f6-c2b4-4ee8-8e2e-c03030352ec3\" (UID: \"1e58a0f6-c2b4-4ee8-8e2e-c03030352ec3\") "
Oct 30 01:13:44.827410 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:44.822304    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1e58a0f6-c2b4-4ee8-8e2e-c03030352ec3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e58a0f6-c2b4-4ee8-8e2e-c03030352ec3-maas-api-xks-ca\") pod \"1e58a0f6-c2b4-4ee8-8e2e-c03030352ec3\" (UID: \"1e58a0f6-c2b4-4ee8-8e2e-c03030352ec3\") "
Oct 30 01:13:44.827410 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:44.824007    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e58a0f6-c2b4-4ee8-8e2e-c03030352ec3-maas-api-openshift-ca" pod "1e58a0f6-c2b4-4ee8-8e2e-c03030352ec3" (UID: "1e58a0f6-c2b4-4ee8-8e2e-c03030352ec3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:13:44.827772 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:44.827417    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e58a0f6-c2b4-4ee8-8e2e-c03030352ec3-maas-api-openshift-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:13:44.865410 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:44.862982    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e58a0f6-c2b4-4ee8-8e2e-c03030352ec3-kube-api-access-xcm6c" pod "1e58a0f6-c2b4-4ee8-8e2e-c03030352ec3" (UID: "1e58a0f6-c2b4-4ee8-8e2e-c03030352ec3"). InnerVolumeSpecName "kube-api-access-xcm6c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:13:44.898400 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:44.892513    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e58a0f6-c2b4-4ee8-8e2e-c03030352ec3-maas-api-xks-ca" pod "1e58a0f6-c2b4-4ee8-8e2e-c03030352ec3" (UID: "1e58a0f6-c2b4-4ee8-8e2e-c03030352ec3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:13:44.952404 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:44.929701    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xcm6c\" (UniqueName: \"kubernetes.io/projected/1e58a0f6-c2b4-4ee8-8e2e-c03030352ec3-kube-api-access-xcm6c\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:13:44.952404 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:44.929740    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e58a0f6-c2b4-4ee8-8e2e-c03030352ec3-maas-api-xks-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:13:45.031134 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:45.030949    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-w4gt5"]
Oct 30 01:13:45.031301 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:45.031252    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-w4gt5" podUID="5e64a3a3-48e4-4e47-bc24-e24fad22c38e" containerName="istio-proxy" containerID="cri-o://44dc2e49c6d87cdcfbb10e60446c1fd1e2ba64f5e1020eee9a93dac6c4a61f79" gracePeriod=30
Oct 30 01:13:45.389453 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:45.389292    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-55a4c7-dxkzn"
Oct 30 01:13:45.389453 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:45.389328    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-55a4c7-dxkzn" event={"ID":"1e58a0f6-c2b4-4ee8-8e2e-c03030352ec3","Type":"ContainerDied","Data":"734d60b0a3d648bbfa87328a63ccb42de3cfa4dae5982cad6af7374d715b7d91"}
Oct 30 01:13:45.389453 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:45.389426    5002 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="734d60b0a3d648bbfa87328a63ccb42de3cfa4dae5982cad6af7374d715b7d91"
Oct 30 01:13:48.611453 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:48.611414    5002 scope.go:122] "RemoveContainer" containerID="77586d68a777918d046f1f66b986c0495cc11601d5e8e1992d3896bd76dd479f"
Oct 30 01:13:48.621954 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:48.621930    5002 scope.go:122] "RemoveContainer" containerID="206d5e7265fe257b14ecab6dd2193303682aae5bcda4e1d225a5101eec1250d4"
Oct 30 01:13:50.253614 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:50.253581    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7d8z4"]
Oct 30 01:13:50.327875 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:50.327841    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-w4gt5"
Oct 30 01:13:50.421345 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:50.421292    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e64a3a3-48e4-4e47-bc24-e24fad22c38e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e64a3a3-48e4-4e47-bc24-e24fad22c38e-workload-certs\") pod \"5e64a3a3-48e4-4e47-bc24-e24fad22c38e\" (UID: \"5e64a3a3-48e4-4e47-bc24-e24fad22c38e\") "
Oct 30 01:13:50.421605 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:50.421368    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e64a3a3-48e4-4e47-bc24-e24fad22c38e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e64a3a3-48e4-4e47-bc24-e24fad22c38e-istio-envoy\") pod \"5e64a3a3-48e4-4e47-bc24-e24fad22c38e\" (UID: \"5e64a3a3-48e4-4e47-bc24-e24fad22c38e\") "
Oct 30 01:13:50.421605 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:50.421539    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e64a3a3-48e4-4e47-bc24-e24fad22c38e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e64a3a3-48e4-4e47-bc24-e24fad22c38e-istio-data\") pod \"5e64a3a3-48e4-4e47-bc24-e24fad22c38e\" (UID: \"5e64a3a3-48e4-4e47-bc24-e24fad22c38e\") "
Oct 30 01:13:50.421750 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:50.421633    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e64a3a3-48e4-4e47-bc24-e24fad22c38e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e64a3a3-48e4-4e47-bc24-e24fad22c38e-istiod-ca-cert\") pod \"5e64a3a3-48e4-4e47-bc24-e24fad22c38e\" (UID: \"5e64a3a3-48e4-4e47-bc24-e24fad22c38e\") "
Oct 30 01:13:50.421750 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:50.421711    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e64a3a3-48e4-4e47-bc24-e24fad22c38e-istio-token\" (UniqueName: \"kubernetes.io/projected/5e64a3a3-48e4-4e47-bc24-e24fad22c38e-istio-token\") pod \"5e64a3a3-48e4-4e47-bc24-e24fad22c38e\" (UID: \"5e64a3a3-48e4-4e47-bc24-e24fad22c38e\") "
Oct 30 01:13:50.421919 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:50.421779    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e64a3a3-48e4-4e47-bc24-e24fad22c38e-kube-api-access-q956l\" (UniqueName: \"kubernetes.io/projected/5e64a3a3-48e4-4e47-bc24-e24fad22c38e-kube-api-access-q956l\") pod \"5e64a3a3-48e4-4e47-bc24-e24fad22c38e\" (UID: \"5e64a3a3-48e4-4e47-bc24-e24fad22c38e\") "
Oct 30 01:13:50.421919 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:50.421874    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e64a3a3-48e4-4e47-bc24-e24fad22c38e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e64a3a3-48e4-4e47-bc24-e24fad22c38e-credential-socket\") pod \"5e64a3a3-48e4-4e47-bc24-e24fad22c38e\" (UID: \"5e64a3a3-48e4-4e47-bc24-e24fad22c38e\") "
Oct 30 01:13:50.421919 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:50.421907    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5e64a3a3-48e4-4e47-bc24-e24fad22c38e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e64a3a3-48e4-4e47-bc24-e24fad22c38e-istio-podinfo\") pod \"5e64a3a3-48e4-4e47-bc24-e24fad22c38e\" (UID: \"5e64a3a3-48e4-4e47-bc24-e24fad22c38e\") "
Oct 30 01:13:50.422113 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:50.421953    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e64a3a3-48e4-4e47-bc24-e24fad22c38e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e64a3a3-48e4-4e47-bc24-e24fad22c38e-workload-socket\") pod \"5e64a3a3-48e4-4e47-bc24-e24fad22c38e\" (UID: \"5e64a3a3-48e4-4e47-bc24-e24fad22c38e\") "
Oct 30 01:13:50.423128 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:50.422578    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e64a3a3-48e4-4e47-bc24-e24fad22c38e-credential-socket" pod "5e64a3a3-48e4-4e47-bc24-e24fad22c38e" (UID: "5e64a3a3-48e4-4e47-bc24-e24fad22c38e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:13:50.423128 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:50.422799    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e64a3a3-48e4-4e47-bc24-e24fad22c38e-workload-socket" pod "5e64a3a3-48e4-4e47-bc24-e24fad22c38e" (UID: "5e64a3a3-48e4-4e47-bc24-e24fad22c38e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:13:50.423128 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:50.422822    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e64a3a3-48e4-4e47-bc24-e24fad22c38e-istio-data" pod "5e64a3a3-48e4-4e47-bc24-e24fad22c38e" (UID: "5e64a3a3-48e4-4e47-bc24-e24fad22c38e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:13:50.423128 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:50.423003    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e64a3a3-48e4-4e47-bc24-e24fad22c38e-istiod-ca-cert" pod "5e64a3a3-48e4-4e47-bc24-e24fad22c38e" (UID: "5e64a3a3-48e4-4e47-bc24-e24fad22c38e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:13:50.423464 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:50.423160    5002 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e64a3a3-48e4-4e47-bc24-e24fad22c38e-credential-socket\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:13:50.424009 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:50.423984    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e64a3a3-48e4-4e47-bc24-e24fad22c38e-workload-certs" pod "5e64a3a3-48e4-4e47-bc24-e24fad22c38e" (UID: "5e64a3a3-48e4-4e47-bc24-e24fad22c38e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:13:50.426294 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:50.426255    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e64a3a3-48e4-4e47-bc24-e24fad22c38e-istio-envoy" pod "5e64a3a3-48e4-4e47-bc24-e24fad22c38e" (UID: "5e64a3a3-48e4-4e47-bc24-e24fad22c38e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:13:50.432100 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:50.431126    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e64a3a3-48e4-4e47-bc24-e24fad22c38e-istio-token" pod "5e64a3a3-48e4-4e47-bc24-e24fad22c38e" (UID: "5e64a3a3-48e4-4e47-bc24-e24fad22c38e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:13:50.432100 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:50.431166    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e64a3a3-48e4-4e47-bc24-e24fad22c38e-kube-api-access-q956l" pod "5e64a3a3-48e4-4e47-bc24-e24fad22c38e" (UID: "5e64a3a3-48e4-4e47-bc24-e24fad22c38e"). InnerVolumeSpecName "kube-api-access-q956l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:13:50.433160 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:50.433110    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5e64a3a3-48e4-4e47-bc24-e24fad22c38e-istio-podinfo" pod "5e64a3a3-48e4-4e47-bc24-e24fad22c38e" (UID: "5e64a3a3-48e4-4e47-bc24-e24fad22c38e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:13:50.441811 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:50.441784    5002 generic.go:356] "Generic (PLEG): container finished" podID="5e64a3a3-48e4-4e47-bc24-e24fad22c38e" containerID="44dc2e49c6d87cdcfbb10e60446c1fd1e2ba64f5e1020eee9a93dac6c4a61f79" exitCode=0
Oct 30 01:13:50.441945 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:50.441837    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-w4gt5"
Oct 30 01:13:50.441945 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:50.441879    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-w4gt5" event={"ID":"5e64a3a3-48e4-4e47-bc24-e24fad22c38e","Type":"ContainerDied","Data":"44dc2e49c6d87cdcfbb10e60446c1fd1e2ba64f5e1020eee9a93dac6c4a61f79"}
Oct 30 01:13:50.442058 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:50.441951    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-w4gt5" event={"ID":"5e64a3a3-48e4-4e47-bc24-e24fad22c38e","Type":"ContainerDied","Data":"05518878beecaaa406aab8185583fadf33923c11c55f8df870f085c1b05729c1"}
Oct 30 01:13:50.442058 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:50.441986    5002 scope.go:122] "RemoveContainer" containerID="44dc2e49c6d87cdcfbb10e60446c1fd1e2ba64f5e1020eee9a93dac6c4a61f79"
Oct 30 01:13:50.468515 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:50.468488    5002 scope.go:122] "RemoveContainer" containerID="44dc2e49c6d87cdcfbb10e60446c1fd1e2ba64f5e1020eee9a93dac6c4a61f79"
Oct 30 01:13:50.468884 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:13:50.468861    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44dc2e49c6d87cdcfbb10e60446c1fd1e2ba64f5e1020eee9a93dac6c4a61f79\": container with ID starting with 44dc2e49c6d87cdcfbb10e60446c1fd1e2ba64f5e1020eee9a93dac6c4a61f79 not found: ID does not exist" containerID="44dc2e49c6d87cdcfbb10e60446c1fd1e2ba64f5e1020eee9a93dac6c4a61f79"
Oct 30 01:13:50.468958 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:50.468895    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44dc2e49c6d87cdcfbb10e60446c1fd1e2ba64f5e1020eee9a93dac6c4a61f79"} err="failed to get container status \"44dc2e49c6d87cdcfbb10e60446c1fd1e2ba64f5e1020eee9a93dac6c4a61f79\": rpc error: code = NotFound desc = could not find container \"44dc2e49c6d87cdcfbb10e60446c1fd1e2ba64f5e1020eee9a93dac6c4a61f79\": container with ID starting with 44dc2e49c6d87cdcfbb10e60446c1fd1e2ba64f5e1020eee9a93dac6c4a61f79 not found: ID does not exist"
Oct 30 01:13:50.507750 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:50.507705    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-w4gt5"]
Oct 30 01:13:50.512325 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:50.512302    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-w4gt5"]
Oct 30 01:13:50.524921 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:50.524892    5002 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e64a3a3-48e4-4e47-bc24-e24fad22c38e-workload-certs\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:13:50.524921 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:50.524919    5002 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e64a3a3-48e4-4e47-bc24-e24fad22c38e-istio-envoy\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:13:50.524921 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:50.524927    5002 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e64a3a3-48e4-4e47-bc24-e24fad22c38e-istio-data\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:13:50.525115 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:50.524937    5002 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e64a3a3-48e4-4e47-bc24-e24fad22c38e-istiod-ca-cert\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:13:50.525115 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:50.524946    5002 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e64a3a3-48e4-4e47-bc24-e24fad22c38e-istio-token\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:13:50.525115 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:50.524955    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q956l\" (UniqueName: \"kubernetes.io/projected/5e64a3a3-48e4-4e47-bc24-e24fad22c38e-kube-api-access-q956l\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:13:50.525115 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:50.524965    5002 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e64a3a3-48e4-4e47-bc24-e24fad22c38e-istio-podinfo\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:13:50.525115 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:50.524975    5002 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e64a3a3-48e4-4e47-bc24-e24fad22c38e-workload-socket\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:13:52.356848 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:52.356814    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-cf4567-8cf8db99b-rw6qt"
Oct 30 01:13:52.507413 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:52.505762    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e64a3a3-48e4-4e47-bc24-e24fad22c38e" path="/var/lib/kubelet/pods/5e64a3a3-48e4-4e47-bc24-e24fad22c38e/volumes"
Oct 30 01:13:53.391409 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:53.390591    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cf4567-9f6f5f475-6wcvd"
Oct 30 01:13:59.779548 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:59.779504    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-172c4d2b-wkgr7"]
Oct 30 01:13:59.780532 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:59.780510    5002 cpu_manager.go:414] "containerMap: removing container" podUID="5e64a3a3-48e4-4e47-bc24-e24fad22c38e" containerName="istio-proxy"
Oct 30 01:13:59.780532 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:59.780532    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e64a3a3-48e4-4e47-bc24-e24fad22c38e" containerName="istio-proxy"
Oct 30 01:13:59.780731 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:59.780578    5002 cpu_manager.go:414] "containerMap: removing container" podUID="1e58a0f6-c2b4-4ee8-8e2e-c03030352ec3" containerName="revoke-keys"
Oct 30 01:13:59.780731 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:59.780586    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e58a0f6-c2b4-4ee8-8e2e-c03030352ec3" containerName="revoke-keys"
Oct 30 01:13:59.780845 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:59.780777    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e64a3a3-48e4-4e47-bc24-e24fad22c38e" containerName="istio-proxy"
Oct 30 01:13:59.780845 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:59.780796    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e58a0f6-c2b4-4ee8-8e2e-c03030352ec3" containerName="revoke-keys"
Oct 30 01:13:59.786525 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:59.786501    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-172c4d2b-wkgr7"
Oct 30 01:13:59.804547 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:59.804514    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-172c4d2b-wkgr7"]
Oct 30 01:13:59.816034 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:59.809417    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:13:59.816034 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:59.809537    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5phsh\""
Oct 30 01:13:59.816034 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:59.809581    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:13:59.816629 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:59.816605    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:13:59.875200 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:59.875163    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ad005b6-91dd-458c-9be3-7696b9ae2e15-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-172c4d2b-wkgr7\" (UID: \"0ad005b6-91dd-458c-9be3-7696b9ae2e15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-172c4d2b-wkgr7"
Oct 30 01:13:59.875408 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:59.875280    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ad005b6-91dd-458c-9be3-7696b9ae2e15-maas-api-xks-ca\") pod \"maas-api-revoke-sub-172c4d2b-wkgr7\" (UID: \"0ad005b6-91dd-458c-9be3-7696b9ae2e15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-172c4d2b-wkgr7"
Oct 30 01:13:59.875491 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:59.875469    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87glj\" (UniqueName: \"kubernetes.io/projected/0ad005b6-91dd-458c-9be3-7696b9ae2e15-kube-api-access-87glj\") pod \"maas-api-revoke-sub-172c4d2b-wkgr7\" (UID: \"0ad005b6-91dd-458c-9be3-7696b9ae2e15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-172c4d2b-wkgr7"
Oct 30 01:13:59.977257 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:59.977215    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ad005b6-91dd-458c-9be3-7696b9ae2e15-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-172c4d2b-wkgr7\" (UID: \"0ad005b6-91dd-458c-9be3-7696b9ae2e15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-172c4d2b-wkgr7"
Oct 30 01:13:59.977508 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:59.977297    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ad005b6-91dd-458c-9be3-7696b9ae2e15-maas-api-xks-ca\") pod \"maas-api-revoke-sub-172c4d2b-wkgr7\" (UID: \"0ad005b6-91dd-458c-9be3-7696b9ae2e15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-172c4d2b-wkgr7"
Oct 30 01:13:59.977508 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:59.977452    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87glj\" (UniqueName: \"kubernetes.io/projected/0ad005b6-91dd-458c-9be3-7696b9ae2e15-kube-api-access-87glj\") pod \"maas-api-revoke-sub-172c4d2b-wkgr7\" (UID: \"0ad005b6-91dd-458c-9be3-7696b9ae2e15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-172c4d2b-wkgr7"
Oct 30 01:13:59.978138 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:59.978106    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ad005b6-91dd-458c-9be3-7696b9ae2e15-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-172c4d2b-wkgr7\" (UID: \"0ad005b6-91dd-458c-9be3-7696b9ae2e15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-172c4d2b-wkgr7"
Oct 30 01:13:59.980510 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:13:59.980483    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ad005b6-91dd-458c-9be3-7696b9ae2e15-maas-api-xks-ca\") pod \"maas-api-revoke-sub-172c4d2b-wkgr7\" (UID: \"0ad005b6-91dd-458c-9be3-7696b9ae2e15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-172c4d2b-wkgr7"
Oct 30 01:14:00.002851 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:00.002822    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-87glj\" (UniqueName: \"kubernetes.io/projected/0ad005b6-91dd-458c-9be3-7696b9ae2e15-kube-api-access-87glj\") pod \"maas-api-revoke-sub-172c4d2b-wkgr7\" (UID: \"0ad005b6-91dd-458c-9be3-7696b9ae2e15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-172c4d2b-wkgr7"
Oct 30 01:14:00.108231 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:00.108150    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-172c4d2b-wkgr7"
Oct 30 01:14:00.470660 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:00.470614    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-172c4d2b-wkgr7"]
Oct 30 01:14:00.604821 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:00.604781    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-172c4d2b-wkgr7" event={"ID":"0ad005b6-91dd-458c-9be3-7696b9ae2e15","Type":"ContainerStarted","Data":"3558b4974cbb66a543dd95199e3cb1012febca7e92ac7eaa86e8fdbd80722d35"}
Oct 30 01:14:01.623408 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:01.621315    5002 generic.go:356] "Generic (PLEG): container finished" podID="0ad005b6-91dd-458c-9be3-7696b9ae2e15" containerID="a7d528c6408b9ba5eabd8cd05c298d1ddbf23c67f108c84f3be409838ab78c36" exitCode=0
Oct 30 01:14:01.623408 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:01.621637    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-172c4d2b-wkgr7" event={"ID":"0ad005b6-91dd-458c-9be3-7696b9ae2e15","Type":"ContainerDied","Data":"a7d528c6408b9ba5eabd8cd05c298d1ddbf23c67f108c84f3be409838ab78c36"}
Oct 30 01:14:02.831006 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:02.830980    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-172c4d2b-wkgr7"
Oct 30 01:14:02.927500 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:02.927399    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ad005b6-91dd-458c-9be3-7696b9ae2e15-kube-api-access-87glj\" (UniqueName: \"kubernetes.io/projected/0ad005b6-91dd-458c-9be3-7696b9ae2e15-kube-api-access-87glj\") pod \"0ad005b6-91dd-458c-9be3-7696b9ae2e15\" (UID: \"0ad005b6-91dd-458c-9be3-7696b9ae2e15\") "
Oct 30 01:14:02.927500 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:02.927494    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0ad005b6-91dd-458c-9be3-7696b9ae2e15-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ad005b6-91dd-458c-9be3-7696b9ae2e15-maas-api-xks-ca\") pod \"0ad005b6-91dd-458c-9be3-7696b9ae2e15\" (UID: \"0ad005b6-91dd-458c-9be3-7696b9ae2e15\") "
Oct 30 01:14:02.927760 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:02.927554    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ad005b6-91dd-458c-9be3-7696b9ae2e15-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ad005b6-91dd-458c-9be3-7696b9ae2e15-maas-api-openshift-ca\") pod \"0ad005b6-91dd-458c-9be3-7696b9ae2e15\" (UID: \"0ad005b6-91dd-458c-9be3-7696b9ae2e15\") "
Oct 30 01:14:02.928024 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:02.927996    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ad005b6-91dd-458c-9be3-7696b9ae2e15-maas-api-openshift-ca" pod "0ad005b6-91dd-458c-9be3-7696b9ae2e15" (UID: "0ad005b6-91dd-458c-9be3-7696b9ae2e15"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:14:02.928373 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:02.928352    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ad005b6-91dd-458c-9be3-7696b9ae2e15-maas-api-openshift-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:14:02.930023 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:02.929996    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ad005b6-91dd-458c-9be3-7696b9ae2e15-maas-api-xks-ca" pod "0ad005b6-91dd-458c-9be3-7696b9ae2e15" (UID: "0ad005b6-91dd-458c-9be3-7696b9ae2e15"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:14:02.930233 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:02.930212    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ad005b6-91dd-458c-9be3-7696b9ae2e15-kube-api-access-87glj" pod "0ad005b6-91dd-458c-9be3-7696b9ae2e15" (UID: "0ad005b6-91dd-458c-9be3-7696b9ae2e15"). InnerVolumeSpecName "kube-api-access-87glj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:14:03.073911 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:03.073871    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-87glj\" (UniqueName: \"kubernetes.io/projected/0ad005b6-91dd-458c-9be3-7696b9ae2e15-kube-api-access-87glj\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:14:03.074118 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:03.074102    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ad005b6-91dd-458c-9be3-7696b9ae2e15-maas-api-xks-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:14:03.638054 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:03.638023    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-172c4d2b-wkgr7"
Oct 30 01:14:03.638217 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:03.638031    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-172c4d2b-wkgr7" event={"ID":"0ad005b6-91dd-458c-9be3-7696b9ae2e15","Type":"ContainerDied","Data":"3558b4974cbb66a543dd95199e3cb1012febca7e92ac7eaa86e8fdbd80722d35"}
Oct 30 01:14:03.638217 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:03.638129    5002 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3558b4974cbb66a543dd95199e3cb1012febca7e92ac7eaa86e8fdbd80722d35"
Oct 30 01:14:10.105242 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:10.105196    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24acd507-28chc"]
Oct 30 01:14:10.106185 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:10.106165    5002 cpu_manager.go:414] "containerMap: removing container" podUID="0ad005b6-91dd-458c-9be3-7696b9ae2e15" containerName="revoke-keys"
Oct 30 01:14:10.106233 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:10.106188    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ad005b6-91dd-458c-9be3-7696b9ae2e15" containerName="revoke-keys"
Oct 30 01:14:10.106435 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:10.106423    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ad005b6-91dd-458c-9be3-7696b9ae2e15" containerName="revoke-keys"
Oct 30 01:14:10.110601 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:10.110582    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24acd507-28chc"
Oct 30 01:14:10.126552 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:10.126525    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24acd507-28chc"]
Oct 30 01:14:10.139094 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:10.135372    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:14:10.139094 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:10.135474    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:14:10.139094 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:10.135494    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5phsh\""
Oct 30 01:14:10.139094 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:10.135372    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:14:10.256877 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:10.256842    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46dd358d-4793-4fa7-ad99-06c2e2147c0f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-24acd507-28chc\" (UID: \"46dd358d-4793-4fa7-ad99-06c2e2147c0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24acd507-28chc"
Oct 30 01:14:10.257050 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:10.256889    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kl8st\" (UniqueName: \"kubernetes.io/projected/46dd358d-4793-4fa7-ad99-06c2e2147c0f-kube-api-access-kl8st\") pod \"maas-api-revoke-sub-24acd507-28chc\" (UID: \"46dd358d-4793-4fa7-ad99-06c2e2147c0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24acd507-28chc"
Oct 30 01:14:10.257050 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:10.257020    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46dd358d-4793-4fa7-ad99-06c2e2147c0f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-24acd507-28chc\" (UID: \"46dd358d-4793-4fa7-ad99-06c2e2147c0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24acd507-28chc"
Oct 30 01:14:10.358948 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:10.358852    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46dd358d-4793-4fa7-ad99-06c2e2147c0f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-24acd507-28chc\" (UID: \"46dd358d-4793-4fa7-ad99-06c2e2147c0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24acd507-28chc"
Oct 30 01:14:10.359193 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:10.359173    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46dd358d-4793-4fa7-ad99-06c2e2147c0f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-24acd507-28chc\" (UID: \"46dd358d-4793-4fa7-ad99-06c2e2147c0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24acd507-28chc"
Oct 30 01:14:10.359281 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:10.359219    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kl8st\" (UniqueName: \"kubernetes.io/projected/46dd358d-4793-4fa7-ad99-06c2e2147c0f-kube-api-access-kl8st\") pod \"maas-api-revoke-sub-24acd507-28chc\" (UID: \"46dd358d-4793-4fa7-ad99-06c2e2147c0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24acd507-28chc"
Oct 30 01:14:10.359574 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:10.359549    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46dd358d-4793-4fa7-ad99-06c2e2147c0f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-24acd507-28chc\" (UID: \"46dd358d-4793-4fa7-ad99-06c2e2147c0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24acd507-28chc"
Oct 30 01:14:10.361610 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:10.361589    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46dd358d-4793-4fa7-ad99-06c2e2147c0f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-24acd507-28chc\" (UID: \"46dd358d-4793-4fa7-ad99-06c2e2147c0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24acd507-28chc"
Oct 30 01:14:10.384748 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:10.384709    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kl8st\" (UniqueName: \"kubernetes.io/projected/46dd358d-4793-4fa7-ad99-06c2e2147c0f-kube-api-access-kl8st\") pod \"maas-api-revoke-sub-24acd507-28chc\" (UID: \"46dd358d-4793-4fa7-ad99-06c2e2147c0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24acd507-28chc"
Oct 30 01:14:10.427070 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:10.427029    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24acd507-28chc"
Oct 30 01:14:10.746234 ip-10-0-128-7 kubenswrapper[5002]: W1030 01:14:10.746202    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46dd358d_4793_4fa7_ad99_06c2e2147c0f.slice/crio-2a7d0d9ab1632e35b519497d20c9ec517e590e70b3e3c9ac1f2b1b3aa1545ce3 WatchSource:0}: Error finding container 2a7d0d9ab1632e35b519497d20c9ec517e590e70b3e3c9ac1f2b1b3aa1545ce3: Status 404 returned error can't find the container with id 2a7d0d9ab1632e35b519497d20c9ec517e590e70b3e3c9ac1f2b1b3aa1545ce3
Oct 30 01:14:10.748814 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:10.748779    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24acd507-28chc"]
Oct 30 01:14:11.709337 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:11.709300    5002 generic.go:356] "Generic (PLEG): container finished" podID="46dd358d-4793-4fa7-ad99-06c2e2147c0f" containerID="5b6f7ddf81803ecf23ca5464b4be2d0086aed004b40663c0f8cb3a03d41cb044" exitCode=0
Oct 30 01:14:11.709788 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:11.709407    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24acd507-28chc" event={"ID":"46dd358d-4793-4fa7-ad99-06c2e2147c0f","Type":"ContainerDied","Data":"5b6f7ddf81803ecf23ca5464b4be2d0086aed004b40663c0f8cb3a03d41cb044"}
Oct 30 01:14:11.709788 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:11.709473    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24acd507-28chc" event={"ID":"46dd358d-4793-4fa7-ad99-06c2e2147c0f","Type":"ContainerStarted","Data":"2a7d0d9ab1632e35b519497d20c9ec517e590e70b3e3c9ac1f2b1b3aa1545ce3"}
Oct 30 01:14:12.869570 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:14:12.869527    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbjdq" containerName="extractor"
Oct 30 01:14:12.912778 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:12.912750    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24acd507-28chc"
Oct 30 01:14:13.019554 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:13.019456    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46dd358d-4793-4fa7-ad99-06c2e2147c0f-kube-api-access-kl8st\" (UniqueName: \"kubernetes.io/projected/46dd358d-4793-4fa7-ad99-06c2e2147c0f-kube-api-access-kl8st\") pod \"46dd358d-4793-4fa7-ad99-06c2e2147c0f\" (UID: \"46dd358d-4793-4fa7-ad99-06c2e2147c0f\") "
Oct 30 01:14:13.019554 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:13.019542    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/46dd358d-4793-4fa7-ad99-06c2e2147c0f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46dd358d-4793-4fa7-ad99-06c2e2147c0f-maas-api-xks-ca\") pod \"46dd358d-4793-4fa7-ad99-06c2e2147c0f\" (UID: \"46dd358d-4793-4fa7-ad99-06c2e2147c0f\") "
Oct 30 01:14:13.019810 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:13.019653    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46dd358d-4793-4fa7-ad99-06c2e2147c0f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46dd358d-4793-4fa7-ad99-06c2e2147c0f-maas-api-openshift-ca\") pod \"46dd358d-4793-4fa7-ad99-06c2e2147c0f\" (UID: \"46dd358d-4793-4fa7-ad99-06c2e2147c0f\") "
Oct 30 01:14:13.020115 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:13.020087    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46dd358d-4793-4fa7-ad99-06c2e2147c0f-maas-api-openshift-ca" pod "46dd358d-4793-4fa7-ad99-06c2e2147c0f" (UID: "46dd358d-4793-4fa7-ad99-06c2e2147c0f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:14:13.020520 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:13.020496    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46dd358d-4793-4fa7-ad99-06c2e2147c0f-maas-api-openshift-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:14:13.021824 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:13.021801    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46dd358d-4793-4fa7-ad99-06c2e2147c0f-maas-api-xks-ca" pod "46dd358d-4793-4fa7-ad99-06c2e2147c0f" (UID: "46dd358d-4793-4fa7-ad99-06c2e2147c0f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:14:13.021930 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:13.021870    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46dd358d-4793-4fa7-ad99-06c2e2147c0f-kube-api-access-kl8st" pod "46dd358d-4793-4fa7-ad99-06c2e2147c0f" (UID: "46dd358d-4793-4fa7-ad99-06c2e2147c0f"). InnerVolumeSpecName "kube-api-access-kl8st". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:14:13.122186 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:13.122152    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kl8st\" (UniqueName: \"kubernetes.io/projected/46dd358d-4793-4fa7-ad99-06c2e2147c0f-kube-api-access-kl8st\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:14:13.122186 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:13.122184    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46dd358d-4793-4fa7-ad99-06c2e2147c0f-maas-api-xks-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:14:13.728469 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:13.728430    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24acd507-28chc"
Oct 30 01:14:13.728469 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:13.728444    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24acd507-28chc" event={"ID":"46dd358d-4793-4fa7-ad99-06c2e2147c0f","Type":"ContainerDied","Data":"2a7d0d9ab1632e35b519497d20c9ec517e590e70b3e3c9ac1f2b1b3aa1545ce3"}
Oct 30 01:14:13.728739 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:13.728508    5002 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2a7d0d9ab1632e35b519497d20c9ec517e590e70b3e3c9ac1f2b1b3aa1545ce3"
Oct 30 01:14:20.966239 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:20.966198    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-73f667-2nstj"]
Oct 30 01:14:20.967118 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:20.967095    5002 cpu_manager.go:414] "containerMap: removing container" podUID="46dd358d-4793-4fa7-ad99-06c2e2147c0f" containerName="revoke-keys"
Oct 30 01:14:20.967230 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:20.967119    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46dd358d-4793-4fa7-ad99-06c2e2147c0f" containerName="revoke-keys"
Oct 30 01:14:20.967315 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:20.967301    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="46dd358d-4793-4fa7-ad99-06c2e2147c0f" containerName="revoke-keys"
Oct 30 01:14:20.971372 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:20.971349    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-73f667-2nstj"
Oct 30 01:14:20.991256 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:20.991188    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-73f667-2nstj"]
Oct 30 01:14:20.999963 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:20.998501    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:14:20.999963 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:20.998645    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:14:21.000270 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:21.000241    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:14:21.005502 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:21.005475    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5phsh\""
Oct 30 01:14:21.012202 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:21.012175    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-635e17f5-v97r9"]
Oct 30 01:14:21.016727 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:21.016706    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-635e17f5-v97r9"
Oct 30 01:14:21.041295 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:21.040941    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-635e17f5-v97r9"]
Oct 30 01:14:21.087453 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:21.087400    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30c897f9-4170-4ebd-bfe0-45c50091de94-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-73f667-2nstj\" (UID: \"30c897f9-4170-4ebd-bfe0-45c50091de94\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-73f667-2nstj"
Oct 30 01:14:21.087670 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:21.087597    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30c897f9-4170-4ebd-bfe0-45c50091de94-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-73f667-2nstj\" (UID: \"30c897f9-4170-4ebd-bfe0-45c50091de94\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-73f667-2nstj"
Oct 30 01:14:21.087827 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:21.087808    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22278f91-d2c8-4808-a572-17290bda0150-maas-api-xks-ca\") pod \"maas-api-revoke-sub-635e17f5-v97r9\" (UID: \"22278f91-d2c8-4808-a572-17290bda0150\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-635e17f5-v97r9"
Oct 30 01:14:21.087890 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:21.087862    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gl94r\" (UniqueName: \"kubernetes.io/projected/30c897f9-4170-4ebd-bfe0-45c50091de94-kube-api-access-gl94r\") pod \"maas-api-revoke-keys-e2e-ipp-a-73f667-2nstj\" (UID: \"30c897f9-4170-4ebd-bfe0-45c50091de94\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-73f667-2nstj"
Oct 30 01:14:21.087983 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:21.087957    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22278f91-d2c8-4808-a572-17290bda0150-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-635e17f5-v97r9\" (UID: \"22278f91-d2c8-4808-a572-17290bda0150\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-635e17f5-v97r9"
Oct 30 01:14:21.088045 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:21.088034    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chwzj\" (UniqueName: \"kubernetes.io/projected/22278f91-d2c8-4808-a572-17290bda0150-kube-api-access-chwzj\") pod \"maas-api-revoke-sub-635e17f5-v97r9\" (UID: \"22278f91-d2c8-4808-a572-17290bda0150\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-635e17f5-v97r9"
Oct 30 01:14:21.189136 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:21.189087    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chwzj\" (UniqueName: \"kubernetes.io/projected/22278f91-d2c8-4808-a572-17290bda0150-kube-api-access-chwzj\") pod \"maas-api-revoke-sub-635e17f5-v97r9\" (UID: \"22278f91-d2c8-4808-a572-17290bda0150\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-635e17f5-v97r9"
Oct 30 01:14:21.189351 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:21.189223    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30c897f9-4170-4ebd-bfe0-45c50091de94-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-73f667-2nstj\" (UID: \"30c897f9-4170-4ebd-bfe0-45c50091de94\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-73f667-2nstj"
Oct 30 01:14:21.189351 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:21.189322    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30c897f9-4170-4ebd-bfe0-45c50091de94-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-73f667-2nstj\" (UID: \"30c897f9-4170-4ebd-bfe0-45c50091de94\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-73f667-2nstj"
Oct 30 01:14:21.189519 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:21.189509    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22278f91-d2c8-4808-a572-17290bda0150-maas-api-xks-ca\") pod \"maas-api-revoke-sub-635e17f5-v97r9\" (UID: \"22278f91-d2c8-4808-a572-17290bda0150\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-635e17f5-v97r9"
Oct 30 01:14:21.189582 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:21.189559    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gl94r\" (UniqueName: \"kubernetes.io/projected/30c897f9-4170-4ebd-bfe0-45c50091de94-kube-api-access-gl94r\") pod \"maas-api-revoke-keys-e2e-ipp-a-73f667-2nstj\" (UID: \"30c897f9-4170-4ebd-bfe0-45c50091de94\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-73f667-2nstj"
Oct 30 01:14:21.189645 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:21.189597    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22278f91-d2c8-4808-a572-17290bda0150-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-635e17f5-v97r9\" (UID: \"22278f91-d2c8-4808-a572-17290bda0150\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-635e17f5-v97r9"
Oct 30 01:14:21.190289 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:21.190258    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30c897f9-4170-4ebd-bfe0-45c50091de94-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-73f667-2nstj\" (UID: \"30c897f9-4170-4ebd-bfe0-45c50091de94\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-73f667-2nstj"
Oct 30 01:14:21.190465 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:21.190422    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22278f91-d2c8-4808-a572-17290bda0150-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-635e17f5-v97r9\" (UID: \"22278f91-d2c8-4808-a572-17290bda0150\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-635e17f5-v97r9"
Oct 30 01:14:21.192737 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:21.192713    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30c897f9-4170-4ebd-bfe0-45c50091de94-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-73f667-2nstj\" (UID: \"30c897f9-4170-4ebd-bfe0-45c50091de94\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-73f667-2nstj"
Oct 30 01:14:21.192862 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:21.192795    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22278f91-d2c8-4808-a572-17290bda0150-maas-api-xks-ca\") pod \"maas-api-revoke-sub-635e17f5-v97r9\" (UID: \"22278f91-d2c8-4808-a572-17290bda0150\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-635e17f5-v97r9"
Oct 30 01:14:21.214489 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:21.214456    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-chwzj\" (UniqueName: \"kubernetes.io/projected/22278f91-d2c8-4808-a572-17290bda0150-kube-api-access-chwzj\") pod \"maas-api-revoke-sub-635e17f5-v97r9\" (UID: \"22278f91-d2c8-4808-a572-17290bda0150\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-635e17f5-v97r9"
Oct 30 01:14:21.214615 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:21.214486    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gl94r\" (UniqueName: \"kubernetes.io/projected/30c897f9-4170-4ebd-bfe0-45c50091de94-kube-api-access-gl94r\") pod \"maas-api-revoke-keys-e2e-ipp-a-73f667-2nstj\" (UID: \"30c897f9-4170-4ebd-bfe0-45c50091de94\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-73f667-2nstj"
Oct 30 01:14:21.295946 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:21.295858    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-73f667-2nstj"
Oct 30 01:14:21.334686 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:21.334650    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-635e17f5-v97r9"
Oct 30 01:14:21.617918 ip-10-0-128-7 kubenswrapper[5002]: W1030 01:14:21.617878    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30c897f9_4170_4ebd_bfe0_45c50091de94.slice/crio-1624ef66b639fada1318b1a7431518cdc66a612692e850fe8be65dbc370b9fbb WatchSource:0}: Error finding container 1624ef66b639fada1318b1a7431518cdc66a612692e850fe8be65dbc370b9fbb: Status 404 returned error can't find the container with id 1624ef66b639fada1318b1a7431518cdc66a612692e850fe8be65dbc370b9fbb
Oct 30 01:14:21.620058 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:21.620033    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-73f667-2nstj"]
Oct 30 01:14:21.730522 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:21.730480    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-635e17f5-v97r9"]
Oct 30 01:14:21.734739 ip-10-0-128-7 kubenswrapper[5002]: W1030 01:14:21.734707    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22278f91_d2c8_4808_a572_17290bda0150.slice/crio-28a6fdaba8a4fcb0d4ada3263c06f557135f3b82d0d3eefa001c08c484d39382 WatchSource:0}: Error finding container 28a6fdaba8a4fcb0d4ada3263c06f557135f3b82d0d3eefa001c08c484d39382: Status 404 returned error can't find the container with id 28a6fdaba8a4fcb0d4ada3263c06f557135f3b82d0d3eefa001c08c484d39382
Oct 30 01:14:21.796530 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:21.796496    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-635e17f5-v97r9" event={"ID":"22278f91-d2c8-4808-a572-17290bda0150","Type":"ContainerStarted","Data":"28a6fdaba8a4fcb0d4ada3263c06f557135f3b82d0d3eefa001c08c484d39382"}
Oct 30 01:14:21.797918 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:21.797886    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-73f667-2nstj" event={"ID":"30c897f9-4170-4ebd-bfe0-45c50091de94","Type":"ContainerStarted","Data":"1624ef66b639fada1318b1a7431518cdc66a612692e850fe8be65dbc370b9fbb"}
Oct 30 01:14:22.806239 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:22.806197    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-73f667-2nstj" event={"ID":"30c897f9-4170-4ebd-bfe0-45c50091de94","Type":"ContainerStarted","Data":"b083d3a73415f8eeb1babf12286851cdf1b73235d006997fd24cc85d1aa70108"}
Oct 30 01:14:22.807956 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:22.807927    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-635e17f5-v97r9" event={"ID":"22278f91-d2c8-4808-a572-17290bda0150","Type":"ContainerStarted","Data":"1b86f6a042ede8c72ad2e2102fed1823cc856adb9a5bde84160e03eea02ed319"}
Oct 30 01:14:22.855421 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:22.855313    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-73f667-2nstj" podStartSLOduration=2.855294623 podStartE2EDuration="2.855294623s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:14:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:14:22.848811933 +0000 UTC m=+2558.465244812" watchObservedRunningTime="2026-10-30 01:14:22.855294623 +0000 UTC m=+2558.471727505"
Oct 30 01:14:22.895454 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:22.895399    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-635e17f5-v97r9" podStartSLOduration=2.89536123 podStartE2EDuration="2.89536123s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:14:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:14:22.891178368 +0000 UTC m=+2558.507611248" watchObservedRunningTime="2026-10-30 01:14:22.89536123 +0000 UTC m=+2558.511794151"
Oct 30 01:14:23.816951 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:23.816911    5002 generic.go:356] "Generic (PLEG): container finished" podID="22278f91-d2c8-4808-a572-17290bda0150" containerID="1b86f6a042ede8c72ad2e2102fed1823cc856adb9a5bde84160e03eea02ed319" exitCode=0
Oct 30 01:14:23.817485 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:23.817004    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-635e17f5-v97r9" event={"ID":"22278f91-d2c8-4808-a572-17290bda0150","Type":"ContainerDied","Data":"1b86f6a042ede8c72ad2e2102fed1823cc856adb9a5bde84160e03eea02ed319"}
Oct 30 01:14:23.818822 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:23.818795    5002 generic.go:356] "Generic (PLEG): container finished" podID="30c897f9-4170-4ebd-bfe0-45c50091de94" containerID="b083d3a73415f8eeb1babf12286851cdf1b73235d006997fd24cc85d1aa70108" exitCode=0
Oct 30 01:14:23.818946 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:23.818879    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-73f667-2nstj" event={"ID":"30c897f9-4170-4ebd-bfe0-45c50091de94","Type":"ContainerDied","Data":"b083d3a73415f8eeb1babf12286851cdf1b73235d006997fd24cc85d1aa70108"}
Oct 30 01:14:25.049180 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:25.049151    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-73f667-2nstj"
Oct 30 01:14:25.053031 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:25.053009    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-635e17f5-v97r9"
Oct 30 01:14:25.160576 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:25.160482    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30c897f9-4170-4ebd-bfe0-45c50091de94-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30c897f9-4170-4ebd-bfe0-45c50091de94-maas-api-openshift-ca\") pod \"30c897f9-4170-4ebd-bfe0-45c50091de94\" (UID: \"30c897f9-4170-4ebd-bfe0-45c50091de94\") "
Oct 30 01:14:25.160964 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:25.160933    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30c897f9-4170-4ebd-bfe0-45c50091de94-maas-api-openshift-ca" pod "30c897f9-4170-4ebd-bfe0-45c50091de94" (UID: "30c897f9-4170-4ebd-bfe0-45c50091de94"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:14:25.161636 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:25.161603    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/30c897f9-4170-4ebd-bfe0-45c50091de94-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30c897f9-4170-4ebd-bfe0-45c50091de94-maas-api-xks-ca\") pod \"30c897f9-4170-4ebd-bfe0-45c50091de94\" (UID: \"30c897f9-4170-4ebd-bfe0-45c50091de94\") "
Oct 30 01:14:25.161931 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:25.161676    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22278f91-d2c8-4808-a572-17290bda0150-kube-api-access-chwzj\" (UniqueName: \"kubernetes.io/projected/22278f91-d2c8-4808-a572-17290bda0150-kube-api-access-chwzj\") pod \"22278f91-d2c8-4808-a572-17290bda0150\" (UID: \"22278f91-d2c8-4808-a572-17290bda0150\") "
Oct 30 01:14:25.161931 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:25.161715    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22278f91-d2c8-4808-a572-17290bda0150-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22278f91-d2c8-4808-a572-17290bda0150-maas-api-openshift-ca\") pod \"22278f91-d2c8-4808-a572-17290bda0150\" (UID: \"22278f91-d2c8-4808-a572-17290bda0150\") "
Oct 30 01:14:25.161931 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:25.161822    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/22278f91-d2c8-4808-a572-17290bda0150-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22278f91-d2c8-4808-a572-17290bda0150-maas-api-xks-ca\") pod \"22278f91-d2c8-4808-a572-17290bda0150\" (UID: \"22278f91-d2c8-4808-a572-17290bda0150\") "
Oct 30 01:14:25.161931 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:25.161898    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30c897f9-4170-4ebd-bfe0-45c50091de94-kube-api-access-gl94r\" (UniqueName: \"kubernetes.io/projected/30c897f9-4170-4ebd-bfe0-45c50091de94-kube-api-access-gl94r\") pod \"30c897f9-4170-4ebd-bfe0-45c50091de94\" (UID: \"30c897f9-4170-4ebd-bfe0-45c50091de94\") "
Oct 30 01:14:25.169016 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:25.168979    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22278f91-d2c8-4808-a572-17290bda0150-maas-api-openshift-ca" pod "22278f91-d2c8-4808-a572-17290bda0150" (UID: "22278f91-d2c8-4808-a572-17290bda0150"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:14:25.172366 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:25.171563    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22278f91-d2c8-4808-a572-17290bda0150-maas-api-xks-ca" pod "22278f91-d2c8-4808-a572-17290bda0150" (UID: "22278f91-d2c8-4808-a572-17290bda0150"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:14:25.172366 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:25.172074    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30c897f9-4170-4ebd-bfe0-45c50091de94-maas-api-openshift-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:14:25.172366 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:25.172098    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22278f91-d2c8-4808-a572-17290bda0150-maas-api-openshift-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:14:25.172366 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:25.172126    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22278f91-d2c8-4808-a572-17290bda0150-maas-api-xks-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:14:25.172366 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:25.172258    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22278f91-d2c8-4808-a572-17290bda0150-kube-api-access-chwzj" pod "22278f91-d2c8-4808-a572-17290bda0150" (UID: "22278f91-d2c8-4808-a572-17290bda0150"). InnerVolumeSpecName "kube-api-access-chwzj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:14:25.172366 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:25.172344    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30c897f9-4170-4ebd-bfe0-45c50091de94-maas-api-xks-ca" pod "30c897f9-4170-4ebd-bfe0-45c50091de94" (UID: "30c897f9-4170-4ebd-bfe0-45c50091de94"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:14:25.174292 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:25.174267    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30c897f9-4170-4ebd-bfe0-45c50091de94-kube-api-access-gl94r" pod "30c897f9-4170-4ebd-bfe0-45c50091de94" (UID: "30c897f9-4170-4ebd-bfe0-45c50091de94"). InnerVolumeSpecName "kube-api-access-gl94r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:14:25.273669 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:25.273629    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30c897f9-4170-4ebd-bfe0-45c50091de94-maas-api-xks-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:14:25.273669 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:25.273670    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-chwzj\" (UniqueName: \"kubernetes.io/projected/22278f91-d2c8-4808-a572-17290bda0150-kube-api-access-chwzj\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:14:25.273894 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:25.273687    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gl94r\" (UniqueName: \"kubernetes.io/projected/30c897f9-4170-4ebd-bfe0-45c50091de94-kube-api-access-gl94r\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:14:25.835102 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:25.835023    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-73f667-2nstj"
Oct 30 01:14:25.835102 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:25.835055    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-73f667-2nstj" event={"ID":"30c897f9-4170-4ebd-bfe0-45c50091de94","Type":"ContainerDied","Data":"1624ef66b639fada1318b1a7431518cdc66a612692e850fe8be65dbc370b9fbb"}
Oct 30 01:14:25.835340 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:25.835106    5002 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1624ef66b639fada1318b1a7431518cdc66a612692e850fe8be65dbc370b9fbb"
Oct 30 01:14:25.836828 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:25.836806    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-635e17f5-v97r9"
Oct 30 01:14:25.836828 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:25.836814    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-635e17f5-v97r9" event={"ID":"22278f91-d2c8-4808-a572-17290bda0150","Type":"ContainerDied","Data":"28a6fdaba8a4fcb0d4ada3263c06f557135f3b82d0d3eefa001c08c484d39382"}
Oct 30 01:14:25.836975 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:25.836852    5002 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="28a6fdaba8a4fcb0d4ada3263c06f557135f3b82d0d3eefa001c08c484d39382"
Oct 30 01:14:31.534568 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:31.534530    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b3ba534-lzjvc"]
Oct 30 01:14:31.535209 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:31.535193    5002 cpu_manager.go:414] "containerMap: removing container" podUID="22278f91-d2c8-4808-a572-17290bda0150" containerName="revoke-keys"
Oct 30 01:14:31.535209 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:31.535208    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22278f91-d2c8-4808-a572-17290bda0150" containerName="revoke-keys"
Oct 30 01:14:31.535369 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:31.535222    5002 cpu_manager.go:414] "containerMap: removing container" podUID="30c897f9-4170-4ebd-bfe0-45c50091de94" containerName="revoke-keys"
Oct 30 01:14:31.535369 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:31.535228    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30c897f9-4170-4ebd-bfe0-45c50091de94" containerName="revoke-keys"
Oct 30 01:14:31.535369 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:31.535354    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="22278f91-d2c8-4808-a572-17290bda0150" containerName="revoke-keys"
Oct 30 01:14:31.535369 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:31.535364    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="30c897f9-4170-4ebd-bfe0-45c50091de94" containerName="revoke-keys"
Oct 30 01:14:31.540246 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:31.540226    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b3ba534-lzjvc"
Oct 30 01:14:31.561917 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:31.561887    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b3ba534-lzjvc"]
Oct 30 01:14:31.575289 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:31.568751    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:14:31.575289 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:31.568872    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:14:31.575661 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:31.575546    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:14:31.575661 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:31.575569    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5phsh\""
Oct 30 01:14:31.589470 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:31.589393    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/477a7d12-e9bf-4fe4-877b-03f469167592-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3b3ba534-lzjvc\" (UID: \"477a7d12-e9bf-4fe4-877b-03f469167592\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b3ba534-lzjvc"
Oct 30 01:14:31.590052 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:31.590016    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/477a7d12-e9bf-4fe4-877b-03f469167592-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3b3ba534-lzjvc\" (UID: \"477a7d12-e9bf-4fe4-877b-03f469167592\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b3ba534-lzjvc"
Oct 30 01:14:31.590221 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:31.590190    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m55gg\" (UniqueName: \"kubernetes.io/projected/477a7d12-e9bf-4fe4-877b-03f469167592-kube-api-access-m55gg\") pod \"maas-api-revoke-sub-3b3ba534-lzjvc\" (UID: \"477a7d12-e9bf-4fe4-877b-03f469167592\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b3ba534-lzjvc"
Oct 30 01:14:31.738365 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:31.737587    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/477a7d12-e9bf-4fe4-877b-03f469167592-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3b3ba534-lzjvc\" (UID: \"477a7d12-e9bf-4fe4-877b-03f469167592\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b3ba534-lzjvc"
Oct 30 01:14:31.738365 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:31.737708    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m55gg\" (UniqueName: \"kubernetes.io/projected/477a7d12-e9bf-4fe4-877b-03f469167592-kube-api-access-m55gg\") pod \"maas-api-revoke-sub-3b3ba534-lzjvc\" (UID: \"477a7d12-e9bf-4fe4-877b-03f469167592\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b3ba534-lzjvc"
Oct 30 01:14:31.738365 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:31.737821    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/477a7d12-e9bf-4fe4-877b-03f469167592-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3b3ba534-lzjvc\" (UID: \"477a7d12-e9bf-4fe4-877b-03f469167592\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b3ba534-lzjvc"
Oct 30 01:14:31.741470 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:31.739546    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/477a7d12-e9bf-4fe4-877b-03f469167592-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3b3ba534-lzjvc\" (UID: \"477a7d12-e9bf-4fe4-877b-03f469167592\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b3ba534-lzjvc"
Oct 30 01:14:31.757055 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:31.756998    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/477a7d12-e9bf-4fe4-877b-03f469167592-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3b3ba534-lzjvc\" (UID: \"477a7d12-e9bf-4fe4-877b-03f469167592\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b3ba534-lzjvc"
Oct 30 01:14:31.765104 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:31.765074    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m55gg\" (UniqueName: \"kubernetes.io/projected/477a7d12-e9bf-4fe4-877b-03f469167592-kube-api-access-m55gg\") pod \"maas-api-revoke-sub-3b3ba534-lzjvc\" (UID: \"477a7d12-e9bf-4fe4-877b-03f469167592\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b3ba534-lzjvc"
Oct 30 01:14:31.861484 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:31.861399    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b3ba534-lzjvc"
Oct 30 01:14:32.173092 ip-10-0-128-7 kubenswrapper[5002]: W1030 01:14:32.173053    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod477a7d12_e9bf_4fe4_877b_03f469167592.slice/crio-f63ec0ed6ee204886ab7eabc1b1cf0dedfe17272d7c9094c8e96fb9ed86d6708 WatchSource:0}: Error finding container f63ec0ed6ee204886ab7eabc1b1cf0dedfe17272d7c9094c8e96fb9ed86d6708: Status 404 returned error can't find the container with id f63ec0ed6ee204886ab7eabc1b1cf0dedfe17272d7c9094c8e96fb9ed86d6708
Oct 30 01:14:32.176345 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:32.176311    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b3ba534-lzjvc"]
Oct 30 01:14:32.907540 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:32.906526    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b3ba534-lzjvc" event={"ID":"477a7d12-e9bf-4fe4-877b-03f469167592","Type":"ContainerStarted","Data":"214aab239baab3265c67e5caf3e6ff8f8957e72b6d9139c2dbfc0c2ee91a0ec7"}
Oct 30 01:14:32.907540 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:32.906602    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b3ba534-lzjvc" event={"ID":"477a7d12-e9bf-4fe4-877b-03f469167592","Type":"ContainerStarted","Data":"f63ec0ed6ee204886ab7eabc1b1cf0dedfe17272d7c9094c8e96fb9ed86d6708"}
Oct 30 01:14:32.991602 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:32.991404    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b3ba534-lzjvc" podStartSLOduration=1.991357771 podStartE2EDuration="1.991357771s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:14:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:14:32.955206256 +0000 UTC m=+2568.571639135" watchObservedRunningTime="2026-10-30 01:14:32.991357771 +0000 UTC m=+2568.607790651"
Oct 30 01:14:33.916899 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:33.916856    5002 generic.go:356] "Generic (PLEG): container finished" podID="477a7d12-e9bf-4fe4-877b-03f469167592" containerID="214aab239baab3265c67e5caf3e6ff8f8957e72b6d9139c2dbfc0c2ee91a0ec7" exitCode=0
Oct 30 01:14:33.917433 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:33.916936    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b3ba534-lzjvc" event={"ID":"477a7d12-e9bf-4fe4-877b-03f469167592","Type":"ContainerDied","Data":"214aab239baab3265c67e5caf3e6ff8f8957e72b6d9139c2dbfc0c2ee91a0ec7"}
Oct 30 01:14:35.488403 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:35.485903    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b3ba534-lzjvc"
Oct 30 01:14:35.563196 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:35.562448    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/477a7d12-e9bf-4fe4-877b-03f469167592-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/477a7d12-e9bf-4fe4-877b-03f469167592-maas-api-xks-ca\") pod \"477a7d12-e9bf-4fe4-877b-03f469167592\" (UID: \"477a7d12-e9bf-4fe4-877b-03f469167592\") "
Oct 30 01:14:35.563196 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:35.562595    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/477a7d12-e9bf-4fe4-877b-03f469167592-kube-api-access-m55gg\" (UniqueName: \"kubernetes.io/projected/477a7d12-e9bf-4fe4-877b-03f469167592-kube-api-access-m55gg\") pod \"477a7d12-e9bf-4fe4-877b-03f469167592\" (UID: \"477a7d12-e9bf-4fe4-877b-03f469167592\") "
Oct 30 01:14:35.563196 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:35.562692    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/477a7d12-e9bf-4fe4-877b-03f469167592-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/477a7d12-e9bf-4fe4-877b-03f469167592-maas-api-openshift-ca\") pod \"477a7d12-e9bf-4fe4-877b-03f469167592\" (UID: \"477a7d12-e9bf-4fe4-877b-03f469167592\") "
Oct 30 01:14:35.574435 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:35.571957    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/477a7d12-e9bf-4fe4-877b-03f469167592-maas-api-openshift-ca" pod "477a7d12-e9bf-4fe4-877b-03f469167592" (UID: "477a7d12-e9bf-4fe4-877b-03f469167592"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:14:35.589063 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:35.589016    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/477a7d12-e9bf-4fe4-877b-03f469167592-kube-api-access-m55gg" pod "477a7d12-e9bf-4fe4-877b-03f469167592" (UID: "477a7d12-e9bf-4fe4-877b-03f469167592"). InnerVolumeSpecName "kube-api-access-m55gg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:14:35.594201 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:35.594002    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/477a7d12-e9bf-4fe4-877b-03f469167592-maas-api-xks-ca" pod "477a7d12-e9bf-4fe4-877b-03f469167592" (UID: "477a7d12-e9bf-4fe4-877b-03f469167592"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:14:35.598866 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:35.596247    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/477a7d12-e9bf-4fe4-877b-03f469167592-maas-api-openshift-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:14:35.598866 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:35.596276    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/477a7d12-e9bf-4fe4-877b-03f469167592-maas-api-xks-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:14:35.598866 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:35.596291    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m55gg\" (UniqueName: \"kubernetes.io/projected/477a7d12-e9bf-4fe4-877b-03f469167592-kube-api-access-m55gg\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:14:35.867980 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:35.867894    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-73f667-2nstj"]
Oct 30 01:14:35.876596 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:35.876561    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-73f667-2nstj"]
Oct 30 01:14:36.253877 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:36.253835    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b3ba534-lzjvc" event={"ID":"477a7d12-e9bf-4fe4-877b-03f469167592","Type":"ContainerDied","Data":"f63ec0ed6ee204886ab7eabc1b1cf0dedfe17272d7c9094c8e96fb9ed86d6708"}
Oct 30 01:14:36.254142 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:36.254125    5002 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f63ec0ed6ee204886ab7eabc1b1cf0dedfe17272d7c9094c8e96fb9ed86d6708"
Oct 30 01:14:36.254297 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:36.254287    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b3ba534-lzjvc"
Oct 30 01:14:36.498838 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:36.498806    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30c897f9-4170-4ebd-bfe0-45c50091de94" path="/var/lib/kubelet/pods/30c897f9-4170-4ebd-bfe0-45c50091de94/volumes"
Oct 30 01:14:36.500556 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:36.500536    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-55a4c7-dxkzn"]
Oct 30 01:14:36.505247 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:36.505192    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-55a4c7-dxkzn"]
Oct 30 01:14:38.499814 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:38.499784    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e58a0f6-c2b4-4ee8-8e2e-c03030352ec3" path="/var/lib/kubelet/pods/1e58a0f6-c2b4-4ee8-8e2e-c03030352ec3/volumes"
Oct 30 01:14:42.864183 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:14:42.864140    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbjdq" containerName="extractor"
Oct 30 01:14:43.033751 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:43.033708    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-72d8b73b-khp4t"]
Oct 30 01:14:43.044602 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:43.044571    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-72d8b73b-khp4t"]
Oct 30 01:14:43.575172 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:43.575131    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-73f667-openshift-default-7bcfd5c9b9-7fcm7"]
Oct 30 01:14:43.575839 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:43.575799    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-73f667-openshift-default-7bcfd5c9b9-7fcm7" podUID="0eb54f67-9f54-468e-a88c-ba3fbcaa6d2d" containerName="istio-proxy" containerID="cri-o://02308566a5d06b2f98c4dcf4fd883478831cfc487f53021637bf30e0a6b094ac" gracePeriod=30
Oct 30 01:14:44.006415 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:44.003019    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6d44c2-lkp8m"]
Oct 30 01:14:44.006415 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:44.004103    5002 cpu_manager.go:414] "containerMap: removing container" podUID="477a7d12-e9bf-4fe4-877b-03f469167592" containerName="revoke-keys"
Oct 30 01:14:44.006415 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:44.004125    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="477a7d12-e9bf-4fe4-877b-03f469167592" containerName="revoke-keys"
Oct 30 01:14:44.006415 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:44.004341    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="477a7d12-e9bf-4fe4-877b-03f469167592" containerName="revoke-keys"
Oct 30 01:14:44.074129 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:44.074073    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6d44c2-lkp8m"
Oct 30 01:14:44.079265 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:44.079227    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6d44c2-lkp8m"]
Oct 30 01:14:44.083368 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:44.083335    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8f3b80a-prpt5"]
Oct 30 01:14:44.089409 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:44.089358    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8f3b80a-prpt5"]
Oct 30 01:14:44.091892 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:44.091866    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5phsh\""
Oct 30 01:14:44.092709 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:44.092100    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:14:44.098861 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:44.098841    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:14:44.099056 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:44.098917    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:14:44.242397 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:44.242350    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59b4b452-11f0-4885-8b7b-d7a249340941-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-6d44c2-lkp8m\" (UID: \"59b4b452-11f0-4885-8b7b-d7a249340941\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6d44c2-lkp8m"
Oct 30 01:14:44.242599 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:44.242581    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22wmq\" (UniqueName: \"kubernetes.io/projected/59b4b452-11f0-4885-8b7b-d7a249340941-kube-api-access-22wmq\") pod \"maas-api-revoke-keys-e2e-ipp-b-6d44c2-lkp8m\" (UID: \"59b4b452-11f0-4885-8b7b-d7a249340941\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6d44c2-lkp8m"
Oct 30 01:14:44.242699 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:44.242686    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59b4b452-11f0-4885-8b7b-d7a249340941-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-6d44c2-lkp8m\" (UID: \"59b4b452-11f0-4885-8b7b-d7a249340941\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6d44c2-lkp8m"
Oct 30 01:14:44.356743 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:44.356632    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22wmq\" (UniqueName: \"kubernetes.io/projected/59b4b452-11f0-4885-8b7b-d7a249340941-kube-api-access-22wmq\") pod \"maas-api-revoke-keys-e2e-ipp-b-6d44c2-lkp8m\" (UID: \"59b4b452-11f0-4885-8b7b-d7a249340941\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6d44c2-lkp8m"
Oct 30 01:14:44.357150 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:44.357117    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59b4b452-11f0-4885-8b7b-d7a249340941-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-6d44c2-lkp8m\" (UID: \"59b4b452-11f0-4885-8b7b-d7a249340941\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6d44c2-lkp8m"
Oct 30 01:14:44.357477 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:44.357452    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59b4b452-11f0-4885-8b7b-d7a249340941-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-6d44c2-lkp8m\" (UID: \"59b4b452-11f0-4885-8b7b-d7a249340941\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6d44c2-lkp8m"
Oct 30 01:14:44.359174 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:44.359152    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59b4b452-11f0-4885-8b7b-d7a249340941-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-6d44c2-lkp8m\" (UID: \"59b4b452-11f0-4885-8b7b-d7a249340941\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6d44c2-lkp8m"
Oct 30 01:14:44.370037 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:44.370009    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59b4b452-11f0-4885-8b7b-d7a249340941-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-6d44c2-lkp8m\" (UID: \"59b4b452-11f0-4885-8b7b-d7a249340941\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6d44c2-lkp8m"
Oct 30 01:14:44.407288 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:44.407251    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-22wmq\" (UniqueName: \"kubernetes.io/projected/59b4b452-11f0-4885-8b7b-d7a249340941-kube-api-access-22wmq\") pod \"maas-api-revoke-keys-e2e-ipp-b-6d44c2-lkp8m\" (UID: \"59b4b452-11f0-4885-8b7b-d7a249340941\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6d44c2-lkp8m"
Oct 30 01:14:44.562788 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:44.562752    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13cd761c-6990-4b8d-afb1-25fb24baaa8f" path="/var/lib/kubelet/pods/13cd761c-6990-4b8d-afb1-25fb24baaa8f/volumes"
Oct 30 01:14:44.563235 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:44.563219    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae9c53bd-abf9-4166-ab45-f9d77de3f8e6" path="/var/lib/kubelet/pods/ae9c53bd-abf9-4166-ab45-f9d77de3f8e6/volumes"
Oct 30 01:14:44.691223 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:44.691190    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d8b2c6-8ndxw"]
Oct 30 01:14:44.697089 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:44.697049    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d8b2c6-8ndxw"
Oct 30 01:14:44.716450 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:44.716413    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d8b2c6-8ndxw"]
Oct 30 01:14:44.718552 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:44.718516    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5phsh\""
Oct 30 01:14:44.723652 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:44.723610    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6d44c2-lkp8m"
Oct 30 01:14:44.765239 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:44.765193    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a0e3f18-a339-4bc3-8d3b-3b38f3a50182-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-d8b2c6-8ndxw\" (UID: \"1a0e3f18-a339-4bc3-8d3b-3b38f3a50182\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d8b2c6-8ndxw"
Oct 30 01:14:44.765525 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:44.765502    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a0e3f18-a339-4bc3-8d3b-3b38f3a50182-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-d8b2c6-8ndxw\" (UID: \"1a0e3f18-a339-4bc3-8d3b-3b38f3a50182\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d8b2c6-8ndxw"
Oct 30 01:14:44.765644 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:44.765623    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tn9ph\" (UniqueName: \"kubernetes.io/projected/1a0e3f18-a339-4bc3-8d3b-3b38f3a50182-kube-api-access-tn9ph\") pod \"maas-api-revoke-keys-e2e-ipp-b-d8b2c6-8ndxw\" (UID: \"1a0e3f18-a339-4bc3-8d3b-3b38f3a50182\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d8b2c6-8ndxw"
Oct 30 01:14:44.867862 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:44.867261    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a0e3f18-a339-4bc3-8d3b-3b38f3a50182-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-d8b2c6-8ndxw\" (UID: \"1a0e3f18-a339-4bc3-8d3b-3b38f3a50182\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d8b2c6-8ndxw"
Oct 30 01:14:44.867862 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:44.867455    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a0e3f18-a339-4bc3-8d3b-3b38f3a50182-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-d8b2c6-8ndxw\" (UID: \"1a0e3f18-a339-4bc3-8d3b-3b38f3a50182\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d8b2c6-8ndxw"
Oct 30 01:14:44.867862 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:44.867516    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tn9ph\" (UniqueName: \"kubernetes.io/projected/1a0e3f18-a339-4bc3-8d3b-3b38f3a50182-kube-api-access-tn9ph\") pod \"maas-api-revoke-keys-e2e-ipp-b-d8b2c6-8ndxw\" (UID: \"1a0e3f18-a339-4bc3-8d3b-3b38f3a50182\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d8b2c6-8ndxw"
Oct 30 01:14:44.868566 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:44.868538    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a0e3f18-a339-4bc3-8d3b-3b38f3a50182-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-d8b2c6-8ndxw\" (UID: \"1a0e3f18-a339-4bc3-8d3b-3b38f3a50182\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d8b2c6-8ndxw"
Oct 30 01:14:44.870776 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:44.870747    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a0e3f18-a339-4bc3-8d3b-3b38f3a50182-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-d8b2c6-8ndxw\" (UID: \"1a0e3f18-a339-4bc3-8d3b-3b38f3a50182\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d8b2c6-8ndxw"
Oct 30 01:14:44.900568 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:44.900538    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tn9ph\" (UniqueName: \"kubernetes.io/projected/1a0e3f18-a339-4bc3-8d3b-3b38f3a50182-kube-api-access-tn9ph\") pod \"maas-api-revoke-keys-e2e-ipp-b-d8b2c6-8ndxw\" (UID: \"1a0e3f18-a339-4bc3-8d3b-3b38f3a50182\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d8b2c6-8ndxw"
Oct 30 01:14:45.015655 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:45.015520    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d8b2c6-8ndxw"
Oct 30 01:14:45.048777 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:45.048744    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6d44c2-lkp8m"]
Oct 30 01:14:45.329969 ip-10-0-128-7 kubenswrapper[5002]: W1030 01:14:45.329927    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a0e3f18_a339_4bc3_8d3b_3b38f3a50182.slice/crio-b4cdab76eff770fb2fce34f58fd2267924fd06b794b957ddfb1d10a5a97a9ba8 WatchSource:0}: Error finding container b4cdab76eff770fb2fce34f58fd2267924fd06b794b957ddfb1d10a5a97a9ba8: Status 404 returned error can't find the container with id b4cdab76eff770fb2fce34f58fd2267924fd06b794b957ddfb1d10a5a97a9ba8
Oct 30 01:14:45.332009 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:45.331977    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d8b2c6-8ndxw"]
Oct 30 01:14:45.440013 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:45.439973    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d8b2c6-8ndxw" event={"ID":"1a0e3f18-a339-4bc3-8d3b-3b38f3a50182","Type":"ContainerStarted","Data":"b4cdab76eff770fb2fce34f58fd2267924fd06b794b957ddfb1d10a5a97a9ba8"}
Oct 30 01:14:45.441803 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:45.441768    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6d44c2-lkp8m" event={"ID":"59b4b452-11f0-4885-8b7b-d7a249340941","Type":"ContainerStarted","Data":"50e645e97f8ac661058b97a771acdd1d180675557263d0ee9906fc24472fead8"}
Oct 30 01:14:46.451170 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:46.451131    5002 generic.go:356] "Generic (PLEG): container finished" podID="59b4b452-11f0-4885-8b7b-d7a249340941" containerID="bd67c8dfee363796e43f186d9a92e2678340c2074e57cf02667b49a4098641b2" exitCode=0
Oct 30 01:14:46.451642 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:46.451212    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6d44c2-lkp8m" event={"ID":"59b4b452-11f0-4885-8b7b-d7a249340941","Type":"ContainerDied","Data":"bd67c8dfee363796e43f186d9a92e2678340c2074e57cf02667b49a4098641b2"}
Oct 30 01:14:46.453040 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:46.453015    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d8b2c6-8ndxw" event={"ID":"1a0e3f18-a339-4bc3-8d3b-3b38f3a50182","Type":"ContainerStarted","Data":"89d1fba3da05502b8b86752b306d87bf56e3e9a31b826b611ff0a10fc08848b2"}
Oct 30 01:14:46.539561 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:46.539509    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d8b2c6-8ndxw" podStartSLOduration=2.539493286 podStartE2EDuration="2.539493286s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:14:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:14:46.536791 +0000 UTC m=+2582.153223885" watchObservedRunningTime="2026-10-30 01:14:46.539493286 +0000 UTC m=+2582.155926165"
Oct 30 01:14:47.462018 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:47.461973    5002 generic.go:356] "Generic (PLEG): container finished" podID="1a0e3f18-a339-4bc3-8d3b-3b38f3a50182" containerID="89d1fba3da05502b8b86752b306d87bf56e3e9a31b826b611ff0a10fc08848b2" exitCode=0
Oct 30 01:14:47.462511 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:47.462329    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d8b2c6-8ndxw" event={"ID":"1a0e3f18-a339-4bc3-8d3b-3b38f3a50182","Type":"ContainerDied","Data":"89d1fba3da05502b8b86752b306d87bf56e3e9a31b826b611ff0a10fc08848b2"}
Oct 30 01:14:47.660916 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:47.660880    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6d44c2-lkp8m"
Oct 30 01:14:47.816488 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:47.814974    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/59b4b452-11f0-4885-8b7b-d7a249340941-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59b4b452-11f0-4885-8b7b-d7a249340941-maas-api-xks-ca\") pod \"59b4b452-11f0-4885-8b7b-d7a249340941\" (UID: \"59b4b452-11f0-4885-8b7b-d7a249340941\") "
Oct 30 01:14:47.816488 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:47.815100    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59b4b452-11f0-4885-8b7b-d7a249340941-kube-api-access-22wmq\" (UniqueName: \"kubernetes.io/projected/59b4b452-11f0-4885-8b7b-d7a249340941-kube-api-access-22wmq\") pod \"59b4b452-11f0-4885-8b7b-d7a249340941\" (UID: \"59b4b452-11f0-4885-8b7b-d7a249340941\") "
Oct 30 01:14:47.816488 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:47.815136    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59b4b452-11f0-4885-8b7b-d7a249340941-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59b4b452-11f0-4885-8b7b-d7a249340941-maas-api-openshift-ca\") pod \"59b4b452-11f0-4885-8b7b-d7a249340941\" (UID: \"59b4b452-11f0-4885-8b7b-d7a249340941\") "
Oct 30 01:14:47.839023 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:47.831103    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59b4b452-11f0-4885-8b7b-d7a249340941-maas-api-openshift-ca" pod "59b4b452-11f0-4885-8b7b-d7a249340941" (UID: "59b4b452-11f0-4885-8b7b-d7a249340941"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:14:47.853369 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:47.844401    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59b4b452-11f0-4885-8b7b-d7a249340941-maas-api-openshift-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:14:47.869442 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:47.866581    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59b4b452-11f0-4885-8b7b-d7a249340941-kube-api-access-22wmq" pod "59b4b452-11f0-4885-8b7b-d7a249340941" (UID: "59b4b452-11f0-4885-8b7b-d7a249340941"). InnerVolumeSpecName "kube-api-access-22wmq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:14:47.869775 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:47.869725    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59b4b452-11f0-4885-8b7b-d7a249340941-maas-api-xks-ca" pod "59b4b452-11f0-4885-8b7b-d7a249340941" (UID: "59b4b452-11f0-4885-8b7b-d7a249340941"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:14:47.946425 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:47.946355    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59b4b452-11f0-4885-8b7b-d7a249340941-maas-api-xks-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:14:47.946425 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:47.946426    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-22wmq\" (UniqueName: \"kubernetes.io/projected/59b4b452-11f0-4885-8b7b-d7a249340941-kube-api-access-22wmq\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:14:48.474331 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:48.474302    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6d44c2-lkp8m"
Oct 30 01:14:48.474868 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:48.474341    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6d44c2-lkp8m" event={"ID":"59b4b452-11f0-4885-8b7b-d7a249340941","Type":"ContainerDied","Data":"50e645e97f8ac661058b97a771acdd1d180675557263d0ee9906fc24472fead8"}
Oct 30 01:14:48.474868 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:48.474430    5002 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="50e645e97f8ac661058b97a771acdd1d180675557263d0ee9906fc24472fead8"
Oct 30 01:14:48.657179 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:48.657152    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d8b2c6-8ndxw"
Oct 30 01:14:48.659029 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:48.659000    5002 scope.go:122] "RemoveContainer" containerID="29794e77725969ea6dc3724d69f11a205055df813994e45e491a58d2f5090d2e"
Oct 30 01:14:48.672746 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:48.672715    5002 scope.go:122] "RemoveContainer" containerID="a5969443ca43eeb7d91cc30253e980fb62e7bbd5c11fb6bc450fb2b21a61e0df"
Oct 30 01:14:48.682829 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:48.682803    5002 scope.go:122] "RemoveContainer" containerID="f29efdc56bf33c25c25df51d44201cf57e77b2c01aeadce7f002e07ac4607714"
Oct 30 01:14:48.692369 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:48.692345    5002 scope.go:122] "RemoveContainer" containerID="8155c960ca17cd303f695344efec7f795ab6964118a072508b07369862bb29f7"
Oct 30 01:14:48.773890 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:48.773855    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a0e3f18-a339-4bc3-8d3b-3b38f3a50182-kube-api-access-tn9ph\" (UniqueName: \"kubernetes.io/projected/1a0e3f18-a339-4bc3-8d3b-3b38f3a50182-kube-api-access-tn9ph\") pod \"1a0e3f18-a339-4bc3-8d3b-3b38f3a50182\" (UID: \"1a0e3f18-a339-4bc3-8d3b-3b38f3a50182\") "
Oct 30 01:14:48.774151 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:48.774135    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a0e3f18-a339-4bc3-8d3b-3b38f3a50182-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a0e3f18-a339-4bc3-8d3b-3b38f3a50182-maas-api-openshift-ca\") pod \"1a0e3f18-a339-4bc3-8d3b-3b38f3a50182\" (UID: \"1a0e3f18-a339-4bc3-8d3b-3b38f3a50182\") "
Oct 30 01:14:48.774204 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:48.774174    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1a0e3f18-a339-4bc3-8d3b-3b38f3a50182-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a0e3f18-a339-4bc3-8d3b-3b38f3a50182-maas-api-xks-ca\") pod \"1a0e3f18-a339-4bc3-8d3b-3b38f3a50182\" (UID: \"1a0e3f18-a339-4bc3-8d3b-3b38f3a50182\") "
Oct 30 01:14:48.774598 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:48.774564    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a0e3f18-a339-4bc3-8d3b-3b38f3a50182-maas-api-openshift-ca" pod "1a0e3f18-a339-4bc3-8d3b-3b38f3a50182" (UID: "1a0e3f18-a339-4bc3-8d3b-3b38f3a50182"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:14:48.774836 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:48.774813    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a0e3f18-a339-4bc3-8d3b-3b38f3a50182-maas-api-openshift-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:14:48.776498 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:48.776473    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a0e3f18-a339-4bc3-8d3b-3b38f3a50182-maas-api-xks-ca" pod "1a0e3f18-a339-4bc3-8d3b-3b38f3a50182" (UID: "1a0e3f18-a339-4bc3-8d3b-3b38f3a50182"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:14:48.776636 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:48.776611    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a0e3f18-a339-4bc3-8d3b-3b38f3a50182-kube-api-access-tn9ph" pod "1a0e3f18-a339-4bc3-8d3b-3b38f3a50182" (UID: "1a0e3f18-a339-4bc3-8d3b-3b38f3a50182"). InnerVolumeSpecName "kube-api-access-tn9ph". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:14:48.802285 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:48.802256    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-wjg5k"]
Oct 30 01:14:48.802485 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:48.802463    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-wjg5k" podUID="c9821719-d280-4bc0-a68b-9ae8def2955f" containerName="istio-proxy" containerID="cri-o://415acff5a5892a0fd490333e00beffe6cc9790c80f032f230b1c475d35735f0e" gracePeriod=30
Oct 30 01:14:48.876010 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:48.875978    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a0e3f18-a339-4bc3-8d3b-3b38f3a50182-maas-api-xks-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:14:48.876010 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:48.876010    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tn9ph\" (UniqueName: \"kubernetes.io/projected/1a0e3f18-a339-4bc3-8d3b-3b38f3a50182-kube-api-access-tn9ph\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:14:49.486854 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:49.486573    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d8b2c6-8ndxw"
Oct 30 01:14:49.486854 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:49.486601    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d8b2c6-8ndxw" event={"ID":"1a0e3f18-a339-4bc3-8d3b-3b38f3a50182","Type":"ContainerDied","Data":"b4cdab76eff770fb2fce34f58fd2267924fd06b794b957ddfb1d10a5a97a9ba8"}
Oct 30 01:14:49.486854 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:49.486673    5002 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b4cdab76eff770fb2fce34f58fd2267924fd06b794b957ddfb1d10a5a97a9ba8"
Oct 30 01:14:49.492361 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:49.492330    5002 generic.go:356] "Generic (PLEG): container finished" podID="0eb54f67-9f54-468e-a88c-ba3fbcaa6d2d" containerID="02308566a5d06b2f98c4dcf4fd883478831cfc487f53021637bf30e0a6b094ac" exitCode=0
Oct 30 01:14:49.492569 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:49.492424    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-73f667-openshift-default-7bcfd5c9b9-7fcm7" event={"ID":"0eb54f67-9f54-468e-a88c-ba3fbcaa6d2d","Type":"ContainerDied","Data":"02308566a5d06b2f98c4dcf4fd883478831cfc487f53021637bf30e0a6b094ac"}
Oct 30 01:14:49.492569 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:49.492471    5002 scope.go:122] "RemoveContainer" containerID="c7ea4c5c15a25ce02e96b7fcc3c1265f8042d3ca9174740c5341b3b07aa2fdd1"
Oct 30 01:14:49.715277 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:49.715208    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-73f667-openshift-default-7bcfd5c9b9-7fcm7"
Oct 30 01:14:49.789899 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:49.789863    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0eb54f67-9f54-468e-a88c-ba3fbcaa6d2d-kube-api-access-5m967\" (UniqueName: \"kubernetes.io/projected/0eb54f67-9f54-468e-a88c-ba3fbcaa6d2d-kube-api-access-5m967\") pod \"0eb54f67-9f54-468e-a88c-ba3fbcaa6d2d\" (UID: \"0eb54f67-9f54-468e-a88c-ba3fbcaa6d2d\") "
Oct 30 01:14:49.790134 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:49.790030    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0eb54f67-9f54-468e-a88c-ba3fbcaa6d2d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0eb54f67-9f54-468e-a88c-ba3fbcaa6d2d-istiod-ca-cert\") pod \"0eb54f67-9f54-468e-a88c-ba3fbcaa6d2d\" (UID: \"0eb54f67-9f54-468e-a88c-ba3fbcaa6d2d\") "
Oct 30 01:14:49.790134 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:49.790070    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0eb54f67-9f54-468e-a88c-ba3fbcaa6d2d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0eb54f67-9f54-468e-a88c-ba3fbcaa6d2d-istio-envoy\") pod \"0eb54f67-9f54-468e-a88c-ba3fbcaa6d2d\" (UID: \"0eb54f67-9f54-468e-a88c-ba3fbcaa6d2d\") "
Oct 30 01:14:49.790134 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:49.790125    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0eb54f67-9f54-468e-a88c-ba3fbcaa6d2d-istio-token\" (UniqueName: \"kubernetes.io/projected/0eb54f67-9f54-468e-a88c-ba3fbcaa6d2d-istio-token\") pod \"0eb54f67-9f54-468e-a88c-ba3fbcaa6d2d\" (UID: \"0eb54f67-9f54-468e-a88c-ba3fbcaa6d2d\") "
Oct 30 01:14:49.790311 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:49.790154    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0eb54f67-9f54-468e-a88c-ba3fbcaa6d2d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0eb54f67-9f54-468e-a88c-ba3fbcaa6d2d-workload-socket\") pod \"0eb54f67-9f54-468e-a88c-ba3fbcaa6d2d\" (UID: \"0eb54f67-9f54-468e-a88c-ba3fbcaa6d2d\") "
Oct 30 01:14:49.790311 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:49.790186    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0eb54f67-9f54-468e-a88c-ba3fbcaa6d2d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0eb54f67-9f54-468e-a88c-ba3fbcaa6d2d-workload-certs\") pod \"0eb54f67-9f54-468e-a88c-ba3fbcaa6d2d\" (UID: \"0eb54f67-9f54-468e-a88c-ba3fbcaa6d2d\") "
Oct 30 01:14:49.790311 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:49.790211    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0eb54f67-9f54-468e-a88c-ba3fbcaa6d2d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0eb54f67-9f54-468e-a88c-ba3fbcaa6d2d-credential-socket\") pod \"0eb54f67-9f54-468e-a88c-ba3fbcaa6d2d\" (UID: \"0eb54f67-9f54-468e-a88c-ba3fbcaa6d2d\") "
Oct 30 01:14:49.790311 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:49.790308    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0eb54f67-9f54-468e-a88c-ba3fbcaa6d2d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0eb54f67-9f54-468e-a88c-ba3fbcaa6d2d-istio-data\") pod \"0eb54f67-9f54-468e-a88c-ba3fbcaa6d2d\" (UID: \"0eb54f67-9f54-468e-a88c-ba3fbcaa6d2d\") "
Oct 30 01:14:49.790587 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:49.790478    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0eb54f67-9f54-468e-a88c-ba3fbcaa6d2d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0eb54f67-9f54-468e-a88c-ba3fbcaa6d2d-istio-podinfo\") pod \"0eb54f67-9f54-468e-a88c-ba3fbcaa6d2d\" (UID: \"0eb54f67-9f54-468e-a88c-ba3fbcaa6d2d\") "
Oct 30 01:14:49.791569 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:49.791193    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0eb54f67-9f54-468e-a88c-ba3fbcaa6d2d-credential-socket" pod "0eb54f67-9f54-468e-a88c-ba3fbcaa6d2d" (UID: "0eb54f67-9f54-468e-a88c-ba3fbcaa6d2d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:14:49.791569 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:49.791407    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0eb54f67-9f54-468e-a88c-ba3fbcaa6d2d-workload-socket" pod "0eb54f67-9f54-468e-a88c-ba3fbcaa6d2d" (UID: "0eb54f67-9f54-468e-a88c-ba3fbcaa6d2d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:14:49.791569 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:49.791494    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0eb54f67-9f54-468e-a88c-ba3fbcaa6d2d-workload-certs" pod "0eb54f67-9f54-468e-a88c-ba3fbcaa6d2d" (UID: "0eb54f67-9f54-468e-a88c-ba3fbcaa6d2d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:14:49.791569 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:49.791547    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0eb54f67-9f54-468e-a88c-ba3fbcaa6d2d-istio-data" pod "0eb54f67-9f54-468e-a88c-ba3fbcaa6d2d" (UID: "0eb54f67-9f54-468e-a88c-ba3fbcaa6d2d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:14:49.792290 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:49.792269    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0eb54f67-9f54-468e-a88c-ba3fbcaa6d2d-istiod-ca-cert" pod "0eb54f67-9f54-468e-a88c-ba3fbcaa6d2d" (UID: "0eb54f67-9f54-468e-a88c-ba3fbcaa6d2d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:14:49.794467 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:49.794432    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0eb54f67-9f54-468e-a88c-ba3fbcaa6d2d-istio-envoy" pod "0eb54f67-9f54-468e-a88c-ba3fbcaa6d2d" (UID: "0eb54f67-9f54-468e-a88c-ba3fbcaa6d2d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:14:49.799450 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:49.795442    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0eb54f67-9f54-468e-a88c-ba3fbcaa6d2d-kube-api-access-5m967" pod "0eb54f67-9f54-468e-a88c-ba3fbcaa6d2d" (UID: "0eb54f67-9f54-468e-a88c-ba3fbcaa6d2d"). InnerVolumeSpecName "kube-api-access-5m967". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:14:49.799450 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:49.798815    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0eb54f67-9f54-468e-a88c-ba3fbcaa6d2d-istio-podinfo" pod "0eb54f67-9f54-468e-a88c-ba3fbcaa6d2d" (UID: "0eb54f67-9f54-468e-a88c-ba3fbcaa6d2d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:14:49.799450 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:49.798938    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0eb54f67-9f54-468e-a88c-ba3fbcaa6d2d-istio-token" pod "0eb54f67-9f54-468e-a88c-ba3fbcaa6d2d" (UID: "0eb54f67-9f54-468e-a88c-ba3fbcaa6d2d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:14:49.892924 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:49.892878    5002 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0eb54f67-9f54-468e-a88c-ba3fbcaa6d2d-istio-podinfo\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:14:49.892924 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:49.892912    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5m967\" (UniqueName: \"kubernetes.io/projected/0eb54f67-9f54-468e-a88c-ba3fbcaa6d2d-kube-api-access-5m967\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:14:49.892924 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:49.892925    5002 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0eb54f67-9f54-468e-a88c-ba3fbcaa6d2d-istiod-ca-cert\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:14:49.893256 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:49.892937    5002 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0eb54f67-9f54-468e-a88c-ba3fbcaa6d2d-istio-envoy\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:14:49.893256 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:49.892945    5002 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0eb54f67-9f54-468e-a88c-ba3fbcaa6d2d-istio-token\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:14:49.893256 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:49.892958    5002 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0eb54f67-9f54-468e-a88c-ba3fbcaa6d2d-workload-socket\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:14:49.893256 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:49.892968    5002 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0eb54f67-9f54-468e-a88c-ba3fbcaa6d2d-workload-certs\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:14:49.893256 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:49.892977    5002 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0eb54f67-9f54-468e-a88c-ba3fbcaa6d2d-credential-socket\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:14:49.893256 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:49.892984    5002 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0eb54f67-9f54-468e-a88c-ba3fbcaa6d2d-istio-data\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:14:50.503647 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:50.503612    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-73f667-openshift-default-7bcfd5c9b9-7fcm7"
Oct 30 01:14:50.504161 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:50.503647    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-73f667-openshift-default-7bcfd5c9b9-7fcm7" event={"ID":"0eb54f67-9f54-468e-a88c-ba3fbcaa6d2d","Type":"ContainerDied","Data":"7829bb756c04b45ee557577cd7a23125b209ab7dbf965046e1d5ec87ff3fdddc"}
Oct 30 01:14:50.504161 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:50.503714    5002 scope.go:122] "RemoveContainer" containerID="02308566a5d06b2f98c4dcf4fd883478831cfc487f53021637bf30e0a6b094ac"
Oct 30 01:14:50.581323 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:50.581285    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-73f667-openshift-default-7bcfd5c9b9-7fcm7"]
Oct 30 01:14:50.592852 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:50.592818    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-73f667-openshift-default-7bcfd5c9b9-7fcm7"]
Oct 30 01:14:52.501567 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:52.501534    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0eb54f67-9f54-468e-a88c-ba3fbcaa6d2d" path="/var/lib/kubelet/pods/0eb54f67-9f54-468e-a88c-ba3fbcaa6d2d/volumes"
Oct 30 01:14:53.032261 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:53.032222    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7edda4b3-jnr46"]
Oct 30 01:14:53.044148 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:53.044114    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7edda4b3-jnr46"]
Oct 30 01:14:54.042131 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:54.042071    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95e2da69-n7tqv"]
Oct 30 01:14:54.055018 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:54.054970    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95e2da69-n7tqv"]
Oct 30 01:14:54.131872 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:54.131846    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-wjg5k"
Oct 30 01:14:54.267265 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:54.267181    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9821719-d280-4bc0-a68b-9ae8def2955f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9821719-d280-4bc0-a68b-9ae8def2955f-workload-certs\") pod \"c9821719-d280-4bc0-a68b-9ae8def2955f\" (UID: \"c9821719-d280-4bc0-a68b-9ae8def2955f\") "
Oct 30 01:14:54.267265 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:54.267268    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9821719-d280-4bc0-a68b-9ae8def2955f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9821719-d280-4bc0-a68b-9ae8def2955f-istio-envoy\") pod \"c9821719-d280-4bc0-a68b-9ae8def2955f\" (UID: \"c9821719-d280-4bc0-a68b-9ae8def2955f\") "
Oct 30 01:14:54.267610 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:54.267366    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9821719-d280-4bc0-a68b-9ae8def2955f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9821719-d280-4bc0-a68b-9ae8def2955f-credential-socket\") pod \"c9821719-d280-4bc0-a68b-9ae8def2955f\" (UID: \"c9821719-d280-4bc0-a68b-9ae8def2955f\") "
Oct 30 01:14:54.267610 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:54.267457    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c9821719-d280-4bc0-a68b-9ae8def2955f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9821719-d280-4bc0-a68b-9ae8def2955f-istio-podinfo\") pod \"c9821719-d280-4bc0-a68b-9ae8def2955f\" (UID: \"c9821719-d280-4bc0-a68b-9ae8def2955f\") "
Oct 30 01:14:54.267610 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:54.267537    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9821719-d280-4bc0-a68b-9ae8def2955f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9821719-d280-4bc0-a68b-9ae8def2955f-istio-data\") pod \"c9821719-d280-4bc0-a68b-9ae8def2955f\" (UID: \"c9821719-d280-4bc0-a68b-9ae8def2955f\") "
Oct 30 01:14:54.267823 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:54.267624    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9821719-d280-4bc0-a68b-9ae8def2955f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9821719-d280-4bc0-a68b-9ae8def2955f-istiod-ca-cert\") pod \"c9821719-d280-4bc0-a68b-9ae8def2955f\" (UID: \"c9821719-d280-4bc0-a68b-9ae8def2955f\") "
Oct 30 01:14:54.267823 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:54.267653    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9821719-d280-4bc0-a68b-9ae8def2955f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9821719-d280-4bc0-a68b-9ae8def2955f-workload-socket\") pod \"c9821719-d280-4bc0-a68b-9ae8def2955f\" (UID: \"c9821719-d280-4bc0-a68b-9ae8def2955f\") "
Oct 30 01:14:54.267823 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:54.267716    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9821719-d280-4bc0-a68b-9ae8def2955f-istio-token\" (UniqueName: \"kubernetes.io/projected/c9821719-d280-4bc0-a68b-9ae8def2955f-istio-token\") pod \"c9821719-d280-4bc0-a68b-9ae8def2955f\" (UID: \"c9821719-d280-4bc0-a68b-9ae8def2955f\") "
Oct 30 01:14:54.267823 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:54.267753    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9821719-d280-4bc0-a68b-9ae8def2955f-kube-api-access-f4m7v\" (UniqueName: \"kubernetes.io/projected/c9821719-d280-4bc0-a68b-9ae8def2955f-kube-api-access-f4m7v\") pod \"c9821719-d280-4bc0-a68b-9ae8def2955f\" (UID: \"c9821719-d280-4bc0-a68b-9ae8def2955f\") "
Oct 30 01:14:54.268030 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:54.267959    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9821719-d280-4bc0-a68b-9ae8def2955f-istio-data" pod "c9821719-d280-4bc0-a68b-9ae8def2955f" (UID: "c9821719-d280-4bc0-a68b-9ae8def2955f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:14:54.268120 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:54.268092    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9821719-d280-4bc0-a68b-9ae8def2955f-istiod-ca-cert" pod "c9821719-d280-4bc0-a68b-9ae8def2955f" (UID: "c9821719-d280-4bc0-a68b-9ae8def2955f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:14:54.268355 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:54.268330    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9821719-d280-4bc0-a68b-9ae8def2955f-workload-socket" pod "c9821719-d280-4bc0-a68b-9ae8def2955f" (UID: "c9821719-d280-4bc0-a68b-9ae8def2955f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:14:54.268591 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:54.268568    5002 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9821719-d280-4bc0-a68b-9ae8def2955f-istio-data\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:14:54.268693 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:54.268597    5002 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9821719-d280-4bc0-a68b-9ae8def2955f-istiod-ca-cert\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:14:54.268693 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:54.268612    5002 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9821719-d280-4bc0-a68b-9ae8def2955f-workload-socket\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:14:54.269138 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:54.269113    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9821719-d280-4bc0-a68b-9ae8def2955f-credential-socket" pod "c9821719-d280-4bc0-a68b-9ae8def2955f" (UID: "c9821719-d280-4bc0-a68b-9ae8def2955f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:14:54.270251 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:54.270197    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9821719-d280-4bc0-a68b-9ae8def2955f-workload-certs" pod "c9821719-d280-4bc0-a68b-9ae8def2955f" (UID: "c9821719-d280-4bc0-a68b-9ae8def2955f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:14:54.271615 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:54.271515    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9821719-d280-4bc0-a68b-9ae8def2955f-istio-token" pod "c9821719-d280-4bc0-a68b-9ae8def2955f" (UID: "c9821719-d280-4bc0-a68b-9ae8def2955f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:14:54.272052 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:54.272021    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9821719-d280-4bc0-a68b-9ae8def2955f-istio-envoy" pod "c9821719-d280-4bc0-a68b-9ae8def2955f" (UID: "c9821719-d280-4bc0-a68b-9ae8def2955f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:14:54.272631 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:54.272354    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9821719-d280-4bc0-a68b-9ae8def2955f-kube-api-access-f4m7v" pod "c9821719-d280-4bc0-a68b-9ae8def2955f" (UID: "c9821719-d280-4bc0-a68b-9ae8def2955f"). InnerVolumeSpecName "kube-api-access-f4m7v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:14:54.273210 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:54.273187    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c9821719-d280-4bc0-a68b-9ae8def2955f-istio-podinfo" pod "c9821719-d280-4bc0-a68b-9ae8def2955f" (UID: "c9821719-d280-4bc0-a68b-9ae8def2955f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:14:54.373230 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:54.373130    5002 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9821719-d280-4bc0-a68b-9ae8def2955f-workload-certs\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:14:54.373230 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:54.373170    5002 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9821719-d280-4bc0-a68b-9ae8def2955f-istio-envoy\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:14:54.373230 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:54.373182    5002 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9821719-d280-4bc0-a68b-9ae8def2955f-credential-socket\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:14:54.373230 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:54.373195    5002 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9821719-d280-4bc0-a68b-9ae8def2955f-istio-podinfo\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:14:54.373230 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:54.373207    5002 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9821719-d280-4bc0-a68b-9ae8def2955f-istio-token\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:14:54.373230 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:54.373219    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f4m7v\" (UniqueName: \"kubernetes.io/projected/c9821719-d280-4bc0-a68b-9ae8def2955f-kube-api-access-f4m7v\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:14:54.499205 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:54.499165    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25e6159a-609a-483f-b679-3e64db06178c" path="/var/lib/kubelet/pods/25e6159a-609a-483f-b679-3e64db06178c/volumes"
Oct 30 01:14:54.499774 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:54.499756    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec788b52-0c21-48a4-b329-27ce6a65433a" path="/var/lib/kubelet/pods/ec788b52-0c21-48a4-b329-27ce6a65433a/volumes"
Oct 30 01:14:54.536102 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:54.536066    5002 generic.go:356] "Generic (PLEG): container finished" podID="c9821719-d280-4bc0-a68b-9ae8def2955f" containerID="415acff5a5892a0fd490333e00beffe6cc9790c80f032f230b1c475d35735f0e" exitCode=0
Oct 30 01:14:54.536285 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:54.536145    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-wjg5k"
Oct 30 01:14:54.536285 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:54.536156    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-wjg5k" event={"ID":"c9821719-d280-4bc0-a68b-9ae8def2955f","Type":"ContainerDied","Data":"415acff5a5892a0fd490333e00beffe6cc9790c80f032f230b1c475d35735f0e"}
Oct 30 01:14:54.536285 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:54.536215    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-wjg5k" event={"ID":"c9821719-d280-4bc0-a68b-9ae8def2955f","Type":"ContainerDied","Data":"eb53ef389b0116b6e54e58b159963fbf155dbd45f4a24d1e26ffa3374df3001a"}
Oct 30 01:14:54.536285 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:54.536241    5002 scope.go:122] "RemoveContainer" containerID="415acff5a5892a0fd490333e00beffe6cc9790c80f032f230b1c475d35735f0e"
Oct 30 01:14:54.548641 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:54.548620    5002 scope.go:122] "RemoveContainer" containerID="415acff5a5892a0fd490333e00beffe6cc9790c80f032f230b1c475d35735f0e"
Oct 30 01:14:54.548951 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:14:54.548928    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"415acff5a5892a0fd490333e00beffe6cc9790c80f032f230b1c475d35735f0e\": container with ID starting with 415acff5a5892a0fd490333e00beffe6cc9790c80f032f230b1c475d35735f0e not found: ID does not exist" containerID="415acff5a5892a0fd490333e00beffe6cc9790c80f032f230b1c475d35735f0e"
Oct 30 01:14:54.549023 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:54.548963    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"415acff5a5892a0fd490333e00beffe6cc9790c80f032f230b1c475d35735f0e"} err="failed to get container status \"415acff5a5892a0fd490333e00beffe6cc9790c80f032f230b1c475d35735f0e\": rpc error: code = NotFound desc = could not find container \"415acff5a5892a0fd490333e00beffe6cc9790c80f032f230b1c475d35735f0e\": container with ID starting with 415acff5a5892a0fd490333e00beffe6cc9790c80f032f230b1c475d35735f0e not found: ID does not exist"
Oct 30 01:14:54.595237 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:54.595193    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-wjg5k"]
Oct 30 01:14:54.608553 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:54.608513    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-wjg5k"]
Oct 30 01:14:56.499625 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:56.499586    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9821719-d280-4bc0-a68b-9ae8def2955f" path="/var/lib/kubelet/pods/c9821719-d280-4bc0-a68b-9ae8def2955f/volumes"
Oct 30 01:14:59.172647 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:59.172598    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-5f387c-openshift-default-5ff776bc59-bdp4r"]
Oct 30 01:14:59.173054 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:14:59.172938    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-5f387c-openshift-default-5ff776bc59-bdp4r" podUID="9bb1708e-878e-461f-9294-265673c31b04" containerName="istio-proxy" containerID="cri-o://8166e672cbf52094e193b459b9fea1ced005ca693a6b82ce4d7f64895bd7d14e" gracePeriod=30
Oct 30 01:15:00.033417 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:00.033368    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-k66r8"]
Oct 30 01:15:00.033610 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:00.033589    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-k66r8" podUID="05c750c5-39da-44a2-8414-3d7ffe09812d" containerName="istio-proxy" containerID="cri-o://472ce76d6c1beb7fc03522ef5e89b6db866188096422af5fcf1ed1e16a23aef1" gracePeriod=30
Oct 30 01:15:00.165037 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:00.164972    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-beeba4-29888715-5t886"]
Oct 30 01:15:00.166234 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:00.166204    5002 cpu_manager.go:414] "containerMap: removing container" podUID="59b4b452-11f0-4885-8b7b-d7a249340941" containerName="revoke-keys"
Oct 30 01:15:00.167454 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:00.167431    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59b4b452-11f0-4885-8b7b-d7a249340941" containerName="revoke-keys"
Oct 30 01:15:00.167606 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:00.167593    5002 cpu_manager.go:414] "containerMap: removing container" podUID="1a0e3f18-a339-4bc3-8d3b-3b38f3a50182" containerName="revoke-keys"
Oct 30 01:15:00.167702 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:00.167691    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a0e3f18-a339-4bc3-8d3b-3b38f3a50182" containerName="revoke-keys"
Oct 30 01:15:00.167791 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:00.167781    5002 cpu_manager.go:414] "containerMap: removing container" podUID="c9821719-d280-4bc0-a68b-9ae8def2955f" containerName="istio-proxy"
Oct 30 01:15:00.167876 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:00.167866    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9821719-d280-4bc0-a68b-9ae8def2955f" containerName="istio-proxy"
Oct 30 01:15:00.167962 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:00.167953    5002 cpu_manager.go:414] "containerMap: removing container" podUID="0eb54f67-9f54-468e-a88c-ba3fbcaa6d2d" containerName="istio-proxy"
Oct 30 01:15:00.168031 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:00.168021    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0eb54f67-9f54-468e-a88c-ba3fbcaa6d2d" containerName="istio-proxy"
Oct 30 01:15:00.168112 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:00.168103    5002 cpu_manager.go:414] "containerMap: removing container" podUID="0eb54f67-9f54-468e-a88c-ba3fbcaa6d2d" containerName="istio-proxy"
Oct 30 01:15:00.168185 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:00.168173    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0eb54f67-9f54-468e-a88c-ba3fbcaa6d2d" containerName="istio-proxy"
Oct 30 01:15:00.168514 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:00.168498    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="59b4b452-11f0-4885-8b7b-d7a249340941" containerName="revoke-keys"
Oct 30 01:15:00.170217 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:00.170197    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a0e3f18-a339-4bc3-8d3b-3b38f3a50182" containerName="revoke-keys"
Oct 30 01:15:00.170800 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:00.170781    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9821719-d280-4bc0-a68b-9ae8def2955f" containerName="istio-proxy"
Oct 30 01:15:00.170932 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:00.170921    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="0eb54f67-9f54-468e-a88c-ba3fbcaa6d2d" containerName="istio-proxy"
Oct 30 01:15:00.171019 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:00.171009    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="0eb54f67-9f54-468e-a88c-ba3fbcaa6d2d" containerName="istio-proxy"
Oct 30 01:15:00.181067 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:00.181040    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-009e78-29888715-526wv"]
Oct 30 01:15:00.186345 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:00.181650    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-beeba4-29888715-5t886"
Oct 30 01:15:00.189109 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:00.189086    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-009e78-29888715-526wv"
Oct 30 01:15:00.191935 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:00.190472    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-d8b2c6-29888715-rqz6d"]
Oct 30 01:15:00.207492 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:00.207460    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5phsh\""
Oct 30 01:15:00.209371 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:00.209347    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:15:00.209690 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:00.209675    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:15:00.222705 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:00.222678    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-d8b2c6-29888715-rqz6d"
Oct 30 01:15:00.224933 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:00.224579    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a7ca1f-29888715-r8jhp"]
Oct 30 01:15:00.251211 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:00.245574    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-7f92f6-29888715-d7v2x"]
Oct 30 01:15:00.257120 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:00.257088    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a7ca1f-29888715-r8jhp"
Oct 30 01:15:00.264136 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:00.264105    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-7f92f6-29888715-d7v2x"
Oct 30 01:15:00.274602 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:00.274564    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/55f94866-dc95-4ee1-b634-a3c434226a03-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-beeba4-29888715-5t886\" (UID: \"55f94866-dc95-4ee1-b634-a3c434226a03\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-beeba4-29888715-5t886"
Oct 30 01:15:00.282366 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:00.280513    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d25m9\" (UniqueName: \"kubernetes.io/projected/6b614db7-0a42-4251-a93b-80ee2f8d3fbe-kube-api-access-d25m9\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-009e78-29888715-526wv\" (UID: \"6b614db7-0a42-4251-a93b-80ee2f8d3fbe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-009e78-29888715-526wv"
Oct 30 01:15:00.282366 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:00.280554    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2w79\" (UniqueName: \"kubernetes.io/projected/55f94866-dc95-4ee1-b634-a3c434226a03-kube-api-access-p2w79\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-beeba4-29888715-5t886\" (UID: \"55f94866-dc95-4ee1-b634-a3c434226a03\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-beeba4-29888715-5t886"
Oct 30 01:15:00.282366 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:00.280615    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b614db7-0a42-4251-a93b-80ee2f8d3fbe-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-009e78-29888715-526wv\" (UID: \"6b614db7-0a42-4251-a93b-80ee2f8d3fbe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-009e78-29888715-526wv"
Oct 30 01:15:00.284892 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:00.284827    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f64dc0-29888715-75wpl"]
Oct 30 01:15:00.303747 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:00.303060    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-beeba4-29888715-5t886"]
Oct 30 01:15:00.303747 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:00.303108    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-009e78-29888715-526wv"]
Oct 30 01:15:00.303747 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:00.303207    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f64dc0-29888715-75wpl"
Oct 30 01:15:00.311362 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:00.308797    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-11f521-29888715-fj8kn"]
Oct 30 01:15:00.321153 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:00.316252    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a7ca1f-29888715-r8jhp"]
Oct 30 01:15:00.321153 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:00.316368    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-11f521-29888715-fj8kn"
Oct 30 01:15:00.323314 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:00.323291    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f64dc0-29888715-75wpl"]
Oct 30 01:15:00.328441 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:00.328406    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-00e07a-29888715-8dq5b"]
Oct 30 01:15:00.334745 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:00.334717    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-a5e9ab-29888715-gltwb"]
Oct 30 01:15:00.347463 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:00.338453    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-00e07a-29888715-8dq5b"
Oct 30 01:15:00.382942 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:00.382904    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7hhq\" (UniqueName: \"kubernetes.io/projected/5834beef-f5d2-4d8f-94da-aa3a86d692cf-kube-api-access-c7hhq\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-f64dc0-29888715-75wpl\" (UID: \"5834beef-f5d2-4d8f-94da-aa3a86d692cf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f64dc0-29888715-75wpl"
Oct 30 01:15:00.383113 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:00.382975    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b614db7-0a42-4251-a93b-80ee2f8d3fbe-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-009e78-29888715-526wv\" (UID: \"6b614db7-0a42-4251-a93b-80ee2f8d3fbe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-009e78-29888715-526wv"
Oct 30 01:15:00.383113 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:00.383039    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5lsv\" (UniqueName: \"kubernetes.io/projected/34ced72e-384a-4e43-90b7-2f907956c2c1-kube-api-access-h5lsv\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-7f92f6-29888715-d7v2x\" (UID: \"34ced72e-384a-4e43-90b7-2f907956c2c1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-7f92f6-29888715-d7v2x"
Oct 30 01:15:00.383113 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:00.383092    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5834beef-f5d2-4d8f-94da-aa3a86d692cf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-f64dc0-29888715-75wpl\" (UID: \"5834beef-f5d2-4d8f-94da-aa3a86d692cf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f64dc0-29888715-75wpl"
Oct 30 01:15:00.383292 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:00.383223    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjvbg\" (UniqueName: \"kubernetes.io/projected/91ed35be-dc11-4d3f-91bc-f4a7b4c7be28-kube-api-access-hjvbg\") pod \"maas-api-key-cleanup-e2e-ipp-b-d8b2c6-29888715-rqz6d\" (UID: \"91ed35be-dc11-4d3f-91bc-f4a7b4c7be28\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-d8b2c6-29888715-rqz6d"
Oct 30 01:15:00.383292 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:00.383267    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/663c6827-07ce-4de7-8d82-033f8b60c129-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-a7ca1f-29888715-r8jhp\" (UID: \"663c6827-07ce-4de7-8d82-033f8b60c129\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a7ca1f-29888715-r8jhp"
Oct 30 01:15:00.383431 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:00.383313    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgshb\" (UniqueName: \"kubernetes.io/projected/663c6827-07ce-4de7-8d82-033f8b60c129-kube-api-access-lgshb\") pod \"maas-api-key-cleanup-e2e-ipp-a-a7ca1f-29888715-r8jhp\" (UID: \"663c6827-07ce-4de7-8d82-033f8b60c129\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a7ca1f-29888715-r8jhp"
Oct 30 01:15:00.383789 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:00.383761    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/55f94866-dc95-4ee1-b634-a3c434226a03-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-beeba4-29888715-5t886\" (UID: \"55f94866-dc95-4ee1-b634-a3c434226a03\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-beeba4-29888715-5t886"
Oct 30 01:15:00.383897 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:00.383836    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d25m9\" (UniqueName: \"kubernetes.io/projected/6b614db7-0a42-4251-a93b-80ee2f8d3fbe-kube-api-access-d25m9\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-009e78-29888715-526wv\" (UID: \"6b614db7-0a42-4251-a93b-80ee2f8d3fbe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-009e78-29888715-526wv"
Oct 30 01:15:00.383897 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:00.383873    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/34ced72e-384a-4e43-90b7-2f907956c2c1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-7f92f6-29888715-d7v2x\" (UID: \"34ced72e-384a-4e43-90b7-2f907956c2c1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-7f92f6-29888715-d7v2x"
Oct 30 01:15:00.384010 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:00.383911    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2w79\" (UniqueName: \"kubernetes.io/projected/55f94866-dc95-4ee1-b634-a3c434226a03-kube-api-access-p2w79\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-beeba4-29888715-5t886\" (UID: \"55f94866-dc95-4ee1-b634-a3c434226a03\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-beeba4-29888715-5t886"
Oct 30 01:15:00.384010 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:00.383955    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/91ed35be-dc11-4d3f-91bc-f4a7b4c7be28-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-d8b2c6-29888715-rqz6d\" (UID: \"91ed35be-dc11-4d3f-91bc-f4a7b4c7be28\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-d8b2c6-29888715-rqz6d"
Oct 30 01:15:00.384838 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:00.384810    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b614db7-0a42-4251-a93b-80ee2f8d3fbe-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-009e78-29888715-526wv\" (UID: \"6b614db7-0a42-4251-a93b-80ee2f8d3fbe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-009e78-29888715-526wv"
Oct 30 01:15:00.385432 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:00.385405    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/55f94866-dc95-4ee1-b634-a3c434226a03-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-beeba4-29888715-5t886\" (UID: \"55f94866-dc95-4ee1-b634-a3c434226a03\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-beeba4-29888715-5t886"
Oct 30 01:15:00.391577 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:00.391551    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-cefb27-29888715-9524l"]
Oct 30 01:15:00.396055 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:00.395958    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-a5e9ab-29888715-gltwb"
Oct 30 01:15:00.402070 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:00.402040    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-11f521-29888715-fj8kn"]
Oct 30 01:15:00.402196 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:00.402091    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-7f92f6-29888715-d7v2x"]
Oct 30 01:15:00.402196 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:00.402119    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-29be73-29888715-p2ctj"]
Oct 30 01:15:00.402315 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:00.402219    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-cefb27-29888715-9524l"
Oct 30 01:15:00.406742 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:00.406717    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-d8b2c6-29888715-rqz6d"]
Oct 30 01:15:00.406873 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:00.406843    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-29be73-29888715-p2ctj"
Oct 30 01:15:00.416571 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:00.416542    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-48df96-29888715-nc8tn"]
Oct 30 01:15:00.421423 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:00.420725    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d25m9\" (UniqueName: \"kubernetes.io/projected/6b614db7-0a42-4251-a93b-80ee2f8d3fbe-kube-api-access-d25m9\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-009e78-29888715-526wv\" (UID: \"6b614db7-0a42-4251-a93b-80ee2f8d3fbe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-009e78-29888715-526wv"
Oct 30 01:15:00.421423 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:00.421077    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2w79\" (UniqueName: \"kubernetes.io/projected/55f94866-dc95-4ee1-b634-a3c434226a03-kube-api-access-p2w79\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-beeba4-29888715-5t886\" (UID: \"55f94866-dc95-4ee1-b634-a3c434226a03\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-beeba4-29888715-5t886"
Oct 30 01:15:00.422393 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:00.422363    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-cefb27-29888715-9524l"]
Oct 30 01:15:00.422499 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:00.422485    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-48df96-29888715-nc8tn"
Oct 30 01:15:00.427371 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:00.427335    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-a5e9ab-29888715-gltwb"]
Oct 30 01:15:00.433350 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:00.433333    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-00e07a-29888715-8dq5b"]
Oct 30 01:15:00.439657 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:00.439629    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-6d44c2-29888715-d8zbn"]
Oct 30 01:15:00.445080 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:00.445057    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-021d6d-29888715-cdjpr"]
Oct 30 01:15:00.445316 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:00.445294    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-6d44c2-29888715-d8zbn"
Oct 30 01:15:00.450504 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:00.450484    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5f387c-29888715-qgmjp"]
Oct 30 01:15:00.450734 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:00.450691    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-021d6d-29888715-cdjpr"
Oct 30 01:15:00.455989 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:00.455967    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-01215f-29888715-bxdtr"]
Oct 30 01:15:00.456705 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:00.456686    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5f387c-29888715-qgmjp"
Oct 30 01:15:00.461444 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:00.461422    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-48df96-29888715-nc8tn"]
Oct 30 01:15:00.461611 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:00.461526    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-01215f-29888715-bxdtr"
Oct 30 01:15:00.463051 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:00.463031    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-6d44c2-29888715-d8zbn"]
Oct 30 01:15:00.467103 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:00.467071    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5f387c-29888715-qgmjp"]
Oct 30 01:15:00.504223 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:00.485448    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tmc5\" (UniqueName: \"kubernetes.io/projected/fd232d4c-4d2b-4137-bf83-b3e140fba4a6-kube-api-access-6tmc5\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-a5e9ab-29888715-gltwb\" (UID: \"fd232d4c-4d2b-4137-bf83-b3e140fba4a6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-a5e9ab-29888715-gltwb"
Oct 30 01:15:00.504223 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:00.485505    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/34ced72e-384a-4e43-90b7-2f907956c2c1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-7f92f6-29888715-d7v2x\" (UID: \"34ced72e-384a-4e43-90b7-2f907956c2c1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-7f92f6-29888715-d7v2x"
Oct 30 01:15:00.504223 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:00.485553    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/91ed35be-dc11-4d3f-91bc-f4a7b4c7be28-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-d8b2c6-29888715-rqz6d\" (UID: \"91ed35be-dc11-4d3f-91bc-f4a7b4c7be28\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-d8b2c6-29888715-rqz6d"
Oct 30 01:15:00.504223 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:00.485588    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7568f0d2-0dfb-47b9-ab97-d06d3c985d90-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-cefb27-29888715-9524l\" (UID: \"7568f0d2-0dfb-47b9-ab97-d06d3c985d90\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-cefb27-29888715-9524l"
Oct 30 01:15:00.504223 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:00.485618    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd232d4c-4d2b-4137-bf83-b3e140fba4a6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-a5e9ab-29888715-gltwb\" (UID: \"fd232d4c-4d2b-4137-bf83-b3e140fba4a6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-a5e9ab-29888715-gltwb"
Oct 30 01:15:00.504223 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:00.485648    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h59hc\" (UniqueName: \"kubernetes.io/projected/5d965e6c-4e56-4dff-918d-c93b9832b72f-kube-api-access-h59hc\") pod \"maas-api-key-cleanup-e2e-worker-w1-29be73-29888715-p2ctj\" (UID: \"5d965e6c-4e56-4dff-918d-c93b9832b72f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-29be73-29888715-p2ctj"
Oct 30 01:15:00.504223 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:00.485685    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7hhq\" (UniqueName: \"kubernetes.io/projected/5834beef-f5d2-4d8f-94da-aa3a86d692cf-kube-api-access-c7hhq\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-f64dc0-29888715-75wpl\" (UID: \"5834beef-f5d2-4d8f-94da-aa3a86d692cf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f64dc0-29888715-75wpl"
Oct 30 01:15:00.504223 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:00.485768    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5lsv\" (UniqueName: \"kubernetes.io/projected/34ced72e-384a-4e43-90b7-2f907956c2c1-kube-api-access-h5lsv\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-7f92f6-29888715-d7v2x\" (UID: \"34ced72e-384a-4e43-90b7-2f907956c2c1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-7f92f6-29888715-d7v2x"
Oct 30 01:15:00.504223 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:00.485818    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5834beef-f5d2-4d8f-94da-aa3a86d692cf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-f64dc0-29888715-75wpl\" (UID: \"5834beef-f5d2-4d8f-94da-aa3a86d692cf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f64dc0-29888715-75wpl"
Oct 30 01:15:00.504223 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:00.485858    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skx8g\" (UniqueName: \"kubernetes.io/projected/82b3d01a-515b-43fe-b05d-ae6d90c678a3-kube-api-access-skx8g\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-00e07a-29888715-8dq5b\" (UID: \"82b3d01a-515b-43fe-b05d-ae6d90c678a3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-00e07a-29888715-8dq5b"
Oct 30 01:15:00.504223 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:00.485894    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdjdh\" (UniqueName: \"kubernetes.io/projected/248fa0d9-fe68-4ba7-82c3-d7911c9b8070-kube-api-access-rdjdh\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-11f521-29888715-fj8kn\" (UID: \"248fa0d9-fe68-4ba7-82c3-d7911c9b8070\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-11f521-29888715-fj8kn"
Oct 30 01:15:00.504223 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:00.485968    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/248fa0d9-fe68-4ba7-82c3-d7911c9b8070-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-11f521-29888715-fj8kn\" (UID: \"248fa0d9-fe68-4ba7-82c3-d7911c9b8070\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-11f521-29888715-fj8kn"
Oct 30 01:15:00.504223 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:00.486028    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxmgv\" (UniqueName: \"kubernetes.io/projected/7568f0d2-0dfb-47b9-ab97-d06d3c985d90-kube-api-access-rxmgv\") pod \"maas-api-key-cleanup-e2e-worker-w3-cefb27-29888715-9524l\" (UID: \"7568f0d2-0dfb-47b9-ab97-d06d3c985d90\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-cefb27-29888715-9524l"
Oct 30 01:15:00.504223 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:00.486066    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjvbg\" (UniqueName: \"kubernetes.io/projected/91ed35be-dc11-4d3f-91bc-f4a7b4c7be28-kube-api-access-hjvbg\") pod \"maas-api-key-cleanup-e2e-ipp-b-d8b2c6-29888715-rqz6d\" (UID: \"91ed35be-dc11-4d3f-91bc-f4a7b4c7be28\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-d8b2c6-29888715-rqz6d"
Oct 30 01:15:00.504223 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:00.486101    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/663c6827-07ce-4de7-8d82-033f8b60c129-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-a7ca1f-29888715-r8jhp\" (UID: \"663c6827-07ce-4de7-8d82-033f8b60c129\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a7ca1f-29888715-r8jhp"
Oct 30 01:15:00.504223 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:00.486140    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgshb\" (UniqueName: \"kubernetes.io/projected/663c6827-07ce-4de7-8d82-033f8b60c129-kube-api-access-lgshb\") pod \"maas-api-key-cleanup-e2e-ipp-a-a7ca1f-29888715-r8jhp\" (UID: \"663c6827-07ce-4de7-8d82-033f8b60c129\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a7ca1f-29888715-r8jhp"
Oct 30 01:15:00.504223 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:00.486184    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/82b3d01a-515b-43fe-b05d-ae6d90c678a3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-00e07a-29888715-8dq5b\" (UID: \"82b3d01a-515b-43fe-b05d-ae6d90c678a3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-00e07a-29888715-8dq5b"
Oct 30 01:15:00.504223 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:00.486245    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d965e6c-4e56-4dff-918d-c93b9832b72f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-29be73-29888715-p2ctj\" (UID: \"5d965e6c-4e56-4dff-918d-c93b9832b72f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-29be73-29888715-p2ctj"
Oct 30 01:15:00.543456 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:00.540904    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/34ced72e-384a-4e43-90b7-2f907956c2c1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-7f92f6-29888715-d7v2x\" (UID: \"34ced72e-384a-4e43-90b7-2f907956c2c1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-7f92f6-29888715-d7v2x"
Oct 30 01:15:00.543456 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:00.542424    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/91ed35be-dc11-4d3f-91bc-f4a7b4c7be28-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-d8b2c6-29888715-rqz6d\" (UID: \"91ed35be-dc11-4d3f-91bc-f4a7b4c7be28\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-d8b2c6-29888715-rqz6d"
Oct 30 01:15:00.548489 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:00.543832    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/663c6827-07ce-4de7-8d82-033f8b60c129-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-a7ca1f-29888715-r8jhp\" (UID: \"663c6827-07ce-4de7-8d82-033f8b60c129\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a7ca1f-29888715-r8jhp"
Oct 30 01:15:00.548489 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:00.544445    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5834beef-f5d2-4d8f-94da-aa3a86d692cf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-f64dc0-29888715-75wpl\" (UID: \"5834beef-f5d2-4d8f-94da-aa3a86d692cf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f64dc0-29888715-75wpl"
Oct 30 01:15:00.557063 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:00.557028    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-29be73-29888715-p2ctj"]
Oct 30 01:15:00.560729 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:00.560702    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-01215f-29888715-bxdtr"]
Oct 30 01:15:00.563699 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:00.563676    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-beeba4-29888715-5t886"
Oct 30 01:15:00.565580 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:00.565515    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-021d6d-29888715-cdjpr"]
Oct 30 01:15:00.573117 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:00.572049    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7hhq\" (UniqueName: \"kubernetes.io/projected/5834beef-f5d2-4d8f-94da-aa3a86d692cf-kube-api-access-c7hhq\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-f64dc0-29888715-75wpl\" (UID: \"5834beef-f5d2-4d8f-94da-aa3a86d692cf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f64dc0-29888715-75wpl"
Oct 30 01:15:00.576500 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:00.576114    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5lsv\" (UniqueName: \"kubernetes.io/projected/34ced72e-384a-4e43-90b7-2f907956c2c1-kube-api-access-h5lsv\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-7f92f6-29888715-d7v2x\" (UID: \"34ced72e-384a-4e43-90b7-2f907956c2c1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-7f92f6-29888715-d7v2x"
Oct 30 01:15:00.577095 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:00.576954    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgshb\" (UniqueName: \"kubernetes.io/projected/663c6827-07ce-4de7-8d82-033f8b60c129-kube-api-access-lgshb\") pod \"maas-api-key-cleanup-e2e-ipp-a-a7ca1f-29888715-r8jhp\" (UID: \"663c6827-07ce-4de7-8d82-033f8b60c129\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a7ca1f-29888715-r8jhp"
Oct 30 01:15:00.578402 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:00.577596    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjvbg\" (UniqueName: \"kubernetes.io/projected/91ed35be-dc11-4d3f-91bc-f4a7b4c7be28-kube-api-access-hjvbg\") pod \"maas-api-key-cleanup-e2e-ipp-b-d8b2c6-29888715-rqz6d\" (UID: \"91ed35be-dc11-4d3f-91bc-f4a7b4c7be28\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-d8b2c6-29888715-rqz6d"
Oct 30 01:15:00.590223 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:00.587108    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-009e78-29888715-526wv"
Oct 30 01:15:00.590223 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:00.587698    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tmc5\" (UniqueName: \"kubernetes.io/projected/fd232d4c-4d2b-4137-bf83-b3e140fba4a6-kube-api-access-6tmc5\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-a5e9ab-29888715-gltwb\" (UID: \"fd232d4c-4d2b-4137-bf83-b3e140fba4a6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-a5e9ab-29888715-gltwb"
Oct 30 01:15:00.590223 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:00.587776    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7568f0d2-0dfb-47b9-ab97-d06d3c985d90-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-cefb27-29888715-9524l\" (UID: \"7568f0d2-0dfb-47b9-ab97-d06d3c985d90\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-cefb27-29888715-9524l"
Oct 30 01:15:00.590223 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:00.587800    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd232d4c-4d2b-4137-bf83-b3e140fba4a6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-a5e9ab-29888715-gltwb\" (UID: \"fd232d4c-4d2b-4137-bf83-b3e140fba4a6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-a5e9ab-29888715-gltwb"
Oct 30 01:15:00.590223 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:00.587821    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvbpv\" (UniqueName: \"kubernetes.io/projected/92776c04-7f5b-457a-be15-a5ce82a98a68-kube-api-access-nvbpv\") pod \"maas-api-key-cleanup-e2e-ipp-a-01215f-29888715-bxdtr\" (UID: \"92776c04-7f5b-457a-be15-a5ce82a98a68\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-01215f-29888715-bxdtr"
Oct 30 01:15:00.590223 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:00.587843    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h59hc\" (UniqueName: \"kubernetes.io/projected/5d965e6c-4e56-4dff-918d-c93b9832b72f-kube-api-access-h59hc\") pod \"maas-api-key-cleanup-e2e-worker-w1-29be73-29888715-p2ctj\" (UID: \"5d965e6c-4e56-4dff-918d-c93b9832b72f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-29be73-29888715-p2ctj"
Oct 30 01:15:00.590223 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:00.587875    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxvs8\" (UniqueName: \"kubernetes.io/projected/8dcf8782-f46c-4c0f-9133-793017d66a37-kube-api-access-wxvs8\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-5f387c-29888715-qgmjp\" (UID: \"8dcf8782-f46c-4c0f-9133-793017d66a37\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5f387c-29888715-qgmjp"
Oct 30 01:15:00.590223 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:00.587928    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4tmmh\" (UniqueName: \"kubernetes.io/projected/71d53e61-57bc-4d66-a029-965ea868d660-kube-api-access-4tmmh\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-48df96-29888715-nc8tn\" (UID: \"71d53e61-57bc-4d66-a029-965ea868d660\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-48df96-29888715-nc8tn"
Oct 30 01:15:00.590223 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:00.587974    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skx8g\" (UniqueName: \"kubernetes.io/projected/82b3d01a-515b-43fe-b05d-ae6d90c678a3-kube-api-access-skx8g\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-00e07a-29888715-8dq5b\" (UID: \"82b3d01a-515b-43fe-b05d-ae6d90c678a3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-00e07a-29888715-8dq5b"
Oct 30 01:15:00.590223 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:00.587993    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aba84a8f-bfa1-4a2e-9569-8194bed6ec61-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-021d6d-29888715-cdjpr\" (UID: \"aba84a8f-bfa1-4a2e-9569-8194bed6ec61\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-021d6d-29888715-cdjpr"
Oct 30 01:15:00.590223 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:00.588013    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dznkn\" (UniqueName: \"kubernetes.io/projected/07d6239e-5b00-4c4a-b8dd-d935c4c44ed8-kube-api-access-dznkn\") pod \"maas-api-key-cleanup-e2e-ipp-b-6d44c2-29888715-d8zbn\" (UID: \"07d6239e-5b00-4c4a-b8dd-d935c4c44ed8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-6d44c2-29888715-d8zbn"
Oct 30 01:15:00.590223 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:00.588029    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8mjg\" (UniqueName: \"kubernetes.io/projected/aba84a8f-bfa1-4a2e-9569-8194bed6ec61-kube-api-access-v8mjg\") pod \"maas-api-key-cleanup-e2e-worker-w6-021d6d-29888715-cdjpr\" (UID: \"aba84a8f-bfa1-4a2e-9569-8194bed6ec61\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-021d6d-29888715-cdjpr"
Oct 30 01:15:00.590223 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:00.588046    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdjdh\" (UniqueName: \"kubernetes.io/projected/248fa0d9-fe68-4ba7-82c3-d7911c9b8070-kube-api-access-rdjdh\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-11f521-29888715-fj8kn\" (UID: \"248fa0d9-fe68-4ba7-82c3-d7911c9b8070\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-11f521-29888715-fj8kn"
Oct 30 01:15:00.590223 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:00.588094    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/248fa0d9-fe68-4ba7-82c3-d7911c9b8070-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-11f521-29888715-fj8kn\" (UID: \"248fa0d9-fe68-4ba7-82c3-d7911c9b8070\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-11f521-29888715-fj8kn"
Oct 30 01:15:00.590223 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:00.588110    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8dcf8782-f46c-4c0f-9133-793017d66a37-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-5f387c-29888715-qgmjp\" (UID: \"8dcf8782-f46c-4c0f-9133-793017d66a37\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5f387c-29888715-qgmjp"
Oct 30 01:15:00.590223 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:00.588256    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxmgv\" (UniqueName: \"kubernetes.io/projected/7568f0d2-0dfb-47b9-ab97-d06d3c985d90-kube-api-access-rxmgv\") pod \"maas-api-key-cleanup-e2e-worker-w3-cefb27-29888715-9524l\" (UID: \"7568f0d2-0dfb-47b9-ab97-d06d3c985d90\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-cefb27-29888715-9524l"
Oct 30 01:15:00.590223 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:00.588357    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/82b3d01a-515b-43fe-b05d-ae6d90c678a3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-00e07a-29888715-8dq5b\" (UID: \"82b3d01a-515b-43fe-b05d-ae6d90c678a3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-00e07a-29888715-8dq5b"
Oct 30 01:15:00.590223 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:00.588458    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/92776c04-7f5b-457a-be15-a5ce82a98a68-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-01215f-29888715-bxdtr\" (UID: \"92776c04-7f5b-457a-be15-a5ce82a98a68\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-01215f-29888715-bxdtr"
Oct 30 01:15:00.590223 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:00.588490    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d965e6c-4e56-4dff-918d-c93b9832b72f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-29be73-29888715-p2ctj\" (UID: \"5d965e6c-4e56-4dff-918d-c93b9832b72f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-29be73-29888715-p2ctj"
Oct 30 01:15:00.590223 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:00.588531    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/07d6239e-5b00-4c4a-b8dd-d935c4c44ed8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-6d44c2-29888715-d8zbn\" (UID: \"07d6239e-5b00-4c4a-b8dd-d935c4c44ed8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-6d44c2-29888715-d8zbn"
Oct 30 01:15:00.590223 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:00.588560    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/71d53e61-57bc-4d66-a029-965ea868d660-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-48df96-29888715-nc8tn\" (UID: \"71d53e61-57bc-4d66-a029-965ea868d660\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-48df96-29888715-nc8tn"
Oct 30 01:15:00.590223 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:00.589779    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/248fa0d9-fe68-4ba7-82c3-d7911c9b8070-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-11f521-29888715-fj8kn\" (UID: \"248fa0d9-fe68-4ba7-82c3-d7911c9b8070\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-11f521-29888715-fj8kn"
Oct 30 01:15:00.590223 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:00.589892    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7568f0d2-0dfb-47b9-ab97-d06d3c985d90-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-cefb27-29888715-9524l\" (UID: \"7568f0d2-0dfb-47b9-ab97-d06d3c985d90\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-cefb27-29888715-9524l"
Oct 30 01:15:00.592556 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:00.590494    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd232d4c-4d2b-4137-bf83-b3e140fba4a6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-a5e9ab-29888715-gltwb\" (UID: \"fd232d4c-4d2b-4137-bf83-b3e140fba4a6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-a5e9ab-29888715-gltwb"
Oct 30 01:15:00.592868 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:00.592747    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d965e6c-4e56-4dff-918d-c93b9832b72f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-29be73-29888715-p2ctj\" (UID: \"5d965e6c-4e56-4dff-918d-c93b9832b72f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-29be73-29888715-p2ctj"
Oct 30 01:15:00.593500 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:00.593476    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/82b3d01a-515b-43fe-b05d-ae6d90c678a3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-00e07a-29888715-8dq5b\" (UID: \"82b3d01a-515b-43fe-b05d-ae6d90c678a3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-00e07a-29888715-8dq5b"
Oct 30 01:15:00.607794 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:00.607757    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-d8b2c6-29888715-rqz6d"
Oct 30 01:15:00.611594 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:00.611367    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a7ca1f-29888715-r8jhp"
Oct 30 01:15:00.617855 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:00.617323    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxmgv\" (UniqueName: \"kubernetes.io/projected/7568f0d2-0dfb-47b9-ab97-d06d3c985d90-kube-api-access-rxmgv\") pod \"maas-api-key-cleanup-e2e-worker-w3-cefb27-29888715-9524l\" (UID: \"7568f0d2-0dfb-47b9-ab97-d06d3c985d90\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-cefb27-29888715-9524l"
Oct 30 01:15:00.619426 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:00.618041    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-skx8g\" (UniqueName: \"kubernetes.io/projected/82b3d01a-515b-43fe-b05d-ae6d90c678a3-kube-api-access-skx8g\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-00e07a-29888715-8dq5b\" (UID: \"82b3d01a-515b-43fe-b05d-ae6d90c678a3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-00e07a-29888715-8dq5b"
Oct 30 01:15:00.619426 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:00.618316    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdjdh\" (UniqueName: \"kubernetes.io/projected/248fa0d9-fe68-4ba7-82c3-d7911c9b8070-kube-api-access-rdjdh\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-11f521-29888715-fj8kn\" (UID: \"248fa0d9-fe68-4ba7-82c3-d7911c9b8070\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-11f521-29888715-fj8kn"
Oct 30 01:15:00.620297 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:00.620272    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h59hc\" (UniqueName: \"kubernetes.io/projected/5d965e6c-4e56-4dff-918d-c93b9832b72f-kube-api-access-h59hc\") pod \"maas-api-key-cleanup-e2e-worker-w1-29be73-29888715-p2ctj\" (UID: \"5d965e6c-4e56-4dff-918d-c93b9832b72f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-29be73-29888715-p2ctj"
Oct 30 01:15:00.621329 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:00.621312    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tmc5\" (UniqueName: \"kubernetes.io/projected/fd232d4c-4d2b-4137-bf83-b3e140fba4a6-kube-api-access-6tmc5\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-a5e9ab-29888715-gltwb\" (UID: \"fd232d4c-4d2b-4137-bf83-b3e140fba4a6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-a5e9ab-29888715-gltwb"
Oct 30 01:15:00.628765 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:00.628738    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-7f92f6-29888715-d7v2x"
Oct 30 01:15:00.678208 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:00.678034    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f64dc0-29888715-75wpl"
Oct 30 01:15:00.693551 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:00.693350    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/71d53e61-57bc-4d66-a029-965ea868d660-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-48df96-29888715-nc8tn\" (UID: \"71d53e61-57bc-4d66-a029-965ea868d660\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-48df96-29888715-nc8tn"
Oct 30 01:15:00.695426 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:00.695239    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/71d53e61-57bc-4d66-a029-965ea868d660-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-48df96-29888715-nc8tn\" (UID: \"71d53e61-57bc-4d66-a029-965ea868d660\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-48df96-29888715-nc8tn"
Oct 30 01:15:00.695988 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:00.695633    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvbpv\" (UniqueName: \"kubernetes.io/projected/92776c04-7f5b-457a-be15-a5ce82a98a68-kube-api-access-nvbpv\") pod \"maas-api-key-cleanup-e2e-ipp-a-01215f-29888715-bxdtr\" (UID: \"92776c04-7f5b-457a-be15-a5ce82a98a68\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-01215f-29888715-bxdtr"
Oct 30 01:15:00.695988 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:00.695696    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxvs8\" (UniqueName: \"kubernetes.io/projected/8dcf8782-f46c-4c0f-9133-793017d66a37-kube-api-access-wxvs8\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-5f387c-29888715-qgmjp\" (UID: \"8dcf8782-f46c-4c0f-9133-793017d66a37\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5f387c-29888715-qgmjp"
Oct 30 01:15:00.695988 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:00.695773    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tmmh\" (UniqueName: \"kubernetes.io/projected/71d53e61-57bc-4d66-a029-965ea868d660-kube-api-access-4tmmh\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-48df96-29888715-nc8tn\" (UID: \"71d53e61-57bc-4d66-a029-965ea868d660\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-48df96-29888715-nc8tn"
Oct 30 01:15:00.696632 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:00.696333    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aba84a8f-bfa1-4a2e-9569-8194bed6ec61-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-021d6d-29888715-cdjpr\" (UID: \"aba84a8f-bfa1-4a2e-9569-8194bed6ec61\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-021d6d-29888715-cdjpr"
Oct 30 01:15:00.696632 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:00.696397    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dznkn\" (UniqueName: \"kubernetes.io/projected/07d6239e-5b00-4c4a-b8dd-d935c4c44ed8-kube-api-access-dznkn\") pod \"maas-api-key-cleanup-e2e-ipp-b-6d44c2-29888715-d8zbn\" (UID: \"07d6239e-5b00-4c4a-b8dd-d935c4c44ed8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-6d44c2-29888715-d8zbn"
Oct 30 01:15:00.696632 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:00.696428    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8mjg\" (UniqueName: \"kubernetes.io/projected/aba84a8f-bfa1-4a2e-9569-8194bed6ec61-kube-api-access-v8mjg\") pod \"maas-api-key-cleanup-e2e-worker-w6-021d6d-29888715-cdjpr\" (UID: \"aba84a8f-bfa1-4a2e-9569-8194bed6ec61\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-021d6d-29888715-cdjpr"
Oct 30 01:15:00.696632 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:00.696508    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8dcf8782-f46c-4c0f-9133-793017d66a37-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-5f387c-29888715-qgmjp\" (UID: \"8dcf8782-f46c-4c0f-9133-793017d66a37\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5f387c-29888715-qgmjp"
Oct 30 01:15:00.699825 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:00.697461    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-11f521-29888715-fj8kn"
Oct 30 01:15:00.707877 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:00.705867    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aba84a8f-bfa1-4a2e-9569-8194bed6ec61-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-021d6d-29888715-cdjpr\" (UID: \"aba84a8f-bfa1-4a2e-9569-8194bed6ec61\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-021d6d-29888715-cdjpr"
Oct 30 01:15:00.707877 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:00.707091    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8dcf8782-f46c-4c0f-9133-793017d66a37-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-5f387c-29888715-qgmjp\" (UID: \"8dcf8782-f46c-4c0f-9133-793017d66a37\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5f387c-29888715-qgmjp"
Oct 30 01:15:00.712736 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:00.710322    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/92776c04-7f5b-457a-be15-a5ce82a98a68-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-01215f-29888715-bxdtr\" (UID: \"92776c04-7f5b-457a-be15-a5ce82a98a68\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-01215f-29888715-bxdtr"
Oct 30 01:15:00.712736 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:00.710418    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/07d6239e-5b00-4c4a-b8dd-d935c4c44ed8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-6d44c2-29888715-d8zbn\" (UID: \"07d6239e-5b00-4c4a-b8dd-d935c4c44ed8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-6d44c2-29888715-d8zbn"
Oct 30 01:15:00.712736 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:00.711250    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/92776c04-7f5b-457a-be15-a5ce82a98a68-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-01215f-29888715-bxdtr\" (UID: \"92776c04-7f5b-457a-be15-a5ce82a98a68\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-01215f-29888715-bxdtr"
Oct 30 01:15:00.712736 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:00.711289    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/07d6239e-5b00-4c4a-b8dd-d935c4c44ed8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-6d44c2-29888715-d8zbn\" (UID: \"07d6239e-5b00-4c4a-b8dd-d935c4c44ed8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-6d44c2-29888715-d8zbn"
Oct 30 01:15:00.712736 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:00.711497    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-00e07a-29888715-8dq5b"
Oct 30 01:15:00.730121 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:00.729952    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tmmh\" (UniqueName: \"kubernetes.io/projected/71d53e61-57bc-4d66-a029-965ea868d660-kube-api-access-4tmmh\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-48df96-29888715-nc8tn\" (UID: \"71d53e61-57bc-4d66-a029-965ea868d660\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-48df96-29888715-nc8tn"
Oct 30 01:15:00.742588 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:00.737088    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxvs8\" (UniqueName: \"kubernetes.io/projected/8dcf8782-f46c-4c0f-9133-793017d66a37-kube-api-access-wxvs8\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-5f387c-29888715-qgmjp\" (UID: \"8dcf8782-f46c-4c0f-9133-793017d66a37\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5f387c-29888715-qgmjp"
Oct 30 01:15:00.742588 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:00.742391    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8mjg\" (UniqueName: \"kubernetes.io/projected/aba84a8f-bfa1-4a2e-9569-8194bed6ec61-kube-api-access-v8mjg\") pod \"maas-api-key-cleanup-e2e-worker-w6-021d6d-29888715-cdjpr\" (UID: \"aba84a8f-bfa1-4a2e-9569-8194bed6ec61\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-021d6d-29888715-cdjpr"
Oct 30 01:15:00.789255 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:00.779410    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-a5e9ab-29888715-gltwb"
Oct 30 01:15:00.789255 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:00.780994    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-cefb27-29888715-9524l"
Oct 30 01:15:00.803361 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:00.803312    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvbpv\" (UniqueName: \"kubernetes.io/projected/92776c04-7f5b-457a-be15-a5ce82a98a68-kube-api-access-nvbpv\") pod \"maas-api-key-cleanup-e2e-ipp-a-01215f-29888715-bxdtr\" (UID: \"92776c04-7f5b-457a-be15-a5ce82a98a68\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-01215f-29888715-bxdtr"
Oct 30 01:15:00.804083 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:00.804055    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dznkn\" (UniqueName: \"kubernetes.io/projected/07d6239e-5b00-4c4a-b8dd-d935c4c44ed8-kube-api-access-dznkn\") pod \"maas-api-key-cleanup-e2e-ipp-b-6d44c2-29888715-d8zbn\" (UID: \"07d6239e-5b00-4c4a-b8dd-d935c4c44ed8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-6d44c2-29888715-d8zbn"
Oct 30 01:15:00.866139 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:00.866070    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-29be73-29888715-p2ctj"
Oct 30 01:15:00.881713 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:00.873421    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-48df96-29888715-nc8tn"
Oct 30 01:15:00.881927 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:00.877759    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-6d44c2-29888715-d8zbn"
Oct 30 01:15:00.963288 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:00.963012    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-021d6d-29888715-cdjpr"
Oct 30 01:15:01.003352 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:01.003141    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5f387c-29888715-qgmjp"
Oct 30 01:15:01.024774 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:01.023224    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-01215f-29888715-bxdtr"
Oct 30 01:15:01.124257 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:01.121127    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-beeba4-29888715-5t886"]
Oct 30 01:15:01.368941 ip-10-0-128-7 kubenswrapper[5002]: W1030 01:15:01.368903    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34ced72e_384a_4e43_90b7_2f907956c2c1.slice/crio-7e1e71ffd83542909bc6bfd2db790c897c6f7ea70d1bec97e92511c764ec7dca WatchSource:0}: Error finding container 7e1e71ffd83542909bc6bfd2db790c897c6f7ea70d1bec97e92511c764ec7dca: Status 404 returned error can't find the container with id 7e1e71ffd83542909bc6bfd2db790c897c6f7ea70d1bec97e92511c764ec7dca
Oct 30 01:15:01.370628 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:01.370606    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-7f92f6-29888715-d7v2x"]
Oct 30 01:15:01.387828 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:01.387799    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-d8b2c6-29888715-rqz6d"]
Oct 30 01:15:01.388338 ip-10-0-128-7 kubenswrapper[5002]: W1030 01:15:01.388309    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91ed35be_dc11_4d3f_91bc_f4a7b4c7be28.slice/crio-424cdec1bbc8085f2a2b9db6170d2734044d32569b5dc1ff7af37c39250e15de WatchSource:0}: Error finding container 424cdec1bbc8085f2a2b9db6170d2734044d32569b5dc1ff7af37c39250e15de: Status 404 returned error can't find the container with id 424cdec1bbc8085f2a2b9db6170d2734044d32569b5dc1ff7af37c39250e15de
Oct 30 01:15:01.449938 ip-10-0-128-7 kubenswrapper[5002]: W1030 01:15:01.449896    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod663c6827_07ce_4de7_8d82_033f8b60c129.slice/crio-34e4f7315658b81f5cfe97e992e00503cb7b9299e7c348c6593a3d5b6ede24e0 WatchSource:0}: Error finding container 34e4f7315658b81f5cfe97e992e00503cb7b9299e7c348c6593a3d5b6ede24e0: Status 404 returned error can't find the container with id 34e4f7315658b81f5cfe97e992e00503cb7b9299e7c348c6593a3d5b6ede24e0
Oct 30 01:15:01.455104 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:01.455075    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a7ca1f-29888715-r8jhp"]
Oct 30 01:15:01.520452 ip-10-0-128-7 kubenswrapper[5002]: W1030 01:15:01.519279    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod248fa0d9_fe68_4ba7_82c3_d7911c9b8070.slice/crio-79b96f91575863859730f0e1a0ed49706c661ab7767f7bcaff975ee894802a6e WatchSource:0}: Error finding container 79b96f91575863859730f0e1a0ed49706c661ab7767f7bcaff975ee894802a6e: Status 404 returned error can't find the container with id 79b96f91575863859730f0e1a0ed49706c661ab7767f7bcaff975ee894802a6e
Oct 30 01:15:01.526868 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:01.526791    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-11f521-29888715-fj8kn"]
Oct 30 01:15:01.534691 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:01.534656    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-00e07a-29888715-8dq5b"]
Oct 30 01:15:01.534899 ip-10-0-128-7 kubenswrapper[5002]: W1030 01:15:01.534873    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82b3d01a_515b_43fe_b05d_ae6d90c678a3.slice/crio-9c193892e89fed33c4a6390183cd7dad03e7ff5d7432a11ce3690beef07ea944 WatchSource:0}: Error finding container 9c193892e89fed33c4a6390183cd7dad03e7ff5d7432a11ce3690beef07ea944: Status 404 returned error can't find the container with id 9c193892e89fed33c4a6390183cd7dad03e7ff5d7432a11ce3690beef07ea944
Oct 30 01:15:01.540826 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:01.540797    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-009e78-29888715-526wv"]
Oct 30 01:15:01.541301 ip-10-0-128-7 kubenswrapper[5002]: W1030 01:15:01.541270    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b614db7_0a42_4251_a93b_80ee2f8d3fbe.slice/crio-11dc2468e22b04d16e259f59b63ffd3794700f105b63961d7d4f27fc9ad73314 WatchSource:0}: Error finding container 11dc2468e22b04d16e259f59b63ffd3794700f105b63961d7d4f27fc9ad73314: Status 404 returned error can't find the container with id 11dc2468e22b04d16e259f59b63ffd3794700f105b63961d7d4f27fc9ad73314
Oct 30 01:15:01.546069 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:01.546048    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f64dc0-29888715-75wpl"]
Oct 30 01:15:01.563487 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:01.561072    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-cefb27-29888715-9524l"]
Oct 30 01:15:01.590635 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:01.590588    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-6d44c2-29888715-d8zbn"]
Oct 30 01:15:01.605744 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:01.605708    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-29be73-29888715-p2ctj"]
Oct 30 01:15:01.609564 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:01.609541    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-009e78-29888715-526wv" event={"ID":"6b614db7-0a42-4251-a93b-80ee2f8d3fbe","Type":"ContainerStarted","Data":"11dc2468e22b04d16e259f59b63ffd3794700f105b63961d7d4f27fc9ad73314"}
Oct 30 01:15:01.610542 ip-10-0-128-7 kubenswrapper[5002]: W1030 01:15:01.610518    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07d6239e_5b00_4c4a_b8dd_d935c4c44ed8.slice/crio-33b7081cfc0f0f76484d465f808326f485b12bbee65ea4c54fb76c88f19013b1 WatchSource:0}: Error finding container 33b7081cfc0f0f76484d465f808326f485b12bbee65ea4c54fb76c88f19013b1: Status 404 returned error can't find the container with id 33b7081cfc0f0f76484d465f808326f485b12bbee65ea4c54fb76c88f19013b1
Oct 30 01:15:01.615836 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:01.615796    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a7ca1f-29888715-r8jhp" event={"ID":"663c6827-07ce-4de7-8d82-033f8b60c129","Type":"ContainerStarted","Data":"34e4f7315658b81f5cfe97e992e00503cb7b9299e7c348c6593a3d5b6ede24e0"}
Oct 30 01:15:01.617696 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:01.617615    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-beeba4-29888715-5t886" event={"ID":"55f94866-dc95-4ee1-b634-a3c434226a03","Type":"ContainerStarted","Data":"3006b2a3a667cd9e90dbc8cc872b9d530eebc1be145ea7fe62dc2b9dddb67bf2"}
Oct 30 01:15:01.617696 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:01.617664    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-beeba4-29888715-5t886" event={"ID":"55f94866-dc95-4ee1-b634-a3c434226a03","Type":"ContainerStarted","Data":"47c109df8632bce8bce5b4257e38ed2e040696e9ce420a06bd8d3590b49e2478"}
Oct 30 01:15:01.621594 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:01.620098    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f64dc0-29888715-75wpl" event={"ID":"5834beef-f5d2-4d8f-94da-aa3a86d692cf","Type":"ContainerStarted","Data":"3e62fd1e7091e70a39a9b7e264c4ab7547dc772081fd0b005f2188f8c7b20be1"}
Oct 30 01:15:01.623636 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:01.623601    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-d8b2c6-29888715-rqz6d" event={"ID":"91ed35be-dc11-4d3f-91bc-f4a7b4c7be28","Type":"ContainerStarted","Data":"424cdec1bbc8085f2a2b9db6170d2734044d32569b5dc1ff7af37c39250e15de"}
Oct 30 01:15:01.628859 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:01.628830    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-11f521-29888715-fj8kn" event={"ID":"248fa0d9-fe68-4ba7-82c3-d7911c9b8070","Type":"ContainerStarted","Data":"79b96f91575863859730f0e1a0ed49706c661ab7767f7bcaff975ee894802a6e"}
Oct 30 01:15:01.630167 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:01.630149    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-01215f-29888715-bxdtr"]
Oct 30 01:15:01.635215 ip-10-0-128-7 kubenswrapper[5002]: W1030 01:15:01.635192    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92776c04_7f5b_457a_be15_a5ce82a98a68.slice/crio-2a21bd55303fa74cfa2d16674d3b76c5e163746c03f51c8dd76b47e8c65219b3 WatchSource:0}: Error finding container 2a21bd55303fa74cfa2d16674d3b76c5e163746c03f51c8dd76b47e8c65219b3: Status 404 returned error can't find the container with id 2a21bd55303fa74cfa2d16674d3b76c5e163746c03f51c8dd76b47e8c65219b3
Oct 30 01:15:01.635552 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:01.635501    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-7f92f6-29888715-d7v2x" event={"ID":"34ced72e-384a-4e43-90b7-2f907956c2c1","Type":"ContainerStarted","Data":"7e1e71ffd83542909bc6bfd2db790c897c6f7ea70d1bec97e92511c764ec7dca"}
Oct 30 01:15:01.637446 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:01.637406    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-00e07a-29888715-8dq5b" event={"ID":"82b3d01a-515b-43fe-b05d-ae6d90c678a3","Type":"ContainerStarted","Data":"9c193892e89fed33c4a6390183cd7dad03e7ff5d7432a11ce3690beef07ea944"}
Oct 30 01:15:01.638961 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:01.638931    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-cefb27-29888715-9524l" event={"ID":"7568f0d2-0dfb-47b9-ab97-d06d3c985d90","Type":"ContainerStarted","Data":"0a850b1d818322abbdd6c5a37637eb88107df5aa51caabcb9e2f65dcd25ff525"}
Oct 30 01:15:01.659057 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:01.659030    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-a5e9ab-29888715-gltwb"]
Oct 30 01:15:01.663013 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:01.661737    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-beeba4-29888715-5t886" podStartSLOduration=1.661720884 podStartE2EDuration="1.661720884s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:15:01.660172351 +0000 UTC m=+2597.276605255" watchObservedRunningTime="2026-10-30 01:15:01.661720884 +0000 UTC m=+2597.278153764"
Oct 30 01:15:01.674097 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:01.674072    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5f387c-29888715-qgmjp"]
Oct 30 01:15:01.679558 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:01.679531    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-48df96-29888715-nc8tn"]
Oct 30 01:15:01.685933 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:01.685903    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-021d6d-29888715-cdjpr"]
Oct 30 01:15:01.864307 ip-10-0-128-7 kubenswrapper[5002]: W1030 01:15:01.864233    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd232d4c_4d2b_4137_bf83_b3e140fba4a6.slice/crio-ee793b9ccfdadb3f8ee4da26abdbc21b18cfff3cbd568f62d619faf48503f237 WatchSource:0}: Error finding container ee793b9ccfdadb3f8ee4da26abdbc21b18cfff3cbd568f62d619faf48503f237: Status 404 returned error can't find the container with id ee793b9ccfdadb3f8ee4da26abdbc21b18cfff3cbd568f62d619faf48503f237
Oct 30 01:15:02.657841 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:02.657801    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a7ca1f-29888715-r8jhp" event={"ID":"663c6827-07ce-4de7-8d82-033f8b60c129","Type":"ContainerStarted","Data":"70378e3017440afc8f255b76be79b9292922351096bd2cc260a971233a04fed7"}
Oct 30 01:15:02.662506 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:02.662439    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5f387c-29888715-qgmjp" event={"ID":"8dcf8782-f46c-4c0f-9133-793017d66a37","Type":"ContainerStarted","Data":"3eed03255dbdbafc2d803d0bbe38c65c13b23a9f4e647518f797c3c2fd457e44"}
Oct 30 01:15:02.665410 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:02.664600    5002 generic.go:356] "Generic (PLEG): container finished" podID="55f94866-dc95-4ee1-b634-a3c434226a03" containerID="3006b2a3a667cd9e90dbc8cc872b9d530eebc1be145ea7fe62dc2b9dddb67bf2" exitCode=60
Oct 30 01:15:02.665410 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:02.664682    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-beeba4-29888715-5t886" event={"ID":"55f94866-dc95-4ee1-b634-a3c434226a03","Type":"ContainerDied","Data":"3006b2a3a667cd9e90dbc8cc872b9d530eebc1be145ea7fe62dc2b9dddb67bf2"}
Oct 30 01:15:02.665410 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:02.665102    5002 scope.go:122] "RemoveContainer" containerID="3006b2a3a667cd9e90dbc8cc872b9d530eebc1be145ea7fe62dc2b9dddb67bf2"
Oct 30 01:15:02.681468 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:02.681445    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f64dc0-29888715-75wpl" event={"ID":"5834beef-f5d2-4d8f-94da-aa3a86d692cf","Type":"ContainerStarted","Data":"08c79ebb8047d1a14a60b532dec758ad9dd5e46f36f4f687b5ef2f19177d6969"}
Oct 30 01:15:02.687717 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:02.687693    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-d8b2c6-29888715-rqz6d" event={"ID":"91ed35be-dc11-4d3f-91bc-f4a7b4c7be28","Type":"ContainerStarted","Data":"65500fa2527ee88dac31a6dbf0f10d8632de184f5b78218ef15a1315e46706b6"}
Oct 30 01:15:02.689586 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:02.689561    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-11f521-29888715-fj8kn" event={"ID":"248fa0d9-fe68-4ba7-82c3-d7911c9b8070","Type":"ContainerStarted","Data":"5e655a10a687c0642635431eddb212eaea7249839af1d0559d550785f2290047"}
Oct 30 01:15:02.692439 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:02.691580    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-7f92f6-29888715-d7v2x" event={"ID":"34ced72e-384a-4e43-90b7-2f907956c2c1","Type":"ContainerStarted","Data":"8e6c0dba751ddf2b317a7d745e260cc2e98d0f54da411785686c1fa1afd769b9"}
Oct 30 01:15:02.694335 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:02.694314    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-6d44c2-29888715-d8zbn" event={"ID":"07d6239e-5b00-4c4a-b8dd-d935c4c44ed8","Type":"ContainerStarted","Data":"9d301926783affaeb89891f9623c6463f648eb1eea2441689df5083df6aad3d7"}
Oct 30 01:15:02.694440 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:02.694349    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-6d44c2-29888715-d8zbn" event={"ID":"07d6239e-5b00-4c4a-b8dd-d935c4c44ed8","Type":"ContainerStarted","Data":"33b7081cfc0f0f76484d465f808326f485b12bbee65ea4c54fb76c88f19013b1"}
Oct 30 01:15:02.696511 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:02.696490    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-29be73-29888715-p2ctj" event={"ID":"5d965e6c-4e56-4dff-918d-c93b9832b72f","Type":"ContainerStarted","Data":"48ea907220398868496390e97a0e50fbf945de8bbdd86090b2445175d3824950"}
Oct 30 01:15:02.696588 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:02.696525    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-29be73-29888715-p2ctj" event={"ID":"5d965e6c-4e56-4dff-918d-c93b9832b72f","Type":"ContainerStarted","Data":"db7b74e9959f9257f1e0322aa9ad0910875293ffa18409f16c54e60d8e7fb1a0"}
Oct 30 01:15:02.698885 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:02.698863    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-00e07a-29888715-8dq5b" event={"ID":"82b3d01a-515b-43fe-b05d-ae6d90c678a3","Type":"ContainerStarted","Data":"9fc1fbeb226aea90918da930b194cab4e1513f598dec1e8cca1943fabb8b2400"}
Oct 30 01:15:02.700592 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:02.700568    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-cefb27-29888715-9524l" event={"ID":"7568f0d2-0dfb-47b9-ab97-d06d3c985d90","Type":"ContainerStarted","Data":"b66b3acd682ee0f2c3987550bd4391fd71d30946fab6f984b55fab662b456f2d"}
Oct 30 01:15:02.702458 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:02.702432    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-01215f-29888715-bxdtr" event={"ID":"92776c04-7f5b-457a-be15-a5ce82a98a68","Type":"ContainerStarted","Data":"2a21bd55303fa74cfa2d16674d3b76c5e163746c03f51c8dd76b47e8c65219b3"}
Oct 30 01:15:02.704105 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:02.704078    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-a5e9ab-29888715-gltwb" event={"ID":"fd232d4c-4d2b-4137-bf83-b3e140fba4a6","Type":"ContainerStarted","Data":"ee793b9ccfdadb3f8ee4da26abdbc21b18cfff3cbd568f62d619faf48503f237"}
Oct 30 01:15:02.704565 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:02.704525    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a7ca1f-29888715-r8jhp" podStartSLOduration=2.704513779 podStartE2EDuration="2.704513779s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:15:02.70243688 +0000 UTC m=+2598.318869761" watchObservedRunningTime="2026-10-30 01:15:02.704513779 +0000 UTC m=+2598.320946658"
Oct 30 01:15:02.705795 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:02.705741    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-009e78-29888715-526wv" event={"ID":"6b614db7-0a42-4251-a93b-80ee2f8d3fbe","Type":"ContainerStarted","Data":"4be3160acaef3b0490800122facd369e9141ea5371754bab5892c053d34f86cd"}
Oct 30 01:15:02.707127 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:02.707105    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-021d6d-29888715-cdjpr" event={"ID":"aba84a8f-bfa1-4a2e-9569-8194bed6ec61","Type":"ContainerStarted","Data":"2d45cfec5ae49877c5e72b80fcebb0c08066c07ce0dcdebcecd39745cca5db7a"}
Oct 30 01:15:02.708709 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:02.708687    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-48df96-29888715-nc8tn" event={"ID":"71d53e61-57bc-4d66-a029-965ea868d660","Type":"ContainerStarted","Data":"f1dd85b0d6309ca0d2e5857f3161f89cb9cd063868672bb2246128f734d70130"}
Oct 30 01:15:02.748166 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:02.748099    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-cefb27-29888715-9524l" podStartSLOduration=2.748079095 podStartE2EDuration="2.748079095s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:15:02.744806744 +0000 UTC m=+2598.361239660" watchObservedRunningTime="2026-10-30 01:15:02.748079095 +0000 UTC m=+2598.364511982"
Oct 30 01:15:02.806404 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:02.799173    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f64dc0-29888715-75wpl" podStartSLOduration=2.799150516 podStartE2EDuration="2.799150516s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:15:02.792871651 +0000 UTC m=+2598.409304528" watchObservedRunningTime="2026-10-30 01:15:02.799150516 +0000 UTC m=+2598.415583397"
Oct 30 01:15:02.880482 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:02.879563    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-00e07a-29888715-8dq5b" podStartSLOduration=2.879540897 podStartE2EDuration="2.879540897s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:15:02.835921367 +0000 UTC m=+2598.452354246" watchObservedRunningTime="2026-10-30 01:15:02.879540897 +0000 UTC m=+2598.495973777"
Oct 30 01:15:02.906173 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:02.905471    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-11f521-29888715-fj8kn" podStartSLOduration=2.904925581 podStartE2EDuration="2.904925581s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:15:02.893001871 +0000 UTC m=+2598.509434767" watchObservedRunningTime="2026-10-30 01:15:02.904925581 +0000 UTC m=+2598.521358461"
Oct 30 01:15:02.940546 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:02.940498    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-6d44c2-29888715-d8zbn" podStartSLOduration=2.940483339 podStartE2EDuration="2.940483339s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:15:02.935847103 +0000 UTC m=+2598.552280001" watchObservedRunningTime="2026-10-30 01:15:02.940483339 +0000 UTC m=+2598.556916254"
Oct 30 01:15:02.988637 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:02.988561    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-d8b2c6-29888715-rqz6d" podStartSLOduration=2.988542082 podStartE2EDuration="2.988542082s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:15:02.986582853 +0000 UTC m=+2598.603015733" watchObservedRunningTime="2026-10-30 01:15:02.988542082 +0000 UTC m=+2598.604974963"
Oct 30 01:15:03.031777 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:03.031721    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-7f92f6-29888715-d7v2x" podStartSLOduration=3.031704821 podStartE2EDuration="3.031704821s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:15:03.028213925 +0000 UTC m=+2598.644646822" watchObservedRunningTime="2026-10-30 01:15:03.031704821 +0000 UTC m=+2598.648137699"
Oct 30 01:15:03.080537 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:03.080478    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-29be73-29888715-p2ctj" podStartSLOduration=3.080458631 podStartE2EDuration="3.080458631s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:15:03.071880169 +0000 UTC m=+2598.688313049" watchObservedRunningTime="2026-10-30 01:15:03.080458631 +0000 UTC m=+2598.696891510"
Oct 30 01:15:03.162362 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:03.162303    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-009e78-29888715-526wv" podStartSLOduration=3.162286241 podStartE2EDuration="3.162286241s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:15:03.157728034 +0000 UTC m=+2598.774160913" watchObservedRunningTime="2026-10-30 01:15:03.162286241 +0000 UTC m=+2598.778719120"
Oct 30 01:15:03.721236 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:03.721196    5002 generic.go:356] "Generic (PLEG): container finished" podID="82b3d01a-515b-43fe-b05d-ae6d90c678a3" containerID="9fc1fbeb226aea90918da930b194cab4e1513f598dec1e8cca1943fabb8b2400" exitCode=60
Oct 30 01:15:03.721708 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:03.721230    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-00e07a-29888715-8dq5b" event={"ID":"82b3d01a-515b-43fe-b05d-ae6d90c678a3","Type":"ContainerDied","Data":"9fc1fbeb226aea90918da930b194cab4e1513f598dec1e8cca1943fabb8b2400"}
Oct 30 01:15:03.721708 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:03.721679    5002 scope.go:122] "RemoveContainer" containerID="9fc1fbeb226aea90918da930b194cab4e1513f598dec1e8cca1943fabb8b2400"
Oct 30 01:15:03.723322 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:03.723297    5002 generic.go:356] "Generic (PLEG): container finished" podID="7568f0d2-0dfb-47b9-ab97-d06d3c985d90" containerID="b66b3acd682ee0f2c3987550bd4391fd71d30946fab6f984b55fab662b456f2d" exitCode=60
Oct 30 01:15:03.723472 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:03.723372    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-cefb27-29888715-9524l" event={"ID":"7568f0d2-0dfb-47b9-ab97-d06d3c985d90","Type":"ContainerDied","Data":"b66b3acd682ee0f2c3987550bd4391fd71d30946fab6f984b55fab662b456f2d"}
Oct 30 01:15:03.723709 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:03.723690    5002 scope.go:122] "RemoveContainer" containerID="b66b3acd682ee0f2c3987550bd4391fd71d30946fab6f984b55fab662b456f2d"
Oct 30 01:15:03.725453 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:03.725428    5002 generic.go:356] "Generic (PLEG): container finished" podID="92776c04-7f5b-457a-be15-a5ce82a98a68" containerID="283abc0b3b05a3b7d3820e5f84225649121cd9c0a4af84f3004f8dd3b5a46233" exitCode=60
Oct 30 01:15:03.725570 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:03.725514    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-01215f-29888715-bxdtr" event={"ID":"92776c04-7f5b-457a-be15-a5ce82a98a68","Type":"ContainerDied","Data":"283abc0b3b05a3b7d3820e5f84225649121cd9c0a4af84f3004f8dd3b5a46233"}
Oct 30 01:15:03.725808 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:03.725785    5002 scope.go:122] "RemoveContainer" containerID="283abc0b3b05a3b7d3820e5f84225649121cd9c0a4af84f3004f8dd3b5a46233"
Oct 30 01:15:03.727768 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:03.727751    5002 generic.go:356] "Generic (PLEG): container finished" podID="fd232d4c-4d2b-4137-bf83-b3e140fba4a6" containerID="893310bdd40578c1d1b5d9ccf2d71f09aa50fe3699eeba7883debc9b51d555a9" exitCode=60
Oct 30 01:15:03.728065 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:03.728052    5002 scope.go:122] "RemoveContainer" containerID="893310bdd40578c1d1b5d9ccf2d71f09aa50fe3699eeba7883debc9b51d555a9"
Oct 30 01:15:03.728286 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:03.728269    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-a5e9ab-29888715-gltwb" event={"ID":"fd232d4c-4d2b-4137-bf83-b3e140fba4a6","Type":"ContainerDied","Data":"893310bdd40578c1d1b5d9ccf2d71f09aa50fe3699eeba7883debc9b51d555a9"}
Oct 30 01:15:03.730054 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:03.730035    5002 generic.go:356] "Generic (PLEG): container finished" podID="6b614db7-0a42-4251-a93b-80ee2f8d3fbe" containerID="4be3160acaef3b0490800122facd369e9141ea5371754bab5892c053d34f86cd" exitCode=60
Oct 30 01:15:03.730126 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:03.730089    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-009e78-29888715-526wv" event={"ID":"6b614db7-0a42-4251-a93b-80ee2f8d3fbe","Type":"ContainerDied","Data":"4be3160acaef3b0490800122facd369e9141ea5371754bab5892c053d34f86cd"}
Oct 30 01:15:03.730328 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:03.730315    5002 scope.go:122] "RemoveContainer" containerID="4be3160acaef3b0490800122facd369e9141ea5371754bab5892c053d34f86cd"
Oct 30 01:15:03.732156 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:03.732136    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-021d6d-29888715-cdjpr" event={"ID":"aba84a8f-bfa1-4a2e-9569-8194bed6ec61","Type":"ContainerStarted","Data":"23d86e27deee3065b5680f02c84e0fb0e4e386164c512e89f5f2fea9e738f8c2"}
Oct 30 01:15:03.733689 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:03.733665    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-48df96-29888715-nc8tn" event={"ID":"71d53e61-57bc-4d66-a029-965ea868d660","Type":"ContainerStarted","Data":"d2f304fb1b84d726eadbb7f5b73b32114943f7ffc2f9627210db3035884b2db8"}
Oct 30 01:15:03.736028 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:03.736007    5002 generic.go:356] "Generic (PLEG): container finished" podID="663c6827-07ce-4de7-8d82-033f8b60c129" containerID="70378e3017440afc8f255b76be79b9292922351096bd2cc260a971233a04fed7" exitCode=60
Oct 30 01:15:03.736085 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:03.736062    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a7ca1f-29888715-r8jhp" event={"ID":"663c6827-07ce-4de7-8d82-033f8b60c129","Type":"ContainerDied","Data":"70378e3017440afc8f255b76be79b9292922351096bd2cc260a971233a04fed7"}
Oct 30 01:15:03.736325 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:03.736309    5002 scope.go:122] "RemoveContainer" containerID="70378e3017440afc8f255b76be79b9292922351096bd2cc260a971233a04fed7"
Oct 30 01:15:03.737835 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:03.737812    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5f387c-29888715-qgmjp" event={"ID":"8dcf8782-f46c-4c0f-9133-793017d66a37","Type":"ContainerStarted","Data":"3bbb94d57e7fc25edb6403e92d9c36e06d9e35d87e584e26786ede4d7ebf2db1"}
Oct 30 01:15:03.739867 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:03.739843    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-beeba4-29888715-5t886" event={"ID":"55f94866-dc95-4ee1-b634-a3c434226a03","Type":"ContainerStarted","Data":"23bd2ab9aaff2196c2ce41d1f1b9a49b6c46d6e2b090fedf9acb902d803f01f8"}
Oct 30 01:15:03.742893 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:03.742473    5002 generic.go:356] "Generic (PLEG): container finished" podID="5834beef-f5d2-4d8f-94da-aa3a86d692cf" containerID="08c79ebb8047d1a14a60b532dec758ad9dd5e46f36f4f687b5ef2f19177d6969" exitCode=60
Oct 30 01:15:03.742893 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:03.742794    5002 scope.go:122] "RemoveContainer" containerID="08c79ebb8047d1a14a60b532dec758ad9dd5e46f36f4f687b5ef2f19177d6969"
Oct 30 01:15:03.742893 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:03.742824    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f64dc0-29888715-75wpl" event={"ID":"5834beef-f5d2-4d8f-94da-aa3a86d692cf","Type":"ContainerDied","Data":"08c79ebb8047d1a14a60b532dec758ad9dd5e46f36f4f687b5ef2f19177d6969"}
Oct 30 01:15:03.757744 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:03.757714    5002 generic.go:356] "Generic (PLEG): container finished" podID="91ed35be-dc11-4d3f-91bc-f4a7b4c7be28" containerID="65500fa2527ee88dac31a6dbf0f10d8632de184f5b78218ef15a1315e46706b6" exitCode=60
Oct 30 01:15:03.757956 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:03.757793    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-d8b2c6-29888715-rqz6d" event={"ID":"91ed35be-dc11-4d3f-91bc-f4a7b4c7be28","Type":"ContainerDied","Data":"65500fa2527ee88dac31a6dbf0f10d8632de184f5b78218ef15a1315e46706b6"}
Oct 30 01:15:03.758148 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:03.758126    5002 scope.go:122] "RemoveContainer" containerID="65500fa2527ee88dac31a6dbf0f10d8632de184f5b78218ef15a1315e46706b6"
Oct 30 01:15:03.762281 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:03.761527    5002 generic.go:356] "Generic (PLEG): container finished" podID="248fa0d9-fe68-4ba7-82c3-d7911c9b8070" containerID="5e655a10a687c0642635431eddb212eaea7249839af1d0559d550785f2290047" exitCode=60
Oct 30 01:15:03.762281 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:03.761602    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-11f521-29888715-fj8kn" event={"ID":"248fa0d9-fe68-4ba7-82c3-d7911c9b8070","Type":"ContainerDied","Data":"5e655a10a687c0642635431eddb212eaea7249839af1d0559d550785f2290047"}
Oct 30 01:15:03.762281 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:03.761859    5002 scope.go:122] "RemoveContainer" containerID="5e655a10a687c0642635431eddb212eaea7249839af1d0559d550785f2290047"
Oct 30 01:15:03.766609 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:03.765283    5002 generic.go:356] "Generic (PLEG): container finished" podID="34ced72e-384a-4e43-90b7-2f907956c2c1" containerID="8e6c0dba751ddf2b317a7d745e260cc2e98d0f54da411785686c1fa1afd769b9" exitCode=60
Oct 30 01:15:03.766609 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:03.765370    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-7f92f6-29888715-d7v2x" event={"ID":"34ced72e-384a-4e43-90b7-2f907956c2c1","Type":"ContainerDied","Data":"8e6c0dba751ddf2b317a7d745e260cc2e98d0f54da411785686c1fa1afd769b9"}
Oct 30 01:15:03.766609 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:03.765748    5002 scope.go:122] "RemoveContainer" containerID="8e6c0dba751ddf2b317a7d745e260cc2e98d0f54da411785686c1fa1afd769b9"
Oct 30 01:15:03.808245 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:03.806697    5002 generic.go:356] "Generic (PLEG): container finished" podID="07d6239e-5b00-4c4a-b8dd-d935c4c44ed8" containerID="9d301926783affaeb89891f9623c6463f648eb1eea2441689df5083df6aad3d7" exitCode=60
Oct 30 01:15:03.808245 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:03.806866    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-6d44c2-29888715-d8zbn" event={"ID":"07d6239e-5b00-4c4a-b8dd-d935c4c44ed8","Type":"ContainerDied","Data":"9d301926783affaeb89891f9623c6463f648eb1eea2441689df5083df6aad3d7"}
Oct 30 01:15:03.808245 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:03.808099    5002 scope.go:122] "RemoveContainer" containerID="9d301926783affaeb89891f9623c6463f648eb1eea2441689df5083df6aad3d7"
Oct 30 01:15:03.813791 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:03.812972    5002 generic.go:356] "Generic (PLEG): container finished" podID="5d965e6c-4e56-4dff-918d-c93b9832b72f" containerID="48ea907220398868496390e97a0e50fbf945de8bbdd86090b2445175d3824950" exitCode=60
Oct 30 01:15:03.813791 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:03.813040    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-29be73-29888715-p2ctj" event={"ID":"5d965e6c-4e56-4dff-918d-c93b9832b72f","Type":"ContainerDied","Data":"48ea907220398868496390e97a0e50fbf945de8bbdd86090b2445175d3824950"}
Oct 30 01:15:03.813791 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:03.813334    5002 scope.go:122] "RemoveContainer" containerID="48ea907220398868496390e97a0e50fbf945de8bbdd86090b2445175d3824950"
Oct 30 01:15:03.905476 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:03.903176    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-48df96-29888715-nc8tn" podStartSLOduration=3.903158704 podStartE2EDuration="3.903158704s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:15:03.858684306 +0000 UTC m=+2599.475117187" watchObservedRunningTime="2026-10-30 01:15:03.903158704 +0000 UTC m=+2599.519591583"
Oct 30 01:15:03.908278 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:03.908230    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5f387c-29888715-qgmjp" podStartSLOduration=3.9082192989999998 podStartE2EDuration="3.908219299s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:15:03.900901789 +0000 UTC m=+2599.517334668" watchObservedRunningTime="2026-10-30 01:15:03.908219299 +0000 UTC m=+2599.524652184"
Oct 30 01:15:03.959452 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:03.958653    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-021d6d-29888715-cdjpr" podStartSLOduration=3.958630905 podStartE2EDuration="3.958630905s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:15:03.94311946 +0000 UTC m=+2599.559552338" watchObservedRunningTime="2026-10-30 01:15:03.958630905 +0000 UTC m=+2599.575063786"
Oct 30 01:15:04.033151 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:04.033114    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a882968-pm4vv"]
Oct 30 01:15:04.041837 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:04.041802    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a882968-pm4vv"]
Oct 30 01:15:04.595628 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:04.595593    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5dab5c4c-0cf8-430f-b6ba-b81775ffa60a" path="/var/lib/kubelet/pods/5dab5c4c-0cf8-430f-b6ba-b81775ffa60a/volumes"
Oct 30 01:15:04.762456 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:04.755655    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-5f387c-openshift-default-5ff776bc59-bdp4r"
Oct 30 01:15:04.826204 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:04.826165    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-29be73-29888715-p2ctj" event={"ID":"5d965e6c-4e56-4dff-918d-c93b9832b72f","Type":"ContainerStarted","Data":"23542a46cd867d1aabf74ab0dacd3ead9b877c819c387b235a8cdc14ad648201"}
Oct 30 01:15:04.829742 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:04.829710    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-00e07a-29888715-8dq5b" event={"ID":"82b3d01a-515b-43fe-b05d-ae6d90c678a3","Type":"ContainerStarted","Data":"2255fcbba7d31cf86be4719a7884880cbcf52c8f0d976dcd99ac3033e505733b"}
Oct 30 01:15:04.832507 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:04.832480    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-cefb27-29888715-9524l" event={"ID":"7568f0d2-0dfb-47b9-ab97-d06d3c985d90","Type":"ContainerStarted","Data":"645814b0bf1ed27656feb00c8e15dfcf1b8264ab84d39187925274efbd6949a3"}
Oct 30 01:15:04.834702 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:04.834678    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-01215f-29888715-bxdtr" event={"ID":"92776c04-7f5b-457a-be15-a5ce82a98a68","Type":"ContainerStarted","Data":"50f107ac2854a1d0ec109dbae8e32f9e0859bb93052aecf60c8bbfd329e571d0"}
Oct 30 01:15:04.837011 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:04.836984    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-a5e9ab-29888715-gltwb" event={"ID":"fd232d4c-4d2b-4137-bf83-b3e140fba4a6","Type":"ContainerStarted","Data":"d9ecb54895eba5673f1094a2001c1141074d7cc26c4c0613d5f021fd49395ac9"}
Oct 30 01:15:04.839284 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:04.839255    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-009e78-29888715-526wv" event={"ID":"6b614db7-0a42-4251-a93b-80ee2f8d3fbe","Type":"ContainerStarted","Data":"c8b8d15d9c8d14dcf01fdf342d673d51d66ffbd0b07c8a1bbdac7924670a2c61"}
Oct 30 01:15:04.841047 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:04.841020    5002 generic.go:356] "Generic (PLEG): container finished" podID="aba84a8f-bfa1-4a2e-9569-8194bed6ec61" containerID="23d86e27deee3065b5680f02c84e0fb0e4e386164c512e89f5f2fea9e738f8c2" exitCode=60
Oct 30 01:15:04.841128 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:04.841089    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-021d6d-29888715-cdjpr" event={"ID":"aba84a8f-bfa1-4a2e-9569-8194bed6ec61","Type":"ContainerDied","Data":"23d86e27deee3065b5680f02c84e0fb0e4e386164c512e89f5f2fea9e738f8c2"}
Oct 30 01:15:04.841347 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:04.841324    5002 scope.go:122] "RemoveContainer" containerID="23d86e27deee3065b5680f02c84e0fb0e4e386164c512e89f5f2fea9e738f8c2"
Oct 30 01:15:04.842994 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:04.842965    5002 generic.go:356] "Generic (PLEG): container finished" podID="71d53e61-57bc-4d66-a029-965ea868d660" containerID="d2f304fb1b84d726eadbb7f5b73b32114943f7ffc2f9627210db3035884b2db8" exitCode=60
Oct 30 01:15:04.843107 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:04.843063    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-48df96-29888715-nc8tn" event={"ID":"71d53e61-57bc-4d66-a029-965ea868d660","Type":"ContainerDied","Data":"d2f304fb1b84d726eadbb7f5b73b32114943f7ffc2f9627210db3035884b2db8"}
Oct 30 01:15:04.843529 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:04.843255    5002 scope.go:122] "RemoveContainer" containerID="d2f304fb1b84d726eadbb7f5b73b32114943f7ffc2f9627210db3035884b2db8"
Oct 30 01:15:04.845141 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:04.845118    5002 generic.go:356] "Generic (PLEG): container finished" podID="9bb1708e-878e-461f-9294-265673c31b04" containerID="8166e672cbf52094e193b459b9fea1ced005ca693a6b82ce4d7f64895bd7d14e" exitCode=0
Oct 30 01:15:04.845224 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:04.845192    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-5f387c-openshift-default-5ff776bc59-bdp4r"
Oct 30 01:15:04.845287 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:04.845238    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-5f387c-openshift-default-5ff776bc59-bdp4r" event={"ID":"9bb1708e-878e-461f-9294-265673c31b04","Type":"ContainerDied","Data":"8166e672cbf52094e193b459b9fea1ced005ca693a6b82ce4d7f64895bd7d14e"}
Oct 30 01:15:04.845287 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:04.845272    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-5f387c-openshift-default-5ff776bc59-bdp4r" event={"ID":"9bb1708e-878e-461f-9294-265673c31b04","Type":"ContainerDied","Data":"aa45ab1198e767cd66826f199f7801061dd2b6c6ef93213f5408006cf85aed03"}
Oct 30 01:15:04.845370 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:04.845292    5002 scope.go:122] "RemoveContainer" containerID="8166e672cbf52094e193b459b9fea1ced005ca693a6b82ce4d7f64895bd7d14e"
Oct 30 01:15:04.855692 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:04.855666    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a7ca1f-29888715-r8jhp" event={"ID":"663c6827-07ce-4de7-8d82-033f8b60c129","Type":"ContainerStarted","Data":"df65a7038cdf1664a4852454ee63255addfe0f962c8d1342bd29de14c22224c6"}
Oct 30 01:15:04.861927 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:04.861898    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bb1708e-878e-461f-9294-265673c31b04-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9bb1708e-878e-461f-9294-265673c31b04-istio-data\") pod \"9bb1708e-878e-461f-9294-265673c31b04\" (UID: \"9bb1708e-878e-461f-9294-265673c31b04\") "
Oct 30 01:15:04.862165 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:04.862140    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9bb1708e-878e-461f-9294-265673c31b04-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9bb1708e-878e-461f-9294-265673c31b04-istiod-ca-cert\") pod \"9bb1708e-878e-461f-9294-265673c31b04\" (UID: \"9bb1708e-878e-461f-9294-265673c31b04\") "
Oct 30 01:15:04.862391 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:04.862357    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bb1708e-878e-461f-9294-265673c31b04-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9bb1708e-878e-461f-9294-265673c31b04-istio-envoy\") pod \"9bb1708e-878e-461f-9294-265673c31b04\" (UID: \"9bb1708e-878e-461f-9294-265673c31b04\") "
Oct 30 01:15:04.862769 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:04.862747    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bb1708e-878e-461f-9294-265673c31b04-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9bb1708e-878e-461f-9294-265673c31b04-workload-certs\") pod \"9bb1708e-878e-461f-9294-265673c31b04\" (UID: \"9bb1708e-878e-461f-9294-265673c31b04\") "
Oct 30 01:15:04.862940 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:04.862924    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9bb1708e-878e-461f-9294-265673c31b04-kube-api-access-m8fjz\" (UniqueName: \"kubernetes.io/projected/9bb1708e-878e-461f-9294-265673c31b04-kube-api-access-m8fjz\") pod \"9bb1708e-878e-461f-9294-265673c31b04\" (UID: \"9bb1708e-878e-461f-9294-265673c31b04\") "
Oct 30 01:15:04.863082 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:04.863068    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9bb1708e-878e-461f-9294-265673c31b04-istio-token\" (UniqueName: \"kubernetes.io/projected/9bb1708e-878e-461f-9294-265673c31b04-istio-token\") pod \"9bb1708e-878e-461f-9294-265673c31b04\" (UID: \"9bb1708e-878e-461f-9294-265673c31b04\") "
Oct 30 01:15:04.877166 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:04.877095    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bb1708e-878e-461f-9294-265673c31b04-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bb1708e-878e-461f-9294-265673c31b04-credential-socket\") pod \"9bb1708e-878e-461f-9294-265673c31b04\" (UID: \"9bb1708e-878e-461f-9294-265673c31b04\") "
Oct 30 01:15:04.877301 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:04.877181    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9bb1708e-878e-461f-9294-265673c31b04-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9bb1708e-878e-461f-9294-265673c31b04-istio-podinfo\") pod \"9bb1708e-878e-461f-9294-265673c31b04\" (UID: \"9bb1708e-878e-461f-9294-265673c31b04\") "
Oct 30 01:15:04.877301 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:04.866193    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9bb1708e-878e-461f-9294-265673c31b04-istiod-ca-cert" pod "9bb1708e-878e-461f-9294-265673c31b04" (UID: "9bb1708e-878e-461f-9294-265673c31b04"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:15:04.877438 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:04.877355    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bb1708e-878e-461f-9294-265673c31b04-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bb1708e-878e-461f-9294-265673c31b04-workload-socket\") pod \"9bb1708e-878e-461f-9294-265673c31b04\" (UID: \"9bb1708e-878e-461f-9294-265673c31b04\") "
Oct 30 01:15:04.885633 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:04.868050    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bb1708e-878e-461f-9294-265673c31b04-workload-certs" pod "9bb1708e-878e-461f-9294-265673c31b04" (UID: "9bb1708e-878e-461f-9294-265673c31b04"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:15:04.885633 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:04.868628    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bb1708e-878e-461f-9294-265673c31b04-istio-data" pod "9bb1708e-878e-461f-9294-265673c31b04" (UID: "9bb1708e-878e-461f-9294-265673c31b04"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:15:04.885633 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:04.878596    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bb1708e-878e-461f-9294-265673c31b04-workload-socket" pod "9bb1708e-878e-461f-9294-265673c31b04" (UID: "9bb1708e-878e-461f-9294-265673c31b04"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:15:04.885633 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:04.880586    5002 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9bb1708e-878e-461f-9294-265673c31b04-istiod-ca-cert\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:15:04.885633 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:04.880616    5002 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9bb1708e-878e-461f-9294-265673c31b04-workload-certs\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:15:04.885633 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:04.880638    5002 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bb1708e-878e-461f-9294-265673c31b04-workload-socket\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:15:04.885633 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:04.880652    5002 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9bb1708e-878e-461f-9294-265673c31b04-istio-data\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:15:04.886134 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:04.885988    5002 generic.go:356] "Generic (PLEG): container finished" podID="8dcf8782-f46c-4c0f-9133-793017d66a37" containerID="3bbb94d57e7fc25edb6403e92d9c36e06d9e35d87e584e26786ede4d7ebf2db1" exitCode=60
Oct 30 01:15:04.886194 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:04.886163    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5f387c-29888715-qgmjp" event={"ID":"8dcf8782-f46c-4c0f-9133-793017d66a37","Type":"ContainerDied","Data":"3bbb94d57e7fc25edb6403e92d9c36e06d9e35d87e584e26786ede4d7ebf2db1"}
Oct 30 01:15:04.887796 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:04.886775    5002 scope.go:122] "RemoveContainer" containerID="3bbb94d57e7fc25edb6403e92d9c36e06d9e35d87e584e26786ede4d7ebf2db1"
Oct 30 01:15:04.893053 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:04.893026    5002 generic.go:356] "Generic (PLEG): container finished" podID="55f94866-dc95-4ee1-b634-a3c434226a03" containerID="23bd2ab9aaff2196c2ce41d1f1b9a49b6c46d6e2b090fedf9acb902d803f01f8" exitCode=60
Oct 30 01:15:04.893328 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:04.893301    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-beeba4-29888715-5t886" event={"ID":"55f94866-dc95-4ee1-b634-a3c434226a03","Type":"ContainerDied","Data":"23bd2ab9aaff2196c2ce41d1f1b9a49b6c46d6e2b090fedf9acb902d803f01f8"}
Oct 30 01:15:04.894505 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:04.894486    5002 scope.go:122] "RemoveContainer" containerID="23bd2ab9aaff2196c2ce41d1f1b9a49b6c46d6e2b090fedf9acb902d803f01f8"
Oct 30 01:15:04.896029 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:15:04.896002    5002 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-beeba4-29888715-5t886_odh-ai-gateway-infra(55f94866-dc95-4ee1-b634-a3c434226a03)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-beeba4-29888715-5t886" podUID="55f94866-dc95-4ee1-b634-a3c434226a03"
Oct 30 01:15:04.908140 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:04.908111    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bb1708e-878e-461f-9294-265673c31b04-credential-socket" pod "9bb1708e-878e-461f-9294-265673c31b04" (UID: "9bb1708e-878e-461f-9294-265673c31b04"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:15:04.908302 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:04.908280    5002 scope.go:122] "RemoveContainer" containerID="8166e672cbf52094e193b459b9fea1ced005ca693a6b82ce4d7f64895bd7d14e"
Oct 30 01:15:04.910053 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:04.910026    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bb1708e-878e-461f-9294-265673c31b04-istio-token" pod "9bb1708e-878e-461f-9294-265673c31b04" (UID: "9bb1708e-878e-461f-9294-265673c31b04"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:15:04.910293 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:15:04.910265    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8166e672cbf52094e193b459b9fea1ced005ca693a6b82ce4d7f64895bd7d14e\": container with ID starting with 8166e672cbf52094e193b459b9fea1ced005ca693a6b82ce4d7f64895bd7d14e not found: ID does not exist" containerID="8166e672cbf52094e193b459b9fea1ced005ca693a6b82ce4d7f64895bd7d14e"
Oct 30 01:15:04.910366 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:04.910313    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8166e672cbf52094e193b459b9fea1ced005ca693a6b82ce4d7f64895bd7d14e"} err="failed to get container status \"8166e672cbf52094e193b459b9fea1ced005ca693a6b82ce4d7f64895bd7d14e\": rpc error: code = NotFound desc = could not find container \"8166e672cbf52094e193b459b9fea1ced005ca693a6b82ce4d7f64895bd7d14e\": container with ID starting with 8166e672cbf52094e193b459b9fea1ced005ca693a6b82ce4d7f64895bd7d14e not found: ID does not exist"
Oct 30 01:15:04.910366 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:04.910334    5002 scope.go:122] "RemoveContainer" containerID="3006b2a3a667cd9e90dbc8cc872b9d530eebc1be145ea7fe62dc2b9dddb67bf2"
Oct 30 01:15:04.911686 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:04.911658    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9bb1708e-878e-461f-9294-265673c31b04-istio-podinfo" pod "9bb1708e-878e-461f-9294-265673c31b04" (UID: "9bb1708e-878e-461f-9294-265673c31b04"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:15:04.932817 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:04.932781    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f64dc0-29888715-75wpl" event={"ID":"5834beef-f5d2-4d8f-94da-aa3a86d692cf","Type":"ContainerStarted","Data":"17e0a6c2cf3abe78ce7d4a834d12509990087dd9538d7dad7bdab6421ec72ff3"}
Oct 30 01:15:04.936728 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:04.936457    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bb1708e-878e-461f-9294-265673c31b04-kube-api-access-m8fjz" pod "9bb1708e-878e-461f-9294-265673c31b04" (UID: "9bb1708e-878e-461f-9294-265673c31b04"). InnerVolumeSpecName "kube-api-access-m8fjz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:15:04.937656 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:04.937626    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-d8b2c6-29888715-rqz6d" event={"ID":"91ed35be-dc11-4d3f-91bc-f4a7b4c7be28","Type":"ContainerStarted","Data":"c1334c2e13aa673e9a0b651b6144991b77895f8f2486c3bdeec62a76950b213b"}
Oct 30 01:15:04.944422 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:04.942311    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bb1708e-878e-461f-9294-265673c31b04-istio-envoy" pod "9bb1708e-878e-461f-9294-265673c31b04" (UID: "9bb1708e-878e-461f-9294-265673c31b04"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:15:04.944772 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:04.944469    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-11f521-29888715-fj8kn" event={"ID":"248fa0d9-fe68-4ba7-82c3-d7911c9b8070","Type":"ContainerStarted","Data":"6d02e29347533557fb6e5ea114602aaf38092b749b7366936bff8383dbc40473"}
Oct 30 01:15:04.954295 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:04.954263    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-7f92f6-29888715-d7v2x" event={"ID":"34ced72e-384a-4e43-90b7-2f907956c2c1","Type":"ContainerStarted","Data":"3d6d154d5ea078834d229b13ea707f618f9040d1d25101fcc00ef3bde333470f"}
Oct 30 01:15:04.963136 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:04.963094    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-6d44c2-29888715-d8zbn" event={"ID":"07d6239e-5b00-4c4a-b8dd-d935c4c44ed8","Type":"ContainerStarted","Data":"14aa48c03a3e6d7be42a1b03c02b8ea6aef907c5c8f22e72936cd38a6703d033"}
Oct 30 01:15:04.984508 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:04.984264    5002 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9bb1708e-878e-461f-9294-265673c31b04-istio-envoy\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:15:04.984508 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:04.984295    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8fjz\" (UniqueName: \"kubernetes.io/projected/9bb1708e-878e-461f-9294-265673c31b04-kube-api-access-m8fjz\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:15:04.984508 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:04.984312    5002 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9bb1708e-878e-461f-9294-265673c31b04-istio-token\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:15:04.984508 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:04.984325    5002 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bb1708e-878e-461f-9294-265673c31b04-credential-socket\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:15:04.984508 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:04.984338    5002 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9bb1708e-878e-461f-9294-265673c31b04-istio-podinfo\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:15:05.179720 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:05.179564    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-a5e9ab-29888715-gltwb" podStartSLOduration=5.179546798 podStartE2EDuration="5.179546798s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:15:05.177474066 +0000 UTC m=+2600.793906966" watchObservedRunningTime="2026-10-30 01:15:05.179546798 +0000 UTC m=+2600.795979678"
Oct 30 01:15:05.339676 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:05.339614    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-01215f-29888715-bxdtr" podStartSLOduration=5.339594436 podStartE2EDuration="5.339594436s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:15:05.278505799 +0000 UTC m=+2600.894938697" watchObservedRunningTime="2026-10-30 01:15:05.339594436 +0000 UTC m=+2600.956027316"
Oct 30 01:15:05.775644 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:05.775617    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-k66r8"
Oct 30 01:15:05.810103 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:05.810062    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-5f387c-openshift-default-5ff776bc59-bdp4r"]
Oct 30 01:15:05.815969 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:05.815927    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-5f387c-openshift-default-5ff776bc59-bdp4r"]
Oct 30 01:15:05.899787 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:05.899744    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05c750c5-39da-44a2-8414-3d7ffe09812d-istio-token\" (UniqueName: \"kubernetes.io/projected/05c750c5-39da-44a2-8414-3d7ffe09812d-istio-token\") pod \"05c750c5-39da-44a2-8414-3d7ffe09812d\" (UID: \"05c750c5-39da-44a2-8414-3d7ffe09812d\") "
Oct 30 01:15:05.899977 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:05.899897    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05c750c5-39da-44a2-8414-3d7ffe09812d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05c750c5-39da-44a2-8414-3d7ffe09812d-workload-socket\") pod \"05c750c5-39da-44a2-8414-3d7ffe09812d\" (UID: \"05c750c5-39da-44a2-8414-3d7ffe09812d\") "
Oct 30 01:15:05.899977 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:05.899956    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05c750c5-39da-44a2-8414-3d7ffe09812d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05c750c5-39da-44a2-8414-3d7ffe09812d-istiod-ca-cert\") pod \"05c750c5-39da-44a2-8414-3d7ffe09812d\" (UID: \"05c750c5-39da-44a2-8414-3d7ffe09812d\") "
Oct 30 01:15:05.900186 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:05.900156    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05c750c5-39da-44a2-8414-3d7ffe09812d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05c750c5-39da-44a2-8414-3d7ffe09812d-credential-socket\") pod \"05c750c5-39da-44a2-8414-3d7ffe09812d\" (UID: \"05c750c5-39da-44a2-8414-3d7ffe09812d\") "
Oct 30 01:15:05.900286 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:05.900186    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05c750c5-39da-44a2-8414-3d7ffe09812d-workload-socket" pod "05c750c5-39da-44a2-8414-3d7ffe09812d" (UID: "05c750c5-39da-44a2-8414-3d7ffe09812d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:15:05.900286 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:05.900225    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05c750c5-39da-44a2-8414-3d7ffe09812d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05c750c5-39da-44a2-8414-3d7ffe09812d-istio-envoy\") pod \"05c750c5-39da-44a2-8414-3d7ffe09812d\" (UID: \"05c750c5-39da-44a2-8414-3d7ffe09812d\") "
Oct 30 01:15:05.900446 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:05.900287    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05c750c5-39da-44a2-8414-3d7ffe09812d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05c750c5-39da-44a2-8414-3d7ffe09812d-workload-certs\") pod \"05c750c5-39da-44a2-8414-3d7ffe09812d\" (UID: \"05c750c5-39da-44a2-8414-3d7ffe09812d\") "
Oct 30 01:15:05.900446 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:05.900338    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05c750c5-39da-44a2-8414-3d7ffe09812d-kube-api-access-zzwvj\" (UniqueName: \"kubernetes.io/projected/05c750c5-39da-44a2-8414-3d7ffe09812d-kube-api-access-zzwvj\") pod \"05c750c5-39da-44a2-8414-3d7ffe09812d\" (UID: \"05c750c5-39da-44a2-8414-3d7ffe09812d\") "
Oct 30 01:15:05.900446 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:05.900426    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05c750c5-39da-44a2-8414-3d7ffe09812d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05c750c5-39da-44a2-8414-3d7ffe09812d-istio-data\") pod \"05c750c5-39da-44a2-8414-3d7ffe09812d\" (UID: \"05c750c5-39da-44a2-8414-3d7ffe09812d\") "
Oct 30 01:15:05.900677 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:05.900657    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/05c750c5-39da-44a2-8414-3d7ffe09812d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05c750c5-39da-44a2-8414-3d7ffe09812d-istio-podinfo\") pod \"05c750c5-39da-44a2-8414-3d7ffe09812d\" (UID: \"05c750c5-39da-44a2-8414-3d7ffe09812d\") "
Oct 30 01:15:05.901462 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:05.901436    5002 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05c750c5-39da-44a2-8414-3d7ffe09812d-workload-socket\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:15:05.903444 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:05.902255    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05c750c5-39da-44a2-8414-3d7ffe09812d-istiod-ca-cert" pod "05c750c5-39da-44a2-8414-3d7ffe09812d" (UID: "05c750c5-39da-44a2-8414-3d7ffe09812d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:15:05.903444 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:05.902329    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05c750c5-39da-44a2-8414-3d7ffe09812d-credential-socket" pod "05c750c5-39da-44a2-8414-3d7ffe09812d" (UID: "05c750c5-39da-44a2-8414-3d7ffe09812d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:15:05.905061 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:05.905025    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05c750c5-39da-44a2-8414-3d7ffe09812d-workload-certs" pod "05c750c5-39da-44a2-8414-3d7ffe09812d" (UID: "05c750c5-39da-44a2-8414-3d7ffe09812d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:15:05.905243 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:05.905138    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05c750c5-39da-44a2-8414-3d7ffe09812d-istio-data" pod "05c750c5-39da-44a2-8414-3d7ffe09812d" (UID: "05c750c5-39da-44a2-8414-3d7ffe09812d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:15:06.010207 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:06.010119    5002 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05c750c5-39da-44a2-8414-3d7ffe09812d-istiod-ca-cert\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:15:06.010488 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:06.010465    5002 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05c750c5-39da-44a2-8414-3d7ffe09812d-credential-socket\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:15:06.010624 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:06.010610    5002 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05c750c5-39da-44a2-8414-3d7ffe09812d-workload-certs\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:15:06.010716 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:06.010707    5002 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05c750c5-39da-44a2-8414-3d7ffe09812d-istio-data\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:15:06.043331 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:06.043240    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-d8b2c6-29888715-rqz6d"]
Oct 30 01:15:06.047220 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:06.045458    5002 generic.go:356] "Generic (PLEG): container finished" podID="248fa0d9-fe68-4ba7-82c3-d7911c9b8070" containerID="6d02e29347533557fb6e5ea114602aaf38092b749b7366936bff8383dbc40473" exitCode=60
Oct 30 01:15:06.047220 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:06.045538    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-11f521-29888715-fj8kn" event={"ID":"248fa0d9-fe68-4ba7-82c3-d7911c9b8070","Type":"ContainerDied","Data":"6d02e29347533557fb6e5ea114602aaf38092b749b7366936bff8383dbc40473"}
Oct 30 01:15:06.047220 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:06.045586    5002 scope.go:122] "RemoveContainer" containerID="5e655a10a687c0642635431eddb212eaea7249839af1d0559d550785f2290047"
Oct 30 01:15:06.047220 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:06.046042    5002 scope.go:122] "RemoveContainer" containerID="6d02e29347533557fb6e5ea114602aaf38092b749b7366936bff8383dbc40473"
Oct 30 01:15:06.047220 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:15:06.046518    5002 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-11f521-29888715-fj8kn_odh-ai-gateway-infra(248fa0d9-fe68-4ba7-82c3-d7911c9b8070)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-11f521-29888715-fj8kn" podUID="248fa0d9-fe68-4ba7-82c3-d7911c9b8070"
Oct 30 01:15:06.067710 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:06.062778    5002 generic.go:356] "Generic (PLEG): container finished" podID="34ced72e-384a-4e43-90b7-2f907956c2c1" containerID="3d6d154d5ea078834d229b13ea707f618f9040d1d25101fcc00ef3bde333470f" exitCode=60
Oct 30 01:15:06.067710 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:06.062878    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-7f92f6-29888715-d7v2x" event={"ID":"34ced72e-384a-4e43-90b7-2f907956c2c1","Type":"ContainerDied","Data":"3d6d154d5ea078834d229b13ea707f618f9040d1d25101fcc00ef3bde333470f"}
Oct 30 01:15:06.067710 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:06.063527    5002 scope.go:122] "RemoveContainer" containerID="3d6d154d5ea078834d229b13ea707f618f9040d1d25101fcc00ef3bde333470f"
Oct 30 01:15:06.067710 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:15:06.064020    5002 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w6-7f92f6-29888715-d7v2x_odh-ai-gateway-infra(34ced72e-384a-4e43-90b7-2f907956c2c1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-7f92f6-29888715-d7v2x" podUID="34ced72e-384a-4e43-90b7-2f907956c2c1"
Oct 30 01:15:06.071407 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:06.071360    5002 generic.go:356] "Generic (PLEG): container finished" podID="07d6239e-5b00-4c4a-b8dd-d935c4c44ed8" containerID="14aa48c03a3e6d7be42a1b03c02b8ea6aef907c5c8f22e72936cd38a6703d033" exitCode=60
Oct 30 01:15:06.071662 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:06.071638    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-6d44c2-29888715-d8zbn" event={"ID":"07d6239e-5b00-4c4a-b8dd-d935c4c44ed8","Type":"ContainerDied","Data":"14aa48c03a3e6d7be42a1b03c02b8ea6aef907c5c8f22e72936cd38a6703d033"}
Oct 30 01:15:06.072179 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:06.071805    5002 scope.go:122] "RemoveContainer" containerID="14aa48c03a3e6d7be42a1b03c02b8ea6aef907c5c8f22e72936cd38a6703d033"
Oct 30 01:15:06.072278 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:15:06.072204    5002 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-6d44c2-29888715-d8zbn_odh-ai-gateway-infra(07d6239e-5b00-4c4a-b8dd-d935c4c44ed8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-6d44c2-29888715-d8zbn" podUID="07d6239e-5b00-4c4a-b8dd-d935c4c44ed8"
Oct 30 01:15:06.075538 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:06.075519    5002 generic.go:356] "Generic (PLEG): container finished" podID="5d965e6c-4e56-4dff-918d-c93b9832b72f" containerID="23542a46cd867d1aabf74ab0dacd3ead9b877c819c387b235a8cdc14ad648201" exitCode=60
Oct 30 01:15:06.075711 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:06.075695    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-29be73-29888715-p2ctj" event={"ID":"5d965e6c-4e56-4dff-918d-c93b9832b72f","Type":"ContainerDied","Data":"23542a46cd867d1aabf74ab0dacd3ead9b877c819c387b235a8cdc14ad648201"}
Oct 30 01:15:06.078649 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:06.078618    5002 scope.go:122] "RemoveContainer" containerID="23542a46cd867d1aabf74ab0dacd3ead9b877c819c387b235a8cdc14ad648201"
Oct 30 01:15:06.079932 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:15:06.079686    5002 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-29be73-29888715-p2ctj_odh-ai-gateway-infra(5d965e6c-4e56-4dff-918d-c93b9832b72f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-29be73-29888715-p2ctj" podUID="5d965e6c-4e56-4dff-918d-c93b9832b72f"
Oct 30 01:15:06.080184 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:06.080066    5002 generic.go:356] "Generic (PLEG): container finished" podID="82b3d01a-515b-43fe-b05d-ae6d90c678a3" containerID="2255fcbba7d31cf86be4719a7884880cbcf52c8f0d976dcd99ac3033e505733b" exitCode=60
Oct 30 01:15:06.080184 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:06.080158    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-00e07a-29888715-8dq5b" event={"ID":"82b3d01a-515b-43fe-b05d-ae6d90c678a3","Type":"ContainerDied","Data":"2255fcbba7d31cf86be4719a7884880cbcf52c8f0d976dcd99ac3033e505733b"}
Oct 30 01:15:06.080905 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:06.080458    5002 scope.go:122] "RemoveContainer" containerID="2255fcbba7d31cf86be4719a7884880cbcf52c8f0d976dcd99ac3033e505733b"
Oct 30 01:15:06.080905 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:15:06.080799    5002 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-00e07a-29888715-8dq5b_odh-ai-gateway-infra(82b3d01a-515b-43fe-b05d-ae6d90c678a3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-00e07a-29888715-8dq5b" podUID="82b3d01a-515b-43fe-b05d-ae6d90c678a3"
Oct 30 01:15:06.083541 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:06.083520    5002 generic.go:356] "Generic (PLEG): container finished" podID="7568f0d2-0dfb-47b9-ab97-d06d3c985d90" containerID="645814b0bf1ed27656feb00c8e15dfcf1b8264ab84d39187925274efbd6949a3" exitCode=60
Oct 30 01:15:06.083723 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:06.083706    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-cefb27-29888715-9524l" event={"ID":"7568f0d2-0dfb-47b9-ab97-d06d3c985d90","Type":"ContainerDied","Data":"645814b0bf1ed27656feb00c8e15dfcf1b8264ab84d39187925274efbd6949a3"}
Oct 30 01:15:06.086039 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:06.086020    5002 scope.go:122] "RemoveContainer" containerID="645814b0bf1ed27656feb00c8e15dfcf1b8264ab84d39187925274efbd6949a3"
Oct 30 01:15:06.086514 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:15:06.086490    5002 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-cefb27-29888715-9524l_odh-ai-gateway-infra(7568f0d2-0dfb-47b9-ab97-d06d3c985d90)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-cefb27-29888715-9524l" podUID="7568f0d2-0dfb-47b9-ab97-d06d3c985d90"
Oct 30 01:15:06.097629 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:06.096612    5002 generic.go:356] "Generic (PLEG): container finished" podID="92776c04-7f5b-457a-be15-a5ce82a98a68" containerID="50f107ac2854a1d0ec109dbae8e32f9e0859bb93052aecf60c8bbfd329e571d0" exitCode=60
Oct 30 01:15:06.097629 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:06.096710    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-01215f-29888715-bxdtr" event={"ID":"92776c04-7f5b-457a-be15-a5ce82a98a68","Type":"ContainerDied","Data":"50f107ac2854a1d0ec109dbae8e32f9e0859bb93052aecf60c8bbfd329e571d0"}
Oct 30 01:15:06.097629 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:06.097165    5002 scope.go:122] "RemoveContainer" containerID="50f107ac2854a1d0ec109dbae8e32f9e0859bb93052aecf60c8bbfd329e571d0"
Oct 30 01:15:06.097629 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:15:06.097561    5002 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-01215f-29888715-bxdtr_odh-ai-gateway-infra(92776c04-7f5b-457a-be15-a5ce82a98a68)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-01215f-29888715-bxdtr" podUID="92776c04-7f5b-457a-be15-a5ce82a98a68"
Oct 30 01:15:06.105859 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:06.105838    5002 scope.go:122] "RemoveContainer" containerID="8e6c0dba751ddf2b317a7d745e260cc2e98d0f54da411785686c1fa1afd769b9"
Oct 30 01:15:06.108692 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:06.108666    5002 generic.go:356] "Generic (PLEG): container finished" podID="fd232d4c-4d2b-4137-bf83-b3e140fba4a6" containerID="d9ecb54895eba5673f1094a2001c1141074d7cc26c4c0613d5f021fd49395ac9" exitCode=60
Oct 30 01:15:06.108802 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:06.108748    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-a5e9ab-29888715-gltwb" event={"ID":"fd232d4c-4d2b-4137-bf83-b3e140fba4a6","Type":"ContainerDied","Data":"d9ecb54895eba5673f1094a2001c1141074d7cc26c4c0613d5f021fd49395ac9"}
Oct 30 01:15:06.109310 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:06.109111    5002 scope.go:122] "RemoveContainer" containerID="d9ecb54895eba5673f1094a2001c1141074d7cc26c4c0613d5f021fd49395ac9"
Oct 30 01:15:06.109633 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:15:06.109609    5002 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w5-a5e9ab-29888715-gltwb_odh-ai-gateway-infra(fd232d4c-4d2b-4137-bf83-b3e140fba4a6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-a5e9ab-29888715-gltwb" podUID="fd232d4c-4d2b-4137-bf83-b3e140fba4a6"
Oct 30 01:15:06.112305 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:06.112280    5002 generic.go:356] "Generic (PLEG): container finished" podID="6b614db7-0a42-4251-a93b-80ee2f8d3fbe" containerID="c8b8d15d9c8d14dcf01fdf342d673d51d66ffbd0b07c8a1bbdac7924670a2c61" exitCode=60
Oct 30 01:15:06.112487 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:06.112464    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-009e78-29888715-526wv" event={"ID":"6b614db7-0a42-4251-a93b-80ee2f8d3fbe","Type":"ContainerDied","Data":"c8b8d15d9c8d14dcf01fdf342d673d51d66ffbd0b07c8a1bbdac7924670a2c61"}
Oct 30 01:15:06.112648 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:06.112634    5002 scope.go:122] "RemoveContainer" containerID="c8b8d15d9c8d14dcf01fdf342d673d51d66ffbd0b07c8a1bbdac7924670a2c61"
Oct 30 01:15:06.112885 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:15:06.112865    5002 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-009e78-29888715-526wv_odh-ai-gateway-infra(6b614db7-0a42-4251-a93b-80ee2f8d3fbe)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-009e78-29888715-526wv" podUID="6b614db7-0a42-4251-a93b-80ee2f8d3fbe"
Oct 30 01:15:06.115139 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:06.115121    5002 generic.go:356] "Generic (PLEG): container finished" podID="05c750c5-39da-44a2-8414-3d7ffe09812d" containerID="472ce76d6c1beb7fc03522ef5e89b6db866188096422af5fcf1ed1e16a23aef1" exitCode=0
Oct 30 01:15:06.115287 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:06.115275    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-k66r8" event={"ID":"05c750c5-39da-44a2-8414-3d7ffe09812d","Type":"ContainerDied","Data":"472ce76d6c1beb7fc03522ef5e89b6db866188096422af5fcf1ed1e16a23aef1"}
Oct 30 01:15:06.115395 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:06.115366    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-k66r8" event={"ID":"05c750c5-39da-44a2-8414-3d7ffe09812d","Type":"ContainerDied","Data":"acecf27eec63f68d538ec430c91d8f9a860785efc088ed6a54b8fa03ecdb920c"}
Oct 30 01:15:06.115531 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:06.115520    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-k66r8"
Oct 30 01:15:06.126289 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:06.126259    5002 scope.go:122] "RemoveContainer" containerID="9d301926783affaeb89891f9623c6463f648eb1eea2441689df5083df6aad3d7"
Oct 30 01:15:06.130478 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:06.130449    5002 generic.go:356] "Generic (PLEG): container finished" podID="663c6827-07ce-4de7-8d82-033f8b60c129" containerID="df65a7038cdf1664a4852454ee63255addfe0f962c8d1342bd29de14c22224c6" exitCode=60
Oct 30 01:15:06.130829 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:06.130619    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a7ca1f-29888715-r8jhp" event={"ID":"663c6827-07ce-4de7-8d82-033f8b60c129","Type":"ContainerDied","Data":"df65a7038cdf1664a4852454ee63255addfe0f962c8d1342bd29de14c22224c6"}
Oct 30 01:15:06.131115 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:06.131059    5002 scope.go:122] "RemoveContainer" containerID="df65a7038cdf1664a4852454ee63255addfe0f962c8d1342bd29de14c22224c6"
Oct 30 01:15:06.131523 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:15:06.131497    5002 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-a7ca1f-29888715-r8jhp_odh-ai-gateway-infra(663c6827-07ce-4de7-8d82-033f8b60c129)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a7ca1f-29888715-r8jhp" podUID="663c6827-07ce-4de7-8d82-033f8b60c129"
Oct 30 01:15:06.133323 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:06.133308    5002 scope.go:122] "RemoveContainer" containerID="23bd2ab9aaff2196c2ce41d1f1b9a49b6c46d6e2b090fedf9acb902d803f01f8"
Oct 30 01:15:06.133757 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:15:06.133738    5002 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-beeba4-29888715-5t886_odh-ai-gateway-infra(55f94866-dc95-4ee1-b634-a3c434226a03)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-beeba4-29888715-5t886" podUID="55f94866-dc95-4ee1-b634-a3c434226a03"
Oct 30 01:15:06.137114 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:06.137089    5002 generic.go:356] "Generic (PLEG): container finished" podID="5834beef-f5d2-4d8f-94da-aa3a86d692cf" containerID="17e0a6c2cf3abe78ce7d4a834d12509990087dd9538d7dad7bdab6421ec72ff3" exitCode=60
Oct 30 01:15:06.137304 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:06.137283    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f64dc0-29888715-75wpl" event={"ID":"5834beef-f5d2-4d8f-94da-aa3a86d692cf","Type":"ContainerDied","Data":"17e0a6c2cf3abe78ce7d4a834d12509990087dd9538d7dad7bdab6421ec72ff3"}
Oct 30 01:15:06.137825 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:06.137806    5002 scope.go:122] "RemoveContainer" containerID="17e0a6c2cf3abe78ce7d4a834d12509990087dd9538d7dad7bdab6421ec72ff3"
Oct 30 01:15:06.138354 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:15:06.138331    5002 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-f64dc0-29888715-75wpl_odh-ai-gateway-infra(5834beef-f5d2-4d8f-94da-aa3a86d692cf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f64dc0-29888715-75wpl" podUID="5834beef-f5d2-4d8f-94da-aa3a86d692cf"
Oct 30 01:15:06.143540 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:06.143513    5002 generic.go:356] "Generic (PLEG): container finished" podID="91ed35be-dc11-4d3f-91bc-f4a7b4c7be28" containerID="c1334c2e13aa673e9a0b651b6144991b77895f8f2486c3bdeec62a76950b213b" exitCode=60
Oct 30 01:15:06.143637 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:06.143558    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-d8b2c6-29888715-rqz6d" event={"ID":"91ed35be-dc11-4d3f-91bc-f4a7b4c7be28","Type":"ContainerDied","Data":"c1334c2e13aa673e9a0b651b6144991b77895f8f2486c3bdeec62a76950b213b"}
Oct 30 01:15:06.143934 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:06.143912    5002 scope.go:122] "RemoveContainer" containerID="c1334c2e13aa673e9a0b651b6144991b77895f8f2486c3bdeec62a76950b213b"
Oct 30 01:15:06.144312 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:15:06.144284    5002 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-d8b2c6-29888715-rqz6d_odh-ai-gateway-infra(91ed35be-dc11-4d3f-91bc-f4a7b4c7be28)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-d8b2c6-29888715-rqz6d" podUID="91ed35be-dc11-4d3f-91bc-f4a7b4c7be28"
Oct 30 01:15:06.159767 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:06.157313    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05c750c5-39da-44a2-8414-3d7ffe09812d-istio-token" pod "05c750c5-39da-44a2-8414-3d7ffe09812d" (UID: "05c750c5-39da-44a2-8414-3d7ffe09812d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:15:06.159767 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:06.158536    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05c750c5-39da-44a2-8414-3d7ffe09812d-kube-api-access-zzwvj" pod "05c750c5-39da-44a2-8414-3d7ffe09812d" (UID: "05c750c5-39da-44a2-8414-3d7ffe09812d"). InnerVolumeSpecName "kube-api-access-zzwvj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:15:06.159767 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:06.158632    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/05c750c5-39da-44a2-8414-3d7ffe09812d-istio-podinfo" pod "05c750c5-39da-44a2-8414-3d7ffe09812d" (UID: "05c750c5-39da-44a2-8414-3d7ffe09812d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:15:06.159767 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:06.158741    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05c750c5-39da-44a2-8414-3d7ffe09812d-istio-envoy" pod "05c750c5-39da-44a2-8414-3d7ffe09812d" (UID: "05c750c5-39da-44a2-8414-3d7ffe09812d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:15:06.211911 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:06.211087    5002 scope.go:122] "RemoveContainer" containerID="48ea907220398868496390e97a0e50fbf945de8bbdd86090b2445175d3824950"
Oct 30 01:15:06.214453 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:06.214313    5002 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05c750c5-39da-44a2-8414-3d7ffe09812d-istio-podinfo\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:15:06.214453 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:06.214340    5002 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05c750c5-39da-44a2-8414-3d7ffe09812d-istio-token\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:15:06.214453 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:06.214352    5002 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05c750c5-39da-44a2-8414-3d7ffe09812d-istio-envoy\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:15:06.214453 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:06.214365    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zzwvj\" (UniqueName: \"kubernetes.io/projected/05c750c5-39da-44a2-8414-3d7ffe09812d-kube-api-access-zzwvj\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:15:06.241906 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:06.241654    5002 scope.go:122] "RemoveContainer" containerID="9fc1fbeb226aea90918da930b194cab4e1513f598dec1e8cca1943fabb8b2400"
Oct 30 01:15:06.286659 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:06.286007    5002 scope.go:122] "RemoveContainer" containerID="b66b3acd682ee0f2c3987550bd4391fd71d30946fab6f984b55fab662b456f2d"
Oct 30 01:15:06.306813 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:06.306662    5002 scope.go:122] "RemoveContainer" containerID="283abc0b3b05a3b7d3820e5f84225649121cd9c0a4af84f3004f8dd3b5a46233"
Oct 30 01:15:06.342647 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:06.342523    5002 scope.go:122] "RemoveContainer" containerID="893310bdd40578c1d1b5d9ccf2d71f09aa50fe3699eeba7883debc9b51d555a9"
Oct 30 01:15:06.370311 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:06.370005    5002 scope.go:122] "RemoveContainer" containerID="4be3160acaef3b0490800122facd369e9141ea5371754bab5892c053d34f86cd"
Oct 30 01:15:06.402523 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:06.402494    5002 scope.go:122] "RemoveContainer" containerID="472ce76d6c1beb7fc03522ef5e89b6db866188096422af5fcf1ed1e16a23aef1"
Oct 30 01:15:06.636415 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:06.599950    5002 scope.go:122] "RemoveContainer" containerID="472ce76d6c1beb7fc03522ef5e89b6db866188096422af5fcf1ed1e16a23aef1"
Oct 30 01:15:06.690404 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:15:06.639092    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"472ce76d6c1beb7fc03522ef5e89b6db866188096422af5fcf1ed1e16a23aef1\": container with ID starting with 472ce76d6c1beb7fc03522ef5e89b6db866188096422af5fcf1ed1e16a23aef1 not found: ID does not exist" containerID="472ce76d6c1beb7fc03522ef5e89b6db866188096422af5fcf1ed1e16a23aef1"
Oct 30 01:15:06.690404 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:06.639147    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"472ce76d6c1beb7fc03522ef5e89b6db866188096422af5fcf1ed1e16a23aef1"} err="failed to get container status \"472ce76d6c1beb7fc03522ef5e89b6db866188096422af5fcf1ed1e16a23aef1\": rpc error: code = NotFound desc = could not find container \"472ce76d6c1beb7fc03522ef5e89b6db866188096422af5fcf1ed1e16a23aef1\": container with ID starting with 472ce76d6c1beb7fc03522ef5e89b6db866188096422af5fcf1ed1e16a23aef1 not found: ID does not exist"
Oct 30 01:15:06.690404 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:06.639173    5002 scope.go:122] "RemoveContainer" containerID="70378e3017440afc8f255b76be79b9292922351096bd2cc260a971233a04fed7"
Oct 30 01:15:06.843652 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:06.843604    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9bb1708e-878e-461f-9294-265673c31b04" path="/var/lib/kubelet/pods/9bb1708e-878e-461f-9294-265673c31b04/volumes"
Oct 30 01:15:06.847440 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:06.847348    5002 scope.go:122] "RemoveContainer" containerID="08c79ebb8047d1a14a60b532dec758ad9dd5e46f36f4f687b5ef2f19177d6969"
Oct 30 01:15:06.880537 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:06.870213    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d8b2c6-8ndxw"]
Oct 30 01:15:06.950403 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:06.946945    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d8b2c6-8ndxw"]
Oct 30 01:15:07.037121 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:15:07.037055    5002 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71d53e61_57bc_4d66_a029_965ea868d660.slice/crio-conmon-905a9de95281c6e73f1e04fa4338036abc144de4fa8bc65e61f69a7f66b4d737.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05c750c5_39da_44a2_8414_3d7ffe09812d.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaba84a8f_bfa1_4a2e_9569_8194bed6ec61.slice/crio-conmon-fe7a45761c593f0a164c74318faf5ae1530d8823c3e8d4021d1dbf53be6a04ac.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71d53e61_57bc_4d66_a029_965ea868d660.slice/crio-905a9de95281c6e73f1e04fa4338036abc144de4fa8bc65e61f69a7f66b4d737.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8dcf8782_f46c_4c0f_9133_793017d66a37.slice/crio-conmon-b4a475af6879a21a3dc26a89fdb0efe18a0a29c561d79be4c06cc7b9deee6190.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05c750c5_39da_44a2_8414_3d7ffe09812d.slice/crio-acecf27eec63f68d538ec430c91d8f9a860785efc088ed6a54b8fa03ecdb920c\": RecentStats: unable to find data in memory cache]"
Oct 30 01:15:07.039175 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:15:07.039008    5002 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8dcf8782_f46c_4c0f_9133_793017d66a37.slice/crio-conmon-b4a475af6879a21a3dc26a89fdb0efe18a0a29c561d79be4c06cc7b9deee6190.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8dcf8782_f46c_4c0f_9133_793017d66a37.slice/crio-b4a475af6879a21a3dc26a89fdb0efe18a0a29c561d79be4c06cc7b9deee6190.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05c750c5_39da_44a2_8414_3d7ffe09812d.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71d53e61_57bc_4d66_a029_965ea868d660.slice/crio-conmon-905a9de95281c6e73f1e04fa4338036abc144de4fa8bc65e61f69a7f66b4d737.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05c750c5_39da_44a2_8414_3d7ffe09812d.slice/crio-acecf27eec63f68d538ec430c91d8f9a860785efc088ed6a54b8fa03ecdb920c\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71d53e61_57bc_4d66_a029_965ea868d660.slice/crio-905a9de95281c6e73f1e04fa4338036abc144de4fa8bc65e61f69a7f66b4d737.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaba84a8f_bfa1_4a2e_9569_8194bed6ec61.slice/crio-conmon-fe7a45761c593f0a164c74318faf5ae1530d8823c3e8d4021d1dbf53be6a04ac.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 01:15:07.070187 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:07.070156    5002 scope.go:122] "RemoveContainer" containerID="65500fa2527ee88dac31a6dbf0f10d8632de184f5b78218ef15a1315e46706b6"
Oct 30 01:15:07.087916 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:07.087847    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-k66r8"]
Oct 30 01:15:07.095443 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:07.095408    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-k66r8"]
Oct 30 01:15:07.116524 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:07.116475    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-f6d5b8-openshift-default-68ddb87d77-n4kjt"]
Oct 30 01:15:07.117925 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:07.117894    5002 cpu_manager.go:414] "containerMap: removing container" podUID="05c750c5-39da-44a2-8414-3d7ffe09812d" containerName="istio-proxy"
Oct 30 01:15:07.118100 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:07.118084    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05c750c5-39da-44a2-8414-3d7ffe09812d" containerName="istio-proxy"
Oct 30 01:15:07.118197 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:07.118186    5002 cpu_manager.go:414] "containerMap: removing container" podUID="9bb1708e-878e-461f-9294-265673c31b04" containerName="istio-proxy"
Oct 30 01:15:07.118279 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:07.118266    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9bb1708e-878e-461f-9294-265673c31b04" containerName="istio-proxy"
Oct 30 01:15:07.118605 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:07.118590    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="05c750c5-39da-44a2-8414-3d7ffe09812d" containerName="istio-proxy"
Oct 30 01:15:07.118685 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:07.118677    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="9bb1708e-878e-461f-9294-265673c31b04" containerName="istio-proxy"
Oct 30 01:15:07.124190 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:07.124170    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f6d5b8-openshift-default-68ddb87d77-n4kjt"
Oct 30 01:15:07.154357 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:07.154254    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-f6d5b8-openshift-default-68ddb87d77-n4kjt"]
Oct 30 01:15:07.154980 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:07.154957    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-f6d5b8-openshift-default-dockercfg-8wlnq\""
Oct 30 01:15:07.182784 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:07.182732    5002 scope.go:122] "RemoveContainer" containerID="df65a7038cdf1664a4852454ee63255addfe0f962c8d1342bd29de14c22224c6"
Oct 30 01:15:07.185111 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:15:07.185071    5002 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-a7ca1f-29888715-r8jhp_odh-ai-gateway-infra(663c6827-07ce-4de7-8d82-033f8b60c129)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a7ca1f-29888715-r8jhp" podUID="663c6827-07ce-4de7-8d82-033f8b60c129"
Oct 30 01:15:07.188341 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:07.188304    5002 generic.go:356] "Generic (PLEG): container finished" podID="8dcf8782-f46c-4c0f-9133-793017d66a37" containerID="b4a475af6879a21a3dc26a89fdb0efe18a0a29c561d79be4c06cc7b9deee6190" exitCode=60
Oct 30 01:15:07.188517 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:07.188468    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5f387c-29888715-qgmjp" event={"ID":"8dcf8782-f46c-4c0f-9133-793017d66a37","Type":"ContainerDied","Data":"b4a475af6879a21a3dc26a89fdb0efe18a0a29c561d79be4c06cc7b9deee6190"}
Oct 30 01:15:07.188614 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:07.188538    5002 scope.go:122] "RemoveContainer" containerID="3bbb94d57e7fc25edb6403e92d9c36e06d9e35d87e584e26786ede4d7ebf2db1"
Oct 30 01:15:07.189074 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:07.189050    5002 scope.go:122] "RemoveContainer" containerID="b4a475af6879a21a3dc26a89fdb0efe18a0a29c561d79be4c06cc7b9deee6190"
Oct 30 01:15:07.190111 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:15:07.190083    5002 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-5f387c-29888715-qgmjp_odh-ai-gateway-infra(8dcf8782-f46c-4c0f-9133-793017d66a37)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5f387c-29888715-qgmjp" podUID="8dcf8782-f46c-4c0f-9133-793017d66a37"
Oct 30 01:15:07.205067 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:07.205043    5002 scope.go:122] "RemoveContainer" containerID="17e0a6c2cf3abe78ce7d4a834d12509990087dd9538d7dad7bdab6421ec72ff3"
Oct 30 01:15:07.205601 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:15:07.205563    5002 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-f64dc0-29888715-75wpl_odh-ai-gateway-infra(5834beef-f5d2-4d8f-94da-aa3a86d692cf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f64dc0-29888715-75wpl" podUID="5834beef-f5d2-4d8f-94da-aa3a86d692cf"
Oct 30 01:15:07.211805 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:07.211783    5002 scope.go:122] "RemoveContainer" containerID="6d02e29347533557fb6e5ea114602aaf38092b749b7366936bff8383dbc40473"
Oct 30 01:15:07.212409 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:15:07.212352    5002 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-11f521-29888715-fj8kn_odh-ai-gateway-infra(248fa0d9-fe68-4ba7-82c3-d7911c9b8070)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-11f521-29888715-fj8kn" podUID="248fa0d9-fe68-4ba7-82c3-d7911c9b8070"
Oct 30 01:15:07.215670 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:07.215648    5002 scope.go:122] "RemoveContainer" containerID="3d6d154d5ea078834d229b13ea707f618f9040d1d25101fcc00ef3bde333470f"
Oct 30 01:15:07.215945 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:15:07.215921    5002 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w6-7f92f6-29888715-d7v2x_odh-ai-gateway-infra(34ced72e-384a-4e43-90b7-2f907956c2c1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-7f92f6-29888715-d7v2x" podUID="34ced72e-384a-4e43-90b7-2f907956c2c1"
Oct 30 01:15:07.218347 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:07.218326    5002 scope.go:122] "RemoveContainer" containerID="14aa48c03a3e6d7be42a1b03c02b8ea6aef907c5c8f22e72936cd38a6703d033"
Oct 30 01:15:07.219215 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:15:07.219186    5002 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-6d44c2-29888715-d8zbn_odh-ai-gateway-infra(07d6239e-5b00-4c4a-b8dd-d935c4c44ed8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-6d44c2-29888715-d8zbn" podUID="07d6239e-5b00-4c4a-b8dd-d935c4c44ed8"
Oct 30 01:15:07.221702 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:07.221684    5002 scope.go:122] "RemoveContainer" containerID="23542a46cd867d1aabf74ab0dacd3ead9b877c819c387b235a8cdc14ad648201"
Oct 30 01:15:07.221984 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:15:07.221946    5002 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-29be73-29888715-p2ctj_odh-ai-gateway-infra(5d965e6c-4e56-4dff-918d-c93b9832b72f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-29be73-29888715-p2ctj" podUID="5d965e6c-4e56-4dff-918d-c93b9832b72f"
Oct 30 01:15:07.224418 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:07.224396    5002 scope.go:122] "RemoveContainer" containerID="2255fcbba7d31cf86be4719a7884880cbcf52c8f0d976dcd99ac3033e505733b"
Oct 30 01:15:07.224703 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:15:07.224678    5002 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-00e07a-29888715-8dq5b_odh-ai-gateway-infra(82b3d01a-515b-43fe-b05d-ae6d90c678a3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-00e07a-29888715-8dq5b" podUID="82b3d01a-515b-43fe-b05d-ae6d90c678a3"
Oct 30 01:15:07.226327 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:07.226306    5002 scope.go:122] "RemoveContainer" containerID="645814b0bf1ed27656feb00c8e15dfcf1b8264ab84d39187925274efbd6949a3"
Oct 30 01:15:07.226967 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:15:07.226942    5002 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-cefb27-29888715-9524l_odh-ai-gateway-infra(7568f0d2-0dfb-47b9-ab97-d06d3c985d90)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-cefb27-29888715-9524l" podUID="7568f0d2-0dfb-47b9-ab97-d06d3c985d90"
Oct 30 01:15:07.228831 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:07.228810    5002 scope.go:122] "RemoveContainer" containerID="50f107ac2854a1d0ec109dbae8e32f9e0859bb93052aecf60c8bbfd329e571d0"
Oct 30 01:15:07.229014 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:15:07.228994    5002 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-01215f-29888715-bxdtr_odh-ai-gateway-infra(92776c04-7f5b-457a-be15-a5ce82a98a68)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-01215f-29888715-bxdtr" podUID="92776c04-7f5b-457a-be15-a5ce82a98a68"
Oct 30 01:15:07.232343 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:07.231005    5002 scope.go:122] "RemoveContainer" containerID="d9ecb54895eba5673f1094a2001c1141074d7cc26c4c0613d5f021fd49395ac9"
Oct 30 01:15:07.232343 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:15:07.231242    5002 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w5-a5e9ab-29888715-gltwb_odh-ai-gateway-infra(fd232d4c-4d2b-4137-bf83-b3e140fba4a6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-a5e9ab-29888715-gltwb" podUID="fd232d4c-4d2b-4137-bf83-b3e140fba4a6"
Oct 30 01:15:07.233065 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:07.233045    5002 scope.go:122] "RemoveContainer" containerID="c8b8d15d9c8d14dcf01fdf342d673d51d66ffbd0b07c8a1bbdac7924670a2c61"
Oct 30 01:15:07.233249 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:15:07.233220    5002 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-009e78-29888715-526wv_odh-ai-gateway-infra(6b614db7-0a42-4251-a93b-80ee2f8d3fbe)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-009e78-29888715-526wv" podUID="6b614db7-0a42-4251-a93b-80ee2f8d3fbe"
Oct 30 01:15:07.235535 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:07.234919    5002 generic.go:356] "Generic (PLEG): container finished" podID="aba84a8f-bfa1-4a2e-9569-8194bed6ec61" containerID="fe7a45761c593f0a164c74318faf5ae1530d8823c3e8d4021d1dbf53be6a04ac" exitCode=60
Oct 30 01:15:07.235535 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:07.234983    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-021d6d-29888715-cdjpr" event={"ID":"aba84a8f-bfa1-4a2e-9569-8194bed6ec61","Type":"ContainerDied","Data":"fe7a45761c593f0a164c74318faf5ae1530d8823c3e8d4021d1dbf53be6a04ac"}
Oct 30 01:15:07.235535 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:07.235018    5002 scope.go:122] "RemoveContainer" containerID="23d86e27deee3065b5680f02c84e0fb0e4e386164c512e89f5f2fea9e738f8c2"
Oct 30 01:15:07.235535 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:07.235258    5002 scope.go:122] "RemoveContainer" containerID="fe7a45761c593f0a164c74318faf5ae1530d8823c3e8d4021d1dbf53be6a04ac"
Oct 30 01:15:07.235535 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:15:07.235492    5002 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-021d6d-29888715-cdjpr_odh-ai-gateway-infra(aba84a8f-bfa1-4a2e-9569-8194bed6ec61)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-021d6d-29888715-cdjpr" podUID="aba84a8f-bfa1-4a2e-9569-8194bed6ec61"
Oct 30 01:15:07.239283 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:07.239263    5002 generic.go:356] "Generic (PLEG): container finished" podID="71d53e61-57bc-4d66-a029-965ea868d660" containerID="905a9de95281c6e73f1e04fa4338036abc144de4fa8bc65e61f69a7f66b4d737" exitCode=60
Oct 30 01:15:07.239361 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:07.239298    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-48df96-29888715-nc8tn" event={"ID":"71d53e61-57bc-4d66-a029-965ea868d660","Type":"ContainerDied","Data":"905a9de95281c6e73f1e04fa4338036abc144de4fa8bc65e61f69a7f66b4d737"}
Oct 30 01:15:07.239650 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:07.239632    5002 scope.go:122] "RemoveContainer" containerID="905a9de95281c6e73f1e04fa4338036abc144de4fa8bc65e61f69a7f66b4d737"
Oct 30 01:15:07.239945 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:15:07.239923    5002 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-48df96-29888715-nc8tn_odh-ai-gateway-infra(71d53e61-57bc-4d66-a029-965ea868d660)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-48df96-29888715-nc8tn" podUID="71d53e61-57bc-4d66-a029-965ea868d660"
Oct 30 01:15:07.246944 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:07.246922    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7d8z4"]
Oct 30 01:15:07.253529 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:07.252925    5002 scope.go:122] "RemoveContainer" containerID="d2f304fb1b84d726eadbb7f5b73b32114943f7ffc2f9627210db3035884b2db8"
Oct 30 01:15:07.262492 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:07.262459    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45d94cd6-d704-48c7-b1fb-e97c89e7d74b-istiod-ca-cert\") pod \"e2e-ipp-a-f6d5b8-openshift-default-68ddb87d77-n4kjt\" (UID: \"45d94cd6-d704-48c7-b1fb-e97c89e7d74b\") " pod="openshift-ingress/e2e-ipp-a-f6d5b8-openshift-default-68ddb87d77-n4kjt"
Oct 30 01:15:07.262606 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:07.262584    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45d94cd6-d704-48c7-b1fb-e97c89e7d74b-workload-certs\") pod \"e2e-ipp-a-f6d5b8-openshift-default-68ddb87d77-n4kjt\" (UID: \"45d94cd6-d704-48c7-b1fb-e97c89e7d74b\") " pod="openshift-ingress/e2e-ipp-a-f6d5b8-openshift-default-68ddb87d77-n4kjt"
Oct 30 01:15:07.262685 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:07.262620    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nkm2\" (UniqueName: \"kubernetes.io/projected/45d94cd6-d704-48c7-b1fb-e97c89e7d74b-kube-api-access-9nkm2\") pod \"e2e-ipp-a-f6d5b8-openshift-default-68ddb87d77-n4kjt\" (UID: \"45d94cd6-d704-48c7-b1fb-e97c89e7d74b\") " pod="openshift-ingress/e2e-ipp-a-f6d5b8-openshift-default-68ddb87d77-n4kjt"
Oct 30 01:15:07.262685 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:07.262657    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45d94cd6-d704-48c7-b1fb-e97c89e7d74b-credential-socket\") pod \"e2e-ipp-a-f6d5b8-openshift-default-68ddb87d77-n4kjt\" (UID: \"45d94cd6-d704-48c7-b1fb-e97c89e7d74b\") " pod="openshift-ingress/e2e-ipp-a-f6d5b8-openshift-default-68ddb87d77-n4kjt"
Oct 30 01:15:07.262798 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:07.262763    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45d94cd6-d704-48c7-b1fb-e97c89e7d74b-istio-data\") pod \"e2e-ipp-a-f6d5b8-openshift-default-68ddb87d77-n4kjt\" (UID: \"45d94cd6-d704-48c7-b1fb-e97c89e7d74b\") " pod="openshift-ingress/e2e-ipp-a-f6d5b8-openshift-default-68ddb87d77-n4kjt"
Oct 30 01:15:07.262798 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:07.262791    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45d94cd6-d704-48c7-b1fb-e97c89e7d74b-istio-token\") pod \"e2e-ipp-a-f6d5b8-openshift-default-68ddb87d77-n4kjt\" (UID: \"45d94cd6-d704-48c7-b1fb-e97c89e7d74b\") " pod="openshift-ingress/e2e-ipp-a-f6d5b8-openshift-default-68ddb87d77-n4kjt"
Oct 30 01:15:07.263229 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:07.263209    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45d94cd6-d704-48c7-b1fb-e97c89e7d74b-istio-envoy\") pod \"e2e-ipp-a-f6d5b8-openshift-default-68ddb87d77-n4kjt\" (UID: \"45d94cd6-d704-48c7-b1fb-e97c89e7d74b\") " pod="openshift-ingress/e2e-ipp-a-f6d5b8-openshift-default-68ddb87d77-n4kjt"
Oct 30 01:15:07.263424 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:07.263394    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45d94cd6-d704-48c7-b1fb-e97c89e7d74b-workload-socket\") pod \"e2e-ipp-a-f6d5b8-openshift-default-68ddb87d77-n4kjt\" (UID: \"45d94cd6-d704-48c7-b1fb-e97c89e7d74b\") " pod="openshift-ingress/e2e-ipp-a-f6d5b8-openshift-default-68ddb87d77-n4kjt"
Oct 30 01:15:07.263577 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:07.263559    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45d94cd6-d704-48c7-b1fb-e97c89e7d74b-istio-podinfo\") pod \"e2e-ipp-a-f6d5b8-openshift-default-68ddb87d77-n4kjt\" (UID: \"45d94cd6-d704-48c7-b1fb-e97c89e7d74b\") " pod="openshift-ingress/e2e-ipp-a-f6d5b8-openshift-default-68ddb87d77-n4kjt"
Oct 30 01:15:07.366392 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:07.366335    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45d94cd6-d704-48c7-b1fb-e97c89e7d74b-istio-envoy\") pod \"e2e-ipp-a-f6d5b8-openshift-default-68ddb87d77-n4kjt\" (UID: \"45d94cd6-d704-48c7-b1fb-e97c89e7d74b\") " pod="openshift-ingress/e2e-ipp-a-f6d5b8-openshift-default-68ddb87d77-n4kjt"
Oct 30 01:15:07.367533 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:07.367510    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45d94cd6-d704-48c7-b1fb-e97c89e7d74b-workload-socket\") pod \"e2e-ipp-a-f6d5b8-openshift-default-68ddb87d77-n4kjt\" (UID: \"45d94cd6-d704-48c7-b1fb-e97c89e7d74b\") " pod="openshift-ingress/e2e-ipp-a-f6d5b8-openshift-default-68ddb87d77-n4kjt"
Oct 30 01:15:07.367929 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:07.367906    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45d94cd6-d704-48c7-b1fb-e97c89e7d74b-istio-podinfo\") pod \"e2e-ipp-a-f6d5b8-openshift-default-68ddb87d77-n4kjt\" (UID: \"45d94cd6-d704-48c7-b1fb-e97c89e7d74b\") " pod="openshift-ingress/e2e-ipp-a-f6d5b8-openshift-default-68ddb87d77-n4kjt"
Oct 30 01:15:07.368642 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:07.368015    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45d94cd6-d704-48c7-b1fb-e97c89e7d74b-workload-socket\") pod \"e2e-ipp-a-f6d5b8-openshift-default-68ddb87d77-n4kjt\" (UID: \"45d94cd6-d704-48c7-b1fb-e97c89e7d74b\") " pod="openshift-ingress/e2e-ipp-a-f6d5b8-openshift-default-68ddb87d77-n4kjt"
Oct 30 01:15:07.368642 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:07.368149    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45d94cd6-d704-48c7-b1fb-e97c89e7d74b-istiod-ca-cert\") pod \"e2e-ipp-a-f6d5b8-openshift-default-68ddb87d77-n4kjt\" (UID: \"45d94cd6-d704-48c7-b1fb-e97c89e7d74b\") " pod="openshift-ingress/e2e-ipp-a-f6d5b8-openshift-default-68ddb87d77-n4kjt"
Oct 30 01:15:07.371448 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:07.368937    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45d94cd6-d704-48c7-b1fb-e97c89e7d74b-workload-certs\") pod \"e2e-ipp-a-f6d5b8-openshift-default-68ddb87d77-n4kjt\" (UID: \"45d94cd6-d704-48c7-b1fb-e97c89e7d74b\") " pod="openshift-ingress/e2e-ipp-a-f6d5b8-openshift-default-68ddb87d77-n4kjt"
Oct 30 01:15:07.371448 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:07.368956    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45d94cd6-d704-48c7-b1fb-e97c89e7d74b-istiod-ca-cert\") pod \"e2e-ipp-a-f6d5b8-openshift-default-68ddb87d77-n4kjt\" (UID: \"45d94cd6-d704-48c7-b1fb-e97c89e7d74b\") " pod="openshift-ingress/e2e-ipp-a-f6d5b8-openshift-default-68ddb87d77-n4kjt"
Oct 30 01:15:07.371448 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:07.368990    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nkm2\" (UniqueName: \"kubernetes.io/projected/45d94cd6-d704-48c7-b1fb-e97c89e7d74b-kube-api-access-9nkm2\") pod \"e2e-ipp-a-f6d5b8-openshift-default-68ddb87d77-n4kjt\" (UID: \"45d94cd6-d704-48c7-b1fb-e97c89e7d74b\") " pod="openshift-ingress/e2e-ipp-a-f6d5b8-openshift-default-68ddb87d77-n4kjt"
Oct 30 01:15:07.371448 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:07.369067    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45d94cd6-d704-48c7-b1fb-e97c89e7d74b-credential-socket\") pod \"e2e-ipp-a-f6d5b8-openshift-default-68ddb87d77-n4kjt\" (UID: \"45d94cd6-d704-48c7-b1fb-e97c89e7d74b\") " pod="openshift-ingress/e2e-ipp-a-f6d5b8-openshift-default-68ddb87d77-n4kjt"
Oct 30 01:15:07.371448 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:07.369121    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45d94cd6-d704-48c7-b1fb-e97c89e7d74b-istio-data\") pod \"e2e-ipp-a-f6d5b8-openshift-default-68ddb87d77-n4kjt\" (UID: \"45d94cd6-d704-48c7-b1fb-e97c89e7d74b\") " pod="openshift-ingress/e2e-ipp-a-f6d5b8-openshift-default-68ddb87d77-n4kjt"
Oct 30 01:15:07.371448 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:07.369145    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45d94cd6-d704-48c7-b1fb-e97c89e7d74b-istio-token\") pod \"e2e-ipp-a-f6d5b8-openshift-default-68ddb87d77-n4kjt\" (UID: \"45d94cd6-d704-48c7-b1fb-e97c89e7d74b\") " pod="openshift-ingress/e2e-ipp-a-f6d5b8-openshift-default-68ddb87d77-n4kjt"
Oct 30 01:15:07.371448 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:07.370134    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45d94cd6-d704-48c7-b1fb-e97c89e7d74b-credential-socket\") pod \"e2e-ipp-a-f6d5b8-openshift-default-68ddb87d77-n4kjt\" (UID: \"45d94cd6-d704-48c7-b1fb-e97c89e7d74b\") " pod="openshift-ingress/e2e-ipp-a-f6d5b8-openshift-default-68ddb87d77-n4kjt"
Oct 30 01:15:07.371448 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:07.370368    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45d94cd6-d704-48c7-b1fb-e97c89e7d74b-istio-data\") pod \"e2e-ipp-a-f6d5b8-openshift-default-68ddb87d77-n4kjt\" (UID: \"45d94cd6-d704-48c7-b1fb-e97c89e7d74b\") " pod="openshift-ingress/e2e-ipp-a-f6d5b8-openshift-default-68ddb87d77-n4kjt"
Oct 30 01:15:07.375423 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:07.372176    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45d94cd6-d704-48c7-b1fb-e97c89e7d74b-workload-certs\") pod \"e2e-ipp-a-f6d5b8-openshift-default-68ddb87d77-n4kjt\" (UID: \"45d94cd6-d704-48c7-b1fb-e97c89e7d74b\") " pod="openshift-ingress/e2e-ipp-a-f6d5b8-openshift-default-68ddb87d77-n4kjt"
Oct 30 01:15:07.390838 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:07.390774    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45d94cd6-d704-48c7-b1fb-e97c89e7d74b-istio-envoy\") pod \"e2e-ipp-a-f6d5b8-openshift-default-68ddb87d77-n4kjt\" (UID: \"45d94cd6-d704-48c7-b1fb-e97c89e7d74b\") " pod="openshift-ingress/e2e-ipp-a-f6d5b8-openshift-default-68ddb87d77-n4kjt"
Oct 30 01:15:07.393541 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:07.393516    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45d94cd6-d704-48c7-b1fb-e97c89e7d74b-istio-podinfo\") pod \"e2e-ipp-a-f6d5b8-openshift-default-68ddb87d77-n4kjt\" (UID: \"45d94cd6-d704-48c7-b1fb-e97c89e7d74b\") " pod="openshift-ingress/e2e-ipp-a-f6d5b8-openshift-default-68ddb87d77-n4kjt"
Oct 30 01:15:07.508510 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:07.452083    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45d94cd6-d704-48c7-b1fb-e97c89e7d74b-istio-token\") pod \"e2e-ipp-a-f6d5b8-openshift-default-68ddb87d77-n4kjt\" (UID: \"45d94cd6-d704-48c7-b1fb-e97c89e7d74b\") " pod="openshift-ingress/e2e-ipp-a-f6d5b8-openshift-default-68ddb87d77-n4kjt"
Oct 30 01:15:07.508510 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:07.478297    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nkm2\" (UniqueName: \"kubernetes.io/projected/45d94cd6-d704-48c7-b1fb-e97c89e7d74b-kube-api-access-9nkm2\") pod \"e2e-ipp-a-f6d5b8-openshift-default-68ddb87d77-n4kjt\" (UID: \"45d94cd6-d704-48c7-b1fb-e97c89e7d74b\") " pod="openshift-ingress/e2e-ipp-a-f6d5b8-openshift-default-68ddb87d77-n4kjt"
Oct 30 01:15:07.508510 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:07.480677    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f6d5b8-openshift-default-68ddb87d77-n4kjt"
Oct 30 01:15:07.818198 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:07.818167    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-d8b2c6-29888715-rqz6d"
Oct 30 01:15:07.895945 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:07.895905    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91ed35be-dc11-4d3f-91bc-f4a7b4c7be28-kube-api-access-hjvbg\" (UniqueName: \"kubernetes.io/projected/91ed35be-dc11-4d3f-91bc-f4a7b4c7be28-kube-api-access-hjvbg\") pod \"91ed35be-dc11-4d3f-91bc-f4a7b4c7be28\" (UID: \"91ed35be-dc11-4d3f-91bc-f4a7b4c7be28\") "
Oct 30 01:15:07.897187 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:07.896106    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91ed35be-dc11-4d3f-91bc-f4a7b4c7be28-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/91ed35be-dc11-4d3f-91bc-f4a7b4c7be28-maas-api-service-ca\") pod \"91ed35be-dc11-4d3f-91bc-f4a7b4c7be28\" (UID: \"91ed35be-dc11-4d3f-91bc-f4a7b4c7be28\") "
Oct 30 01:15:07.897187 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:07.897051    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91ed35be-dc11-4d3f-91bc-f4a7b4c7be28-maas-api-service-ca" pod "91ed35be-dc11-4d3f-91bc-f4a7b4c7be28" (UID: "91ed35be-dc11-4d3f-91bc-f4a7b4c7be28"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:15:07.900503 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:07.900475    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91ed35be-dc11-4d3f-91bc-f4a7b4c7be28-kube-api-access-hjvbg" pod "91ed35be-dc11-4d3f-91bc-f4a7b4c7be28" (UID: "91ed35be-dc11-4d3f-91bc-f4a7b4c7be28"). InnerVolumeSpecName "kube-api-access-hjvbg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:15:07.998532 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:07.998488    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/91ed35be-dc11-4d3f-91bc-f4a7b4c7be28-maas-api-service-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:15:07.998532 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:07.998528    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hjvbg\" (UniqueName: \"kubernetes.io/projected/91ed35be-dc11-4d3f-91bc-f4a7b4c7be28-kube-api-access-hjvbg\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:15:08.077273 ip-10-0-128-7 kubenswrapper[5002]: W1030 01:15:08.077238    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45d94cd6_d704_48c7_b1fb_e97c89e7d74b.slice/crio-ae77c938d542d367725032c1e41dd2e2ceda2595ad40ca36c1e148d9149d55d3 WatchSource:0}: Error finding container ae77c938d542d367725032c1e41dd2e2ceda2595ad40ca36c1e148d9149d55d3: Status 404 returned error can't find the container with id ae77c938d542d367725032c1e41dd2e2ceda2595ad40ca36c1e148d9149d55d3
Oct 30 01:15:08.079156 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:08.079122    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-f6d5b8-openshift-default-68ddb87d77-n4kjt"]
Oct 30 01:15:08.081208 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:08.081172    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:15:08.081305 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:08.081251    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:15:08.081344 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:08.081316    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:15:08.253105 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:08.253073    5002 scope.go:122] "RemoveContainer" containerID="fe7a45761c593f0a164c74318faf5ae1530d8823c3e8d4021d1dbf53be6a04ac"
Oct 30 01:15:08.253716 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:15:08.253683    5002 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-021d6d-29888715-cdjpr_odh-ai-gateway-infra(aba84a8f-bfa1-4a2e-9569-8194bed6ec61)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-021d6d-29888715-cdjpr" podUID="aba84a8f-bfa1-4a2e-9569-8194bed6ec61"
Oct 30 01:15:08.255081 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:08.255060    5002 scope.go:122] "RemoveContainer" containerID="905a9de95281c6e73f1e04fa4338036abc144de4fa8bc65e61f69a7f66b4d737"
Oct 30 01:15:08.255822 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:15:08.255795    5002 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-48df96-29888715-nc8tn_odh-ai-gateway-infra(71d53e61-57bc-4d66-a029-965ea868d660)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-48df96-29888715-nc8tn" podUID="71d53e61-57bc-4d66-a029-965ea868d660"
Oct 30 01:15:08.257879 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:08.257857    5002 scope.go:122] "RemoveContainer" containerID="b4a475af6879a21a3dc26a89fdb0efe18a0a29c561d79be4c06cc7b9deee6190"
Oct 30 01:15:08.258300 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:15:08.258271    5002 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-5f387c-29888715-qgmjp_odh-ai-gateway-infra(8dcf8782-f46c-4c0f-9133-793017d66a37)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5f387c-29888715-qgmjp" podUID="8dcf8782-f46c-4c0f-9133-793017d66a37"
Oct 30 01:15:08.259109 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:08.259086    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f6d5b8-openshift-default-68ddb87d77-n4kjt" event={"ID":"45d94cd6-d704-48c7-b1fb-e97c89e7d74b","Type":"ContainerStarted","Data":"3ffbaae21284766ab63e7484c47642a5f0e7df30c05526a2204493a9825c63c1"}
Oct 30 01:15:08.259215 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:08.259140    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f6d5b8-openshift-default-68ddb87d77-n4kjt" event={"ID":"45d94cd6-d704-48c7-b1fb-e97c89e7d74b","Type":"ContainerStarted","Data":"ae77c938d542d367725032c1e41dd2e2ceda2595ad40ca36c1e148d9149d55d3"}
Oct 30 01:15:08.261158 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:08.261136    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-d8b2c6-29888715-rqz6d"
Oct 30 01:15:08.261267 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:08.261187    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-d8b2c6-29888715-rqz6d" event={"ID":"91ed35be-dc11-4d3f-91bc-f4a7b4c7be28","Type":"ContainerDied","Data":"424cdec1bbc8085f2a2b9db6170d2734044d32569b5dc1ff7af37c39250e15de"}
Oct 30 01:15:08.261267 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:08.261239    5002 scope.go:122] "RemoveContainer" containerID="c1334c2e13aa673e9a0b651b6144991b77895f8f2486c3bdeec62a76950b213b"
Oct 30 01:15:08.346249 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:08.346140    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-f6d5b8-openshift-default-68ddb87d77-n4kjt" podStartSLOduration=1.346120206 podStartE2EDuration="1.346120206s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:15:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:15:08.343572474 +0000 UTC m=+2603.960005392" watchObservedRunningTime="2026-10-30 01:15:08.346120206 +0000 UTC m=+2603.962553086"
Oct 30 01:15:08.387718 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:08.387666    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-d8b2c6-29888715-rqz6d"]
Oct 30 01:15:08.394780 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:08.394739    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-d8b2c6-29888715-rqz6d"]
Oct 30 01:15:08.485408 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:08.484486    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-f6d5b8-openshift-default-68ddb87d77-n4kjt"
Oct 30 01:15:08.540403 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:08.540136    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05c750c5-39da-44a2-8414-3d7ffe09812d" path="/var/lib/kubelet/pods/05c750c5-39da-44a2-8414-3d7ffe09812d/volumes"
Oct 30 01:15:08.545408 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:08.540985    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a0e3f18-a339-4bc3-8d3b-3b38f3a50182" path="/var/lib/kubelet/pods/1a0e3f18-a339-4bc3-8d3b-3b38f3a50182/volumes"
Oct 30 01:15:08.545408 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:08.541700    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91ed35be-dc11-4d3f-91bc-f4a7b4c7be28" path="/var/lib/kubelet/pods/91ed35be-dc11-4d3f-91bc-f4a7b4c7be28/volumes"
Oct 30 01:15:09.490974 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:09.490934    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-f6d5b8-openshift-default-68ddb87d77-n4kjt"
Oct 30 01:15:10.360503 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:10.360458    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-f6d5b8-openshift-default-68ddb87d77-n4kjt"
Oct 30 01:15:10.361590 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:10.361566    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-f6d5b8-openshift-default-68ddb87d77-n4kjt"
Oct 30 01:15:12.697261 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:12.697224    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f6d5b8-865b864594-d5n25"]
Oct 30 01:15:12.698103 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:12.698082    5002 cpu_manager.go:414] "containerMap: removing container" podUID="91ed35be-dc11-4d3f-91bc-f4a7b4c7be28" containerName="cleanup"
Oct 30 01:15:12.698156 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:12.698106    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91ed35be-dc11-4d3f-91bc-f4a7b4c7be28" containerName="cleanup"
Oct 30 01:15:12.698353 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:12.698338    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="91ed35be-dc11-4d3f-91bc-f4a7b4c7be28" containerName="cleanup"
Oct 30 01:15:12.702148 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:12.702129    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f6d5b8-865b864594-d5n25"
Oct 30 01:15:12.721563 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:12.721535    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f6d5b8-865b864594-d5n25"]
Oct 30 01:15:12.729549 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:12.729522    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-f6d5b8\""
Oct 30 01:15:12.816552 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:12.816497    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f6d5b8-55c9ff76c5-xh6w2"]
Oct 30 01:15:12.818047 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:12.818011    5002 cpu_manager.go:414] "containerMap: removing container" podUID="91ed35be-dc11-4d3f-91bc-f4a7b4c7be28" containerName="cleanup"
Oct 30 01:15:12.818216 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:12.818198    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91ed35be-dc11-4d3f-91bc-f4a7b4c7be28" containerName="cleanup"
Oct 30 01:15:12.818617 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:12.818596    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="91ed35be-dc11-4d3f-91bc-f4a7b4c7be28" containerName="cleanup"
Oct 30 01:15:12.981859 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:12.981753    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xm42x\" (UniqueName: \"kubernetes.io/projected/9064e71e-4762-451a-8f62-bfb099e6c396-kube-api-access-xm42x\") pod \"payload-processing-e2e-ipp-a-f6d5b8-865b864594-d5n25\" (UID: \"9064e71e-4762-451a-8f62-bfb099e6c396\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f6d5b8-865b864594-d5n25"
Oct 30 01:15:13.021747 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:13.010319    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9064e71e-4762-451a-8f62-bfb099e6c396-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f6d5b8-865b864594-d5n25\" (UID: \"9064e71e-4762-451a-8f62-bfb099e6c396\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f6d5b8-865b864594-d5n25"
Oct 30 01:15:13.129575 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:13.129534    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xm42x\" (UniqueName: \"kubernetes.io/projected/9064e71e-4762-451a-8f62-bfb099e6c396-kube-api-access-xm42x\") pod \"payload-processing-e2e-ipp-a-f6d5b8-865b864594-d5n25\" (UID: \"9064e71e-4762-451a-8f62-bfb099e6c396\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f6d5b8-865b864594-d5n25"
Oct 30 01:15:13.130030 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:13.130006    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9064e71e-4762-451a-8f62-bfb099e6c396-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f6d5b8-865b864594-d5n25\" (UID: \"9064e71e-4762-451a-8f62-bfb099e6c396\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f6d5b8-865b864594-d5n25"
Oct 30 01:15:13.131292 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:13.131270    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9064e71e-4762-451a-8f62-bfb099e6c396-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f6d5b8-865b864594-d5n25\" (UID: \"9064e71e-4762-451a-8f62-bfb099e6c396\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f6d5b8-865b864594-d5n25"
Oct 30 01:15:13.217404 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:13.212260    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xm42x\" (UniqueName: \"kubernetes.io/projected/9064e71e-4762-451a-8f62-bfb099e6c396-kube-api-access-xm42x\") pod \"payload-processing-e2e-ipp-a-f6d5b8-865b864594-d5n25\" (UID: \"9064e71e-4762-451a-8f62-bfb099e6c396\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f6d5b8-865b864594-d5n25"
Oct 30 01:15:13.222397 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:13.218637    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f6d5b8-55c9ff76c5-xh6w2"
Oct 30 01:15:13.276661 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:13.276569    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-f6d5b8-dockercfg-qvpkx\""
Oct 30 01:15:13.327411 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:13.327350    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f6d5b8-865b864594-d5n25"
Oct 30 01:15:13.343031 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:13.342997    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f6d5b8-55c9ff76c5-xh6w2"]
Oct 30 01:15:13.367572 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:13.366829    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7dd42ca9-65d9-415d-9a0c-0151c50a72d6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f6d5b8-55c9ff76c5-xh6w2\" (UID: \"7dd42ca9-65d9-415d-9a0c-0151c50a72d6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f6d5b8-55c9ff76c5-xh6w2"
Oct 30 01:15:13.368855 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:13.368822    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njv7k\" (UniqueName: \"kubernetes.io/projected/7dd42ca9-65d9-415d-9a0c-0151c50a72d6-kube-api-access-njv7k\") pod \"payload-pre-processing-e2e-ipp-a-f6d5b8-55c9ff76c5-xh6w2\" (UID: \"7dd42ca9-65d9-415d-9a0c-0151c50a72d6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f6d5b8-55c9ff76c5-xh6w2"
Oct 30 01:15:13.403089 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:15:13.403054    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbjdq" containerName="extractor"
Oct 30 01:15:13.472404 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:13.471617    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7dd42ca9-65d9-415d-9a0c-0151c50a72d6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f6d5b8-55c9ff76c5-xh6w2\" (UID: \"7dd42ca9-65d9-415d-9a0c-0151c50a72d6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f6d5b8-55c9ff76c5-xh6w2"
Oct 30 01:15:13.472404 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:13.471801    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njv7k\" (UniqueName: \"kubernetes.io/projected/7dd42ca9-65d9-415d-9a0c-0151c50a72d6-kube-api-access-njv7k\") pod \"payload-pre-processing-e2e-ipp-a-f6d5b8-55c9ff76c5-xh6w2\" (UID: \"7dd42ca9-65d9-415d-9a0c-0151c50a72d6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f6d5b8-55c9ff76c5-xh6w2"
Oct 30 01:15:13.481133 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:13.480993    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7dd42ca9-65d9-415d-9a0c-0151c50a72d6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f6d5b8-55c9ff76c5-xh6w2\" (UID: \"7dd42ca9-65d9-415d-9a0c-0151c50a72d6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f6d5b8-55c9ff76c5-xh6w2"
Oct 30 01:15:13.497261 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:13.497225    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-njv7k\" (UniqueName: \"kubernetes.io/projected/7dd42ca9-65d9-415d-9a0c-0151c50a72d6-kube-api-access-njv7k\") pod \"payload-pre-processing-e2e-ipp-a-f6d5b8-55c9ff76c5-xh6w2\" (UID: \"7dd42ca9-65d9-415d-9a0c-0151c50a72d6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f6d5b8-55c9ff76c5-xh6w2"
Oct 30 01:15:13.698952 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:13.698922    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f6d5b8-55c9ff76c5-xh6w2"
Oct 30 01:15:13.741299 ip-10-0-128-7 kubenswrapper[5002]: W1030 01:15:13.741250    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9064e71e_4762_451a_8f62_bfb099e6c396.slice/crio-395ff2a2c136e24d715e48509183b19030d10dde2f3e5699c27762d69c7cf1da WatchSource:0}: Error finding container 395ff2a2c136e24d715e48509183b19030d10dde2f3e5699c27762d69c7cf1da: Status 404 returned error can't find the container with id 395ff2a2c136e24d715e48509183b19030d10dde2f3e5699c27762d69c7cf1da
Oct 30 01:15:13.743349 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:13.743299    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f6d5b8-865b864594-d5n25"]
Oct 30 01:15:14.041917 ip-10-0-128-7 kubenswrapper[5002]: W1030 01:15:14.041887    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7dd42ca9_65d9_415d_9a0c_0151c50a72d6.slice/crio-c06211688781955f873243b8e3ed1f42d2da4fe8ad2a122cd96729b10f499a0f WatchSource:0}: Error finding container c06211688781955f873243b8e3ed1f42d2da4fe8ad2a122cd96729b10f499a0f: Status 404 returned error can't find the container with id c06211688781955f873243b8e3ed1f42d2da4fe8ad2a122cd96729b10f499a0f
Oct 30 01:15:14.044051 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:14.044022    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f6d5b8-55c9ff76c5-xh6w2"]
Oct 30 01:15:14.437641 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:14.436374    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f6d5b8-865b864594-d5n25" event={"ID":"9064e71e-4762-451a-8f62-bfb099e6c396","Type":"ContainerStarted","Data":"9556aa13062f228be38f6103ba1f06cc992d9b4b41446d28a78583c44c9aa4d5"}
Oct 30 01:15:14.437641 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:14.436529    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f6d5b8-865b864594-d5n25" event={"ID":"9064e71e-4762-451a-8f62-bfb099e6c396","Type":"ContainerStarted","Data":"395ff2a2c136e24d715e48509183b19030d10dde2f3e5699c27762d69c7cf1da"}
Oct 30 01:15:14.437641 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:14.436893    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-f6d5b8-865b864594-d5n25"
Oct 30 01:15:14.440598 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:14.439366    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f6d5b8-55c9ff76c5-xh6w2" event={"ID":"7dd42ca9-65d9-415d-9a0c-0151c50a72d6","Type":"ContainerStarted","Data":"1a03a5f8f3ce2ca39b8e836070e109f0aa870c55d64ae0d0416edabc862a9438"}
Oct 30 01:15:14.440598 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:14.439449    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f6d5b8-55c9ff76c5-xh6w2" event={"ID":"7dd42ca9-65d9-415d-9a0c-0151c50a72d6","Type":"ContainerStarted","Data":"c06211688781955f873243b8e3ed1f42d2da4fe8ad2a122cd96729b10f499a0f"}
Oct 30 01:15:14.440598 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:14.440060    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f6d5b8-55c9ff76c5-xh6w2"
Oct 30 01:15:14.505663 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:14.503617    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-f6d5b8-865b864594-d5n25" podStartSLOduration=2.50359875 podStartE2EDuration="2.50359875s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:15:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:15:14.498797656 +0000 UTC m=+2610.115230538" watchObservedRunningTime="2026-10-30 01:15:14.50359875 +0000 UTC m=+2610.120031629"
Oct 30 01:15:14.587292 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:14.587228    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f6d5b8-55c9ff76c5-xh6w2" podStartSLOduration=2.587212376 podStartE2EDuration="2.587212376s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:15:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:15:14.58278266 +0000 UTC m=+2610.199215540" watchObservedRunningTime="2026-10-30 01:15:14.587212376 +0000 UTC m=+2610.203645252"
Oct 30 01:15:15.036897 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:15.036862    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a13c059-746ns"]
Oct 30 01:15:15.040914 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:15.040882    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a13c059-746ns"]
Oct 30 01:15:15.488042 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:15.487997    5002 scope.go:122] "RemoveContainer" containerID="c8b8d15d9c8d14dcf01fdf342d673d51d66ffbd0b07c8a1bbdac7924670a2c61"
Oct 30 01:15:16.477069 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:16.475991    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-009e78-29888715-526wv" event={"ID":"6b614db7-0a42-4251-a93b-80ee2f8d3fbe","Type":"ContainerStarted","Data":"c2eb12798e2a39adfe24b691d2aa674789ebfd5171925c31c6cab5c3b36cfce5"}
Oct 30 01:15:16.500205 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:16.500170    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d3c5627-d6b7-4703-afd1-3b965ae529c1" path="/var/lib/kubelet/pods/6d3c5627-d6b7-4703-afd1-3b965ae529c1/volumes"
Oct 30 01:15:16.501257 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:16.501234    5002 scope.go:122] "RemoveContainer" containerID="17e0a6c2cf3abe78ce7d4a834d12509990087dd9538d7dad7bdab6421ec72ff3"
Oct 30 01:15:16.501372 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:16.501343    5002 scope.go:122] "RemoveContainer" containerID="d9ecb54895eba5673f1094a2001c1141074d7cc26c4c0613d5f021fd49395ac9"
Oct 30 01:15:16.501644 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:16.501624    5002 scope.go:122] "RemoveContainer" containerID="645814b0bf1ed27656feb00c8e15dfcf1b8264ab84d39187925274efbd6949a3"
Oct 30 01:15:17.487968 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:17.487930    5002 scope.go:122] "RemoveContainer" containerID="6d02e29347533557fb6e5ea114602aaf38092b749b7366936bff8383dbc40473"
Oct 30 01:15:17.488500 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:17.488273    5002 scope.go:122] "RemoveContainer" containerID="3d6d154d5ea078834d229b13ea707f618f9040d1d25101fcc00ef3bde333470f"
Oct 30 01:15:17.488500 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:17.488464    5002 scope.go:122] "RemoveContainer" containerID="905a9de95281c6e73f1e04fa4338036abc144de4fa8bc65e61f69a7f66b4d737"
Oct 30 01:15:17.488500 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:17.488274    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f64dc0-29888715-75wpl" event={"ID":"5834beef-f5d2-4d8f-94da-aa3a86d692cf","Type":"ContainerStarted","Data":"1f1e5c79a38dded554910d84743d8d2ec4f5633b8f1568ff92140ed373e9f227"}
Oct 30 01:15:17.488829 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:17.488662    5002 scope.go:122] "RemoveContainer" containerID="50f107ac2854a1d0ec109dbae8e32f9e0859bb93052aecf60c8bbfd329e571d0"
Oct 30 01:15:17.496482 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:17.496453    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-cefb27-29888715-9524l" event={"ID":"7568f0d2-0dfb-47b9-ab97-d06d3c985d90","Type":"ContainerStarted","Data":"7b5e40f9bc8fdd90cf0a433425bc9ff090711f900bd3481f16fee94080f20018"}
Oct 30 01:15:17.500222 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:17.500194    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-a5e9ab-29888715-gltwb" event={"ID":"fd232d4c-4d2b-4137-bf83-b3e140fba4a6","Type":"ContainerStarted","Data":"e00d9719582ebf29f1ab9aa04e3685b117c52b198a19c849ccdc209e4220ba86"}
Oct 30 01:15:17.502322 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:17.502300    5002 generic.go:356] "Generic (PLEG): container finished" podID="6b614db7-0a42-4251-a93b-80ee2f8d3fbe" containerID="c2eb12798e2a39adfe24b691d2aa674789ebfd5171925c31c6cab5c3b36cfce5" exitCode=60
Oct 30 01:15:17.502515 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:17.502342    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-009e78-29888715-526wv" event={"ID":"6b614db7-0a42-4251-a93b-80ee2f8d3fbe","Type":"ContainerDied","Data":"c2eb12798e2a39adfe24b691d2aa674789ebfd5171925c31c6cab5c3b36cfce5"}
Oct 30 01:15:17.502515 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:17.502445    5002 scope.go:122] "RemoveContainer" containerID="c8b8d15d9c8d14dcf01fdf342d673d51d66ffbd0b07c8a1bbdac7924670a2c61"
Oct 30 01:15:17.502791 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:17.502771    5002 scope.go:122] "RemoveContainer" containerID="c2eb12798e2a39adfe24b691d2aa674789ebfd5171925c31c6cab5c3b36cfce5"
Oct 30 01:15:17.503070 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:15:17.503040    5002 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-009e78-29888715-526wv_odh-ai-gateway-infra(6b614db7-0a42-4251-a93b-80ee2f8d3fbe)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-009e78-29888715-526wv" podUID="6b614db7-0a42-4251-a93b-80ee2f8d3fbe"
Oct 30 01:15:17.533855 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:17.533799    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-009e78-29888715-526wv"]
Oct 30 01:15:18.262539 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:18.262499    5002 patch_prober.go:29] interesting pod/e2e-ipp-a-01215f-openshift-default-95bfd666f-rkm78 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.214:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 01:15:18.262694 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:18.262567    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-01215f-openshift-default-95bfd666f-rkm78" podUID="4c140dcb-40bd-4cd2-ac16-df81ed774c16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.214:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:15:18.499656 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:18.499618    5002 scope.go:122] "RemoveContainer" containerID="df65a7038cdf1664a4852454ee63255addfe0f962c8d1342bd29de14c22224c6"
Oct 30 01:15:18.501101 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:18.501063    5002 scope.go:122] "RemoveContainer" containerID="23bd2ab9aaff2196c2ce41d1f1b9a49b6c46d6e2b090fedf9acb902d803f01f8"
Oct 30 01:15:18.501257 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:18.501189    5002 scope.go:122] "RemoveContainer" containerID="2255fcbba7d31cf86be4719a7884880cbcf52c8f0d976dcd99ac3033e505733b"
Oct 30 01:15:18.502457 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:18.502435    5002 scope.go:122] "RemoveContainer" containerID="14aa48c03a3e6d7be42a1b03c02b8ea6aef907c5c8f22e72936cd38a6703d033"
Oct 30 01:15:18.523448 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:18.522391    5002 generic.go:356] "Generic (PLEG): container finished" podID="5834beef-f5d2-4d8f-94da-aa3a86d692cf" containerID="1f1e5c79a38dded554910d84743d8d2ec4f5633b8f1568ff92140ed373e9f227" exitCode=60
Oct 30 01:15:18.523448 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:18.522506    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f64dc0-29888715-75wpl" event={"ID":"5834beef-f5d2-4d8f-94da-aa3a86d692cf","Type":"ContainerDied","Data":"1f1e5c79a38dded554910d84743d8d2ec4f5633b8f1568ff92140ed373e9f227"}
Oct 30 01:15:18.523448 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:18.522565    5002 scope.go:122] "RemoveContainer" containerID="17e0a6c2cf3abe78ce7d4a834d12509990087dd9538d7dad7bdab6421ec72ff3"
Oct 30 01:15:18.523448 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:18.523035    5002 scope.go:122] "RemoveContainer" containerID="1f1e5c79a38dded554910d84743d8d2ec4f5633b8f1568ff92140ed373e9f227"
Oct 30 01:15:18.523448 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:15:18.523368    5002 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-f64dc0-29888715-75wpl_odh-ai-gateway-infra(5834beef-f5d2-4d8f-94da-aa3a86d692cf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f64dc0-29888715-75wpl" podUID="5834beef-f5d2-4d8f-94da-aa3a86d692cf"
Oct 30 01:15:18.526504 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:18.526478    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-11f521-29888715-fj8kn" event={"ID":"248fa0d9-fe68-4ba7-82c3-d7911c9b8070","Type":"ContainerStarted","Data":"8ff3b0500901205e10a71a95b0d47ecf59b07c2227fec1afd98e63b286148ee6"}
Oct 30 01:15:18.529534 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:18.528873    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-7f92f6-29888715-d7v2x" event={"ID":"34ced72e-384a-4e43-90b7-2f907956c2c1","Type":"ContainerStarted","Data":"5674fc59e947e49dc4fd07f1676f7f6c080e2a949a8ab4579e3d906eafb1702e"}
Oct 30 01:15:18.531437 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:18.531409    5002 generic.go:356] "Generic (PLEG): container finished" podID="7568f0d2-0dfb-47b9-ab97-d06d3c985d90" containerID="7b5e40f9bc8fdd90cf0a433425bc9ff090711f900bd3481f16fee94080f20018" exitCode=60
Oct 30 01:15:18.531564 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:18.531494    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-cefb27-29888715-9524l" event={"ID":"7568f0d2-0dfb-47b9-ab97-d06d3c985d90","Type":"ContainerDied","Data":"7b5e40f9bc8fdd90cf0a433425bc9ff090711f900bd3481f16fee94080f20018"}
Oct 30 01:15:18.531937 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:18.531918    5002 scope.go:122] "RemoveContainer" containerID="7b5e40f9bc8fdd90cf0a433425bc9ff090711f900bd3481f16fee94080f20018"
Oct 30 01:15:18.532279 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:15:18.532255    5002 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-cefb27-29888715-9524l_odh-ai-gateway-infra(7568f0d2-0dfb-47b9-ab97-d06d3c985d90)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-cefb27-29888715-9524l" podUID="7568f0d2-0dfb-47b9-ab97-d06d3c985d90"
Oct 30 01:15:18.535202 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:18.535177    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-01215f-29888715-bxdtr" event={"ID":"92776c04-7f5b-457a-be15-a5ce82a98a68","Type":"ContainerStarted","Data":"73714508be7d7627b4cbfa2b418d04356a6fd11a746288db2064cead1e17c4ba"}
Oct 30 01:15:18.539477 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:18.539445    5002 generic.go:356] "Generic (PLEG): container finished" podID="fd232d4c-4d2b-4137-bf83-b3e140fba4a6" containerID="e00d9719582ebf29f1ab9aa04e3685b117c52b198a19c849ccdc209e4220ba86" exitCode=60
Oct 30 01:15:18.539592 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:18.539507    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-a5e9ab-29888715-gltwb" event={"ID":"fd232d4c-4d2b-4137-bf83-b3e140fba4a6","Type":"ContainerDied","Data":"e00d9719582ebf29f1ab9aa04e3685b117c52b198a19c849ccdc209e4220ba86"}
Oct 30 01:15:18.539923 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:18.539904    5002 scope.go:122] "RemoveContainer" containerID="e00d9719582ebf29f1ab9aa04e3685b117c52b198a19c849ccdc209e4220ba86"
Oct 30 01:15:18.540341 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:15:18.540315    5002 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w5-a5e9ab-29888715-gltwb_odh-ai-gateway-infra(fd232d4c-4d2b-4137-bf83-b3e140fba4a6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-a5e9ab-29888715-gltwb" podUID="fd232d4c-4d2b-4137-bf83-b3e140fba4a6"
Oct 30 01:15:18.547442 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:18.547413    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-48df96-29888715-nc8tn" event={"ID":"71d53e61-57bc-4d66-a029-965ea868d660","Type":"ContainerStarted","Data":"255c2a03584ce6c0ed51cf5d91952ffb8faaa215eb0bf2934a21b9e091c34670"}
Oct 30 01:15:18.559625 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:18.559597    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f64dc0-29888715-75wpl"]
Oct 30 01:15:18.603402 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:18.601724    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-a5e9ab-29888715-gltwb"]
Oct 30 01:15:18.625868 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:18.625838    5002 scope.go:122] "RemoveContainer" containerID="645814b0bf1ed27656feb00c8e15dfcf1b8264ab84d39187925274efbd6949a3"
Oct 30 01:15:18.657734 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:18.656964    5002 scope.go:122] "RemoveContainer" containerID="d9ecb54895eba5673f1094a2001c1141074d7cc26c4c0613d5f021fd49395ac9"
Oct 30 01:15:18.701995 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:18.660477    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-cefb27-29888715-9524l"]
Oct 30 01:15:19.093567 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:19.093535    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-009e78-29888715-526wv"
Oct 30 01:15:19.160625 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:19.160589    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b614db7-0a42-4251-a93b-80ee2f8d3fbe-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b614db7-0a42-4251-a93b-80ee2f8d3fbe-maas-api-service-ca\") pod \"6b614db7-0a42-4251-a93b-80ee2f8d3fbe\" (UID: \"6b614db7-0a42-4251-a93b-80ee2f8d3fbe\") "
Oct 30 01:15:19.160810 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:19.160776    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b614db7-0a42-4251-a93b-80ee2f8d3fbe-kube-api-access-d25m9\" (UniqueName: \"kubernetes.io/projected/6b614db7-0a42-4251-a93b-80ee2f8d3fbe-kube-api-access-d25m9\") pod \"6b614db7-0a42-4251-a93b-80ee2f8d3fbe\" (UID: \"6b614db7-0a42-4251-a93b-80ee2f8d3fbe\") "
Oct 30 01:15:19.162235 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:19.162204    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b614db7-0a42-4251-a93b-80ee2f8d3fbe-maas-api-service-ca" pod "6b614db7-0a42-4251-a93b-80ee2f8d3fbe" (UID: "6b614db7-0a42-4251-a93b-80ee2f8d3fbe"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:15:19.164554 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:19.164520    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b614db7-0a42-4251-a93b-80ee2f8d3fbe-kube-api-access-d25m9" pod "6b614db7-0a42-4251-a93b-80ee2f8d3fbe" (UID: "6b614db7-0a42-4251-a93b-80ee2f8d3fbe"). InnerVolumeSpecName "kube-api-access-d25m9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:15:19.264140 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:19.264073    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b614db7-0a42-4251-a93b-80ee2f8d3fbe-maas-api-service-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:15:19.264140 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:19.264104    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d25m9\" (UniqueName: \"kubernetes.io/projected/6b614db7-0a42-4251-a93b-80ee2f8d3fbe-kube-api-access-d25m9\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:15:19.487336 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:19.487306    5002 scope.go:122] "RemoveContainer" containerID="23542a46cd867d1aabf74ab0dacd3ead9b877c819c387b235a8cdc14ad648201"
Oct 30 01:15:19.565783 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:19.565745    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a7ca1f-29888715-r8jhp" event={"ID":"663c6827-07ce-4de7-8d82-033f8b60c129","Type":"ContainerStarted","Data":"f80558127124592773fc7079a2faec9a66e7f1724ad0537746cb39dcd4049259"}
Oct 30 01:15:19.570309 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:19.570275    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-beeba4-29888715-5t886" event={"ID":"55f94866-dc95-4ee1-b634-a3c434226a03","Type":"ContainerStarted","Data":"e6d8a59d8e01248217c97a89ef4ef1b930fc2179be1bf8514ab6049f3d906319"}
Oct 30 01:15:19.575910 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:19.575883    5002 generic.go:356] "Generic (PLEG): container finished" podID="248fa0d9-fe68-4ba7-82c3-d7911c9b8070" containerID="8ff3b0500901205e10a71a95b0d47ecf59b07c2227fec1afd98e63b286148ee6" exitCode=60
Oct 30 01:15:19.576029 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:19.575943    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-11f521-29888715-fj8kn" event={"ID":"248fa0d9-fe68-4ba7-82c3-d7911c9b8070","Type":"ContainerDied","Data":"8ff3b0500901205e10a71a95b0d47ecf59b07c2227fec1afd98e63b286148ee6"}
Oct 30 01:15:19.576029 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:19.575979    5002 scope.go:122] "RemoveContainer" containerID="6d02e29347533557fb6e5ea114602aaf38092b749b7366936bff8383dbc40473"
Oct 30 01:15:19.576265 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:19.576251    5002 scope.go:122] "RemoveContainer" containerID="8ff3b0500901205e10a71a95b0d47ecf59b07c2227fec1afd98e63b286148ee6"
Oct 30 01:15:19.576535 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:15:19.576507    5002 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-11f521-29888715-fj8kn_odh-ai-gateway-infra(248fa0d9-fe68-4ba7-82c3-d7911c9b8070)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-11f521-29888715-fj8kn" podUID="248fa0d9-fe68-4ba7-82c3-d7911c9b8070"
Oct 30 01:15:19.579319 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:19.579292    5002 generic.go:356] "Generic (PLEG): container finished" podID="34ced72e-384a-4e43-90b7-2f907956c2c1" containerID="5674fc59e947e49dc4fd07f1676f7f6c080e2a949a8ab4579e3d906eafb1702e" exitCode=60
Oct 30 01:15:19.579448 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:19.579435    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-7f92f6-29888715-d7v2x" event={"ID":"34ced72e-384a-4e43-90b7-2f907956c2c1","Type":"ContainerDied","Data":"5674fc59e947e49dc4fd07f1676f7f6c080e2a949a8ab4579e3d906eafb1702e"}
Oct 30 01:15:19.579811 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:19.579790    5002 scope.go:122] "RemoveContainer" containerID="5674fc59e947e49dc4fd07f1676f7f6c080e2a949a8ab4579e3d906eafb1702e"
Oct 30 01:15:19.580118 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:15:19.580094    5002 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w6-7f92f6-29888715-d7v2x_odh-ai-gateway-infra(34ced72e-384a-4e43-90b7-2f907956c2c1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-7f92f6-29888715-d7v2x" podUID="34ced72e-384a-4e43-90b7-2f907956c2c1"
Oct 30 01:15:19.581643 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:19.581621    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-6d44c2-29888715-d8zbn" event={"ID":"07d6239e-5b00-4c4a-b8dd-d935c4c44ed8","Type":"ContainerStarted","Data":"73cbf10aa23091f2a934e7170773dc5e0d84204d35291a66cdcb6b9089b4ccab"}
Oct 30 01:15:19.585264 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:19.585238    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-00e07a-29888715-8dq5b" event={"ID":"82b3d01a-515b-43fe-b05d-ae6d90c678a3","Type":"ContainerStarted","Data":"652a74c435342050fff5844397565f7de79f038ad2731f8535b33122fe1fef1f"}
Oct 30 01:15:19.589671 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:19.589646    5002 generic.go:356] "Generic (PLEG): container finished" podID="92776c04-7f5b-457a-be15-a5ce82a98a68" containerID="73714508be7d7627b4cbfa2b418d04356a6fd11a746288db2064cead1e17c4ba" exitCode=60
Oct 30 01:15:19.589855 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:19.589712    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-01215f-29888715-bxdtr" event={"ID":"92776c04-7f5b-457a-be15-a5ce82a98a68","Type":"ContainerDied","Data":"73714508be7d7627b4cbfa2b418d04356a6fd11a746288db2064cead1e17c4ba"}
Oct 30 01:15:19.590182 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:19.590165    5002 scope.go:122] "RemoveContainer" containerID="73714508be7d7627b4cbfa2b418d04356a6fd11a746288db2064cead1e17c4ba"
Oct 30 01:15:19.590496 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:15:19.590472    5002 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-01215f-29888715-bxdtr_odh-ai-gateway-infra(92776c04-7f5b-457a-be15-a5ce82a98a68)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-01215f-29888715-bxdtr" podUID="92776c04-7f5b-457a-be15-a5ce82a98a68"
Oct 30 01:15:19.595462 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:19.594009    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-009e78-29888715-526wv" event={"ID":"6b614db7-0a42-4251-a93b-80ee2f8d3fbe","Type":"ContainerDied","Data":"11dc2468e22b04d16e259f59b63ffd3794700f105b63961d7d4f27fc9ad73314"}
Oct 30 01:15:19.595462 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:19.594091    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-009e78-29888715-526wv"
Oct 30 01:15:19.599024 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:19.598997    5002 generic.go:356] "Generic (PLEG): container finished" podID="71d53e61-57bc-4d66-a029-965ea868d660" containerID="255c2a03584ce6c0ed51cf5d91952ffb8faaa215eb0bf2934a21b9e091c34670" exitCode=60
Oct 30 01:15:19.599125 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:19.599089    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-48df96-29888715-nc8tn" event={"ID":"71d53e61-57bc-4d66-a029-965ea868d660","Type":"ContainerDied","Data":"255c2a03584ce6c0ed51cf5d91952ffb8faaa215eb0bf2934a21b9e091c34670"}
Oct 30 01:15:19.599513 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:19.599466    5002 scope.go:122] "RemoveContainer" containerID="255c2a03584ce6c0ed51cf5d91952ffb8faaa215eb0bf2934a21b9e091c34670"
Oct 30 01:15:19.599785 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:15:19.599763    5002 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-48df96-29888715-nc8tn_odh-ai-gateway-infra(71d53e61-57bc-4d66-a029-965ea868d660)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-48df96-29888715-nc8tn" podUID="71d53e61-57bc-4d66-a029-965ea868d660"
Oct 30 01:15:19.647669 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:19.647639    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-7f92f6-29888715-d7v2x"]
Oct 30 01:15:19.698517 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:19.698312    5002 scope.go:122] "RemoveContainer" containerID="3d6d154d5ea078834d229b13ea707f618f9040d1d25101fcc00ef3bde333470f"
Oct 30 01:15:19.705052 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:19.704989    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-48df96-29888715-nc8tn"]
Oct 30 01:15:19.733126 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:19.733074    5002 scope.go:122] "RemoveContainer" containerID="50f107ac2854a1d0ec109dbae8e32f9e0859bb93052aecf60c8bbfd329e571d0"
Oct 30 01:15:19.771815 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:19.771793    5002 scope.go:122] "RemoveContainer" containerID="c2eb12798e2a39adfe24b691d2aa674789ebfd5171925c31c6cab5c3b36cfce5"
Oct 30 01:15:19.792334 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:19.792232    5002 scope.go:122] "RemoveContainer" containerID="905a9de95281c6e73f1e04fa4338036abc144de4fa8bc65e61f69a7f66b4d737"
Oct 30 01:15:20.012160 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:20.012066    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-01215f-29888715-bxdtr"]
Oct 30 01:15:20.106898 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:20.106860    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-11f521-29888715-fj8kn"]
Oct 30 01:15:20.158472 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:20.158408    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-009e78-29888715-526wv"]
Oct 30 01:15:20.163478 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:20.163448    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-009e78-29888715-526wv"]
Oct 30 01:15:20.236192 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:20.236164    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-a5e9ab-29888715-gltwb"
Oct 30 01:15:20.240549 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:20.240522    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-cefb27-29888715-9524l"
Oct 30 01:15:20.244739 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:20.244712    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f64dc0-29888715-75wpl"
Oct 30 01:15:20.290080 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:20.289990    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5834beef-f5d2-4d8f-94da-aa3a86d692cf-kube-api-access-c7hhq\" (UniqueName: \"kubernetes.io/projected/5834beef-f5d2-4d8f-94da-aa3a86d692cf-kube-api-access-c7hhq\") pod \"5834beef-f5d2-4d8f-94da-aa3a86d692cf\" (UID: \"5834beef-f5d2-4d8f-94da-aa3a86d692cf\") "
Oct 30 01:15:20.290249 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:20.290157    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5834beef-f5d2-4d8f-94da-aa3a86d692cf-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5834beef-f5d2-4d8f-94da-aa3a86d692cf-maas-api-service-ca\") pod \"5834beef-f5d2-4d8f-94da-aa3a86d692cf\" (UID: \"5834beef-f5d2-4d8f-94da-aa3a86d692cf\") "
Oct 30 01:15:20.290328 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:20.290313    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7568f0d2-0dfb-47b9-ab97-d06d3c985d90-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7568f0d2-0dfb-47b9-ab97-d06d3c985d90-maas-api-service-ca\") pod \"7568f0d2-0dfb-47b9-ab97-d06d3c985d90\" (UID: \"7568f0d2-0dfb-47b9-ab97-d06d3c985d90\") "
Oct 30 01:15:20.290435 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:20.290348    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd232d4c-4d2b-4137-bf83-b3e140fba4a6-kube-api-access-6tmc5\" (UniqueName: \"kubernetes.io/projected/fd232d4c-4d2b-4137-bf83-b3e140fba4a6-kube-api-access-6tmc5\") pod \"fd232d4c-4d2b-4137-bf83-b3e140fba4a6\" (UID: \"fd232d4c-4d2b-4137-bf83-b3e140fba4a6\") "
Oct 30 01:15:20.290435 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:20.290427    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd232d4c-4d2b-4137-bf83-b3e140fba4a6-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd232d4c-4d2b-4137-bf83-b3e140fba4a6-maas-api-service-ca\") pod \"fd232d4c-4d2b-4137-bf83-b3e140fba4a6\" (UID: \"fd232d4c-4d2b-4137-bf83-b3e140fba4a6\") "
Oct 30 01:15:20.290560 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:20.290478    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7568f0d2-0dfb-47b9-ab97-d06d3c985d90-kube-api-access-rxmgv\" (UniqueName: \"kubernetes.io/projected/7568f0d2-0dfb-47b9-ab97-d06d3c985d90-kube-api-access-rxmgv\") pod \"7568f0d2-0dfb-47b9-ab97-d06d3c985d90\" (UID: \"7568f0d2-0dfb-47b9-ab97-d06d3c985d90\") "
Oct 30 01:15:20.290969 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:20.290936    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7568f0d2-0dfb-47b9-ab97-d06d3c985d90-maas-api-service-ca" pod "7568f0d2-0dfb-47b9-ab97-d06d3c985d90" (UID: "7568f0d2-0dfb-47b9-ab97-d06d3c985d90"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:15:20.291576 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:20.291552    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5834beef-f5d2-4d8f-94da-aa3a86d692cf-maas-api-service-ca" pod "5834beef-f5d2-4d8f-94da-aa3a86d692cf" (UID: "5834beef-f5d2-4d8f-94da-aa3a86d692cf"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:15:20.292393 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:20.292353    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd232d4c-4d2b-4137-bf83-b3e140fba4a6-maas-api-service-ca" pod "fd232d4c-4d2b-4137-bf83-b3e140fba4a6" (UID: "fd232d4c-4d2b-4137-bf83-b3e140fba4a6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:15:20.292634 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:20.292612    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7568f0d2-0dfb-47b9-ab97-d06d3c985d90-maas-api-service-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:15:20.292766 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:20.292751    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd232d4c-4d2b-4137-bf83-b3e140fba4a6-maas-api-service-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:15:20.292875 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:20.292862    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5834beef-f5d2-4d8f-94da-aa3a86d692cf-maas-api-service-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:15:20.298553 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:20.298529    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5834beef-f5d2-4d8f-94da-aa3a86d692cf-kube-api-access-c7hhq" pod "5834beef-f5d2-4d8f-94da-aa3a86d692cf" (UID: "5834beef-f5d2-4d8f-94da-aa3a86d692cf"). InnerVolumeSpecName "kube-api-access-c7hhq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:15:20.298680 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:20.298562    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd232d4c-4d2b-4137-bf83-b3e140fba4a6-kube-api-access-6tmc5" pod "fd232d4c-4d2b-4137-bf83-b3e140fba4a6" (UID: "fd232d4c-4d2b-4137-bf83-b3e140fba4a6"). InnerVolumeSpecName "kube-api-access-6tmc5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:15:20.298680 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:20.298579    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7568f0d2-0dfb-47b9-ab97-d06d3c985d90-kube-api-access-rxmgv" pod "7568f0d2-0dfb-47b9-ab97-d06d3c985d90" (UID: "7568f0d2-0dfb-47b9-ab97-d06d3c985d90"). InnerVolumeSpecName "kube-api-access-rxmgv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:15:20.394354 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:20.394311    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6tmc5\" (UniqueName: \"kubernetes.io/projected/fd232d4c-4d2b-4137-bf83-b3e140fba4a6-kube-api-access-6tmc5\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:15:20.394354 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:20.394354    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rxmgv\" (UniqueName: \"kubernetes.io/projected/7568f0d2-0dfb-47b9-ab97-d06d3c985d90-kube-api-access-rxmgv\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:15:20.394354 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:20.394368    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c7hhq\" (UniqueName: \"kubernetes.io/projected/5834beef-f5d2-4d8f-94da-aa3a86d692cf-kube-api-access-c7hhq\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:15:20.487519 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:20.487483    5002 scope.go:122] "RemoveContainer" containerID="fe7a45761c593f0a164c74318faf5ae1530d8823c3e8d4021d1dbf53be6a04ac"
Oct 30 01:15:20.489273 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:20.489246    5002 scope.go:122] "RemoveContainer" containerID="b4a475af6879a21a3dc26a89fdb0efe18a0a29c561d79be4c06cc7b9deee6190"
Oct 30 01:15:20.516790 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:20.514677    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b614db7-0a42-4251-a93b-80ee2f8d3fbe" path="/var/lib/kubelet/pods/6b614db7-0a42-4251-a93b-80ee2f8d3fbe/volumes"
Oct 30 01:15:20.611134 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:20.611102    5002 generic.go:356] "Generic (PLEG): container finished" podID="663c6827-07ce-4de7-8d82-033f8b60c129" containerID="f80558127124592773fc7079a2faec9a66e7f1724ad0537746cb39dcd4049259" exitCode=60
Oct 30 01:15:20.611556 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:20.611170    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a7ca1f-29888715-r8jhp" event={"ID":"663c6827-07ce-4de7-8d82-033f8b60c129","Type":"ContainerDied","Data":"f80558127124592773fc7079a2faec9a66e7f1724ad0537746cb39dcd4049259"}
Oct 30 01:15:20.611556 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:20.611225    5002 scope.go:122] "RemoveContainer" containerID="df65a7038cdf1664a4852454ee63255addfe0f962c8d1342bd29de14c22224c6"
Oct 30 01:15:20.611689 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:20.611640    5002 scope.go:122] "RemoveContainer" containerID="f80558127124592773fc7079a2faec9a66e7f1724ad0537746cb39dcd4049259"
Oct 30 01:15:20.611936 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:15:20.611917    5002 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-a7ca1f-29888715-r8jhp_odh-ai-gateway-infra(663c6827-07ce-4de7-8d82-033f8b60c129)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a7ca1f-29888715-r8jhp" podUID="663c6827-07ce-4de7-8d82-033f8b60c129"
Oct 30 01:15:20.614135 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:20.614113    5002 generic.go:356] "Generic (PLEG): container finished" podID="55f94866-dc95-4ee1-b634-a3c434226a03" containerID="e6d8a59d8e01248217c97a89ef4ef1b930fc2179be1bf8514ab6049f3d906319" exitCode=60
Oct 30 01:15:20.614251 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:20.614191    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-beeba4-29888715-5t886" event={"ID":"55f94866-dc95-4ee1-b634-a3c434226a03","Type":"ContainerDied","Data":"e6d8a59d8e01248217c97a89ef4ef1b930fc2179be1bf8514ab6049f3d906319"}
Oct 30 01:15:20.614668 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:20.614650    5002 scope.go:122] "RemoveContainer" containerID="e6d8a59d8e01248217c97a89ef4ef1b930fc2179be1bf8514ab6049f3d906319"
Oct 30 01:15:20.614958 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:15:20.614932    5002 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-beeba4-29888715-5t886_odh-ai-gateway-infra(55f94866-dc95-4ee1-b634-a3c434226a03)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-beeba4-29888715-5t886" podUID="55f94866-dc95-4ee1-b634-a3c434226a03"
Oct 30 01:15:20.615759 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:20.615737    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f64dc0-29888715-75wpl" event={"ID":"5834beef-f5d2-4d8f-94da-aa3a86d692cf","Type":"ContainerDied","Data":"3e62fd1e7091e70a39a9b7e264c4ab7547dc772081fd0b005f2188f8c7b20be1"}
Oct 30 01:15:20.615759 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:20.615751    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f64dc0-29888715-75wpl"
Oct 30 01:15:20.623208 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:20.623179    5002 generic.go:356] "Generic (PLEG): container finished" podID="07d6239e-5b00-4c4a-b8dd-d935c4c44ed8" containerID="73cbf10aa23091f2a934e7170773dc5e0d84204d35291a66cdcb6b9089b4ccab" exitCode=60
Oct 30 01:15:20.623538 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:20.623517    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-6d44c2-29888715-d8zbn" event={"ID":"07d6239e-5b00-4c4a-b8dd-d935c4c44ed8","Type":"ContainerDied","Data":"73cbf10aa23091f2a934e7170773dc5e0d84204d35291a66cdcb6b9089b4ccab"}
Oct 30 01:15:20.624066 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:20.624047    5002 scope.go:122] "RemoveContainer" containerID="73cbf10aa23091f2a934e7170773dc5e0d84204d35291a66cdcb6b9089b4ccab"
Oct 30 01:15:20.624524 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:15:20.624501    5002 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-6d44c2-29888715-d8zbn_odh-ai-gateway-infra(07d6239e-5b00-4c4a-b8dd-d935c4c44ed8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-6d44c2-29888715-d8zbn" podUID="07d6239e-5b00-4c4a-b8dd-d935c4c44ed8"
Oct 30 01:15:20.625327 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:20.625303    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a7ca1f-29888715-r8jhp"]
Oct 30 01:15:20.627709 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:20.627683    5002 generic.go:356] "Generic (PLEG): container finished" podID="5d965e6c-4e56-4dff-918d-c93b9832b72f" containerID="7f84e1267e79b985c7ed94248089f75ffe7fa5e911ff50fcf32343e00c6ab09e" exitCode=60
Oct 30 01:15:20.627813 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:20.627797    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-29be73-29888715-p2ctj" event={"ID":"5d965e6c-4e56-4dff-918d-c93b9832b72f","Type":"ContainerDied","Data":"7f84e1267e79b985c7ed94248089f75ffe7fa5e911ff50fcf32343e00c6ab09e"}
Oct 30 01:15:20.628240 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:20.628210    5002 scope.go:122] "RemoveContainer" containerID="7f84e1267e79b985c7ed94248089f75ffe7fa5e911ff50fcf32343e00c6ab09e"
Oct 30 01:15:20.628715 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:15:20.628561    5002 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-29be73-29888715-p2ctj_odh-ai-gateway-infra(5d965e6c-4e56-4dff-918d-c93b9832b72f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-29be73-29888715-p2ctj" podUID="5d965e6c-4e56-4dff-918d-c93b9832b72f"
Oct 30 01:15:20.631780 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:20.631582    5002 scope.go:122] "RemoveContainer" containerID="23bd2ab9aaff2196c2ce41d1f1b9a49b6c46d6e2b090fedf9acb902d803f01f8"
Oct 30 01:15:20.633594 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:20.632516    5002 generic.go:356] "Generic (PLEG): container finished" podID="82b3d01a-515b-43fe-b05d-ae6d90c678a3" containerID="652a74c435342050fff5844397565f7de79f038ad2731f8535b33122fe1fef1f" exitCode=60
Oct 30 01:15:20.633594 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:20.632581    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-00e07a-29888715-8dq5b" event={"ID":"82b3d01a-515b-43fe-b05d-ae6d90c678a3","Type":"ContainerDied","Data":"652a74c435342050fff5844397565f7de79f038ad2731f8535b33122fe1fef1f"}
Oct 30 01:15:20.633594 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:20.632880    5002 scope.go:122] "RemoveContainer" containerID="652a74c435342050fff5844397565f7de79f038ad2731f8535b33122fe1fef1f"
Oct 30 01:15:20.633594 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:15:20.633125    5002 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-00e07a-29888715-8dq5b_odh-ai-gateway-infra(82b3d01a-515b-43fe-b05d-ae6d90c678a3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-00e07a-29888715-8dq5b" podUID="82b3d01a-515b-43fe-b05d-ae6d90c678a3"
Oct 30 01:15:20.635606 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:20.635580    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-cefb27-29888715-9524l" event={"ID":"7568f0d2-0dfb-47b9-ab97-d06d3c985d90","Type":"ContainerDied","Data":"0a850b1d818322abbdd6c5a37637eb88107df5aa51caabcb9e2f65dcd25ff525"}
Oct 30 01:15:20.639608 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:20.636078    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-cefb27-29888715-9524l"
Oct 30 01:15:20.642230 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:20.642202    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-a5e9ab-29888715-gltwb" event={"ID":"fd232d4c-4d2b-4137-bf83-b3e140fba4a6","Type":"ContainerDied","Data":"ee793b9ccfdadb3f8ee4da26abdbc21b18cfff3cbd568f62d619faf48503f237"}
Oct 30 01:15:20.642402 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:20.642312    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-a5e9ab-29888715-gltwb"
Oct 30 01:15:20.678278 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:20.678247    5002 scope.go:122] "RemoveContainer" containerID="1f1e5c79a38dded554910d84743d8d2ec4f5633b8f1568ff92140ed373e9f227"
Oct 30 01:15:20.689798 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:20.688286    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-00e07a-29888715-8dq5b"]
Oct 30 01:15:20.704740 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:20.704723    5002 scope.go:122] "RemoveContainer" containerID="14aa48c03a3e6d7be42a1b03c02b8ea6aef907c5c8f22e72936cd38a6703d033"
Oct 30 01:15:20.723631 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:20.723609    5002 scope.go:122] "RemoveContainer" containerID="23542a46cd867d1aabf74ab0dacd3ead9b877c819c387b235a8cdc14ad648201"
Oct 30 01:15:20.780036 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:20.780005    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-6d44c2-29888715-d8zbn"]
Oct 30 01:15:20.799666 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:20.798870    5002 scope.go:122] "RemoveContainer" containerID="2255fcbba7d31cf86be4719a7884880cbcf52c8f0d976dcd99ac3033e505733b"
Oct 30 01:15:20.812605 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:20.812529    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-a5e9ab-29888715-gltwb"]
Oct 30 01:15:20.820716 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:20.820679    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-a5e9ab-29888715-gltwb"]
Oct 30 01:15:20.835110 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:20.835028    5002 scope.go:122] "RemoveContainer" containerID="7b5e40f9bc8fdd90cf0a433425bc9ff090711f900bd3481f16fee94080f20018"
Oct 30 01:15:20.854664 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:20.853459    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-cefb27-29888715-9524l"]
Oct 30 01:15:20.860572 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:20.857577    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-cefb27-29888715-9524l"]
Oct 30 01:15:20.861952 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:20.861915    5002 scope.go:122] "RemoveContainer" containerID="e00d9719582ebf29f1ab9aa04e3685b117c52b198a19c849ccdc209e4220ba86"
Oct 30 01:15:20.913553 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:20.913522    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-beeba4-29888715-5t886"]
Oct 30 01:15:20.930514 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:20.930482    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f64dc0-29888715-75wpl"]
Oct 30 01:15:20.941504 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:20.941471    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f64dc0-29888715-75wpl"]
Oct 30 01:15:21.144562 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:21.144521    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-7f92f6-29888715-d7v2x"
Oct 30 01:15:21.150915 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:21.150880    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-11f521-29888715-fj8kn"
Oct 30 01:15:21.156832 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:21.156802    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-48df96-29888715-nc8tn"
Oct 30 01:15:21.158945 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:21.158919    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-01215f-29888715-bxdtr"
Oct 30 01:15:21.210871 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:21.210834    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92776c04-7f5b-457a-be15-a5ce82a98a68-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/92776c04-7f5b-457a-be15-a5ce82a98a68-maas-api-service-ca\") pod \"92776c04-7f5b-457a-be15-a5ce82a98a68\" (UID: \"92776c04-7f5b-457a-be15-a5ce82a98a68\") "
Oct 30 01:15:21.211045 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:21.210893    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71d53e61-57bc-4d66-a029-965ea868d660-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/71d53e61-57bc-4d66-a029-965ea868d660-maas-api-service-ca\") pod \"71d53e61-57bc-4d66-a029-965ea868d660\" (UID: \"71d53e61-57bc-4d66-a029-965ea868d660\") "
Oct 30 01:15:21.211045 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:21.210969    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/248fa0d9-fe68-4ba7-82c3-d7911c9b8070-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/248fa0d9-fe68-4ba7-82c3-d7911c9b8070-maas-api-service-ca\") pod \"248fa0d9-fe68-4ba7-82c3-d7911c9b8070\" (UID: \"248fa0d9-fe68-4ba7-82c3-d7911c9b8070\") "
Oct 30 01:15:21.211045 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:21.211019    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/248fa0d9-fe68-4ba7-82c3-d7911c9b8070-kube-api-access-rdjdh\" (UniqueName: \"kubernetes.io/projected/248fa0d9-fe68-4ba7-82c3-d7911c9b8070-kube-api-access-rdjdh\") pod \"248fa0d9-fe68-4ba7-82c3-d7911c9b8070\" (UID: \"248fa0d9-fe68-4ba7-82c3-d7911c9b8070\") "
Oct 30 01:15:21.211168 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:21.211152    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34ced72e-384a-4e43-90b7-2f907956c2c1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/34ced72e-384a-4e43-90b7-2f907956c2c1-maas-api-service-ca\") pod \"34ced72e-384a-4e43-90b7-2f907956c2c1\" (UID: \"34ced72e-384a-4e43-90b7-2f907956c2c1\") "
Oct 30 01:15:21.211285 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:21.211258    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92776c04-7f5b-457a-be15-a5ce82a98a68-maas-api-service-ca" pod "92776c04-7f5b-457a-be15-a5ce82a98a68" (UID: "92776c04-7f5b-457a-be15-a5ce82a98a68"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:15:21.211329 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:21.211301    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92776c04-7f5b-457a-be15-a5ce82a98a68-kube-api-access-nvbpv\" (UniqueName: \"kubernetes.io/projected/92776c04-7f5b-457a-be15-a5ce82a98a68-kube-api-access-nvbpv\") pod \"92776c04-7f5b-457a-be15-a5ce82a98a68\" (UID: \"92776c04-7f5b-457a-be15-a5ce82a98a68\") "
Oct 30 01:15:21.211466 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:21.211440    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71d53e61-57bc-4d66-a029-965ea868d660-maas-api-service-ca" pod "71d53e61-57bc-4d66-a029-965ea868d660" (UID: "71d53e61-57bc-4d66-a029-965ea868d660"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:15:21.211546 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:21.211444    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71d53e61-57bc-4d66-a029-965ea868d660-kube-api-access-4tmmh\" (UniqueName: \"kubernetes.io/projected/71d53e61-57bc-4d66-a029-965ea868d660-kube-api-access-4tmmh\") pod \"71d53e61-57bc-4d66-a029-965ea868d660\" (UID: \"71d53e61-57bc-4d66-a029-965ea868d660\") "
Oct 30 01:15:21.211546 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:21.211464    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/248fa0d9-fe68-4ba7-82c3-d7911c9b8070-maas-api-service-ca" pod "248fa0d9-fe68-4ba7-82c3-d7911c9b8070" (UID: "248fa0d9-fe68-4ba7-82c3-d7911c9b8070"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:15:21.211546 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:21.211522    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34ced72e-384a-4e43-90b7-2f907956c2c1-maas-api-service-ca" pod "34ced72e-384a-4e43-90b7-2f907956c2c1" (UID: "34ced72e-384a-4e43-90b7-2f907956c2c1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:15:21.211712 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:21.211545    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34ced72e-384a-4e43-90b7-2f907956c2c1-kube-api-access-h5lsv\" (UniqueName: \"kubernetes.io/projected/34ced72e-384a-4e43-90b7-2f907956c2c1-kube-api-access-h5lsv\") pod \"34ced72e-384a-4e43-90b7-2f907956c2c1\" (UID: \"34ced72e-384a-4e43-90b7-2f907956c2c1\") "
Oct 30 01:15:21.212318 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:21.212289    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/248fa0d9-fe68-4ba7-82c3-d7911c9b8070-maas-api-service-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:15:21.212500 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:21.212323    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/34ced72e-384a-4e43-90b7-2f907956c2c1-maas-api-service-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:15:21.212500 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:21.212338    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/92776c04-7f5b-457a-be15-a5ce82a98a68-maas-api-service-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:15:21.212500 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:21.212352    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/71d53e61-57bc-4d66-a029-965ea868d660-maas-api-service-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:15:21.213725 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:21.213696    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/248fa0d9-fe68-4ba7-82c3-d7911c9b8070-kube-api-access-rdjdh" pod "248fa0d9-fe68-4ba7-82c3-d7911c9b8070" (UID: "248fa0d9-fe68-4ba7-82c3-d7911c9b8070"). InnerVolumeSpecName "kube-api-access-rdjdh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:15:21.213833 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:21.213767    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71d53e61-57bc-4d66-a029-965ea868d660-kube-api-access-4tmmh" pod "71d53e61-57bc-4d66-a029-965ea868d660" (UID: "71d53e61-57bc-4d66-a029-965ea868d660"). InnerVolumeSpecName "kube-api-access-4tmmh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:15:21.213939 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:21.213921    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92776c04-7f5b-457a-be15-a5ce82a98a68-kube-api-access-nvbpv" pod "92776c04-7f5b-457a-be15-a5ce82a98a68" (UID: "92776c04-7f5b-457a-be15-a5ce82a98a68"). InnerVolumeSpecName "kube-api-access-nvbpv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:15:21.213980 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:21.213942    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34ced72e-384a-4e43-90b7-2f907956c2c1-kube-api-access-h5lsv" pod "34ced72e-384a-4e43-90b7-2f907956c2c1" (UID: "34ced72e-384a-4e43-90b7-2f907956c2c1"). InnerVolumeSpecName "kube-api-access-h5lsv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:15:21.390466 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:21.350505    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nvbpv\" (UniqueName: \"kubernetes.io/projected/92776c04-7f5b-457a-be15-a5ce82a98a68-kube-api-access-nvbpv\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:15:21.404292 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:21.390687    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4tmmh\" (UniqueName: \"kubernetes.io/projected/71d53e61-57bc-4d66-a029-965ea868d660-kube-api-access-4tmmh\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:15:21.415486 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:21.415442    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5lsv\" (UniqueName: \"kubernetes.io/projected/34ced72e-384a-4e43-90b7-2f907956c2c1-kube-api-access-h5lsv\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:15:21.431669 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:21.431636    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rdjdh\" (UniqueName: \"kubernetes.io/projected/248fa0d9-fe68-4ba7-82c3-d7911c9b8070-kube-api-access-rdjdh\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:15:21.677276 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:21.677241    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-11f521-29888715-fj8kn" event={"ID":"248fa0d9-fe68-4ba7-82c3-d7911c9b8070","Type":"ContainerDied","Data":"79b96f91575863859730f0e1a0ed49706c661ab7767f7bcaff975ee894802a6e"}
Oct 30 01:15:21.677682 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:21.677301    5002 scope.go:122] "RemoveContainer" containerID="8ff3b0500901205e10a71a95b0d47ecf59b07c2227fec1afd98e63b286148ee6"
Oct 30 01:15:21.677682 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:21.677396    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-11f521-29888715-fj8kn"
Oct 30 01:15:21.684751 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:21.684723    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-7f92f6-29888715-d7v2x" event={"ID":"34ced72e-384a-4e43-90b7-2f907956c2c1","Type":"ContainerDied","Data":"7e1e71ffd83542909bc6bfd2db790c897c6f7ea70d1bec97e92511c764ec7dca"}
Oct 30 01:15:21.684967 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:21.684950    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-7f92f6-29888715-d7v2x"
Oct 30 01:15:21.702154 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:21.699146    5002 scope.go:122] "RemoveContainer" containerID="5674fc59e947e49dc4fd07f1676f7f6c080e2a949a8ab4579e3d906eafb1702e"
Oct 30 01:15:21.735824 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:21.734706    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-01215f-29888715-bxdtr"
Oct 30 01:15:21.736486 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:21.736318    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-29be73-29888715-p2ctj"]
Oct 30 01:15:21.760166 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:21.751833    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-01215f-29888715-bxdtr" event={"ID":"92776c04-7f5b-457a-be15-a5ce82a98a68","Type":"ContainerDied","Data":"2a21bd55303fa74cfa2d16674d3b76c5e163746c03f51c8dd76b47e8c65219b3"}
Oct 30 01:15:21.760166 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:21.752004    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-021d6d-29888715-cdjpr" event={"ID":"aba84a8f-bfa1-4a2e-9569-8194bed6ec61","Type":"ContainerStarted","Data":"376f4a229095cb1c5991f71d395fa87a58bab69778ed6de44269592b3a867678"}
Oct 30 01:15:21.760166 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:21.752312    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-48df96-29888715-nc8tn" event={"ID":"71d53e61-57bc-4d66-a029-965ea868d660","Type":"ContainerDied","Data":"f1dd85b0d6309ca0d2e5857f3161f89cb9cd063868672bb2246128f734d70130"}
Oct 30 01:15:21.760166 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:21.752718    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-48df96-29888715-nc8tn"
Oct 30 01:15:21.760166 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:21.753216    5002 scope.go:122] "RemoveContainer" containerID="73714508be7d7627b4cbfa2b418d04356a6fd11a746288db2064cead1e17c4ba"
Oct 30 01:15:21.764505 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:21.763547    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5f387c-29888715-qgmjp" event={"ID":"8dcf8782-f46c-4c0f-9133-793017d66a37","Type":"ContainerStarted","Data":"732f8571b2d6934413b47ae3888c8d1c3e55bcf67becf663728e99a0f5f60070"}
Oct 30 01:15:21.947765 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:21.947737    5002 scope.go:122] "RemoveContainer" containerID="255c2a03584ce6c0ed51cf5d91952ffb8faaa215eb0bf2934a21b9e091c34670"
Oct 30 01:15:22.121338 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:22.121308    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-01215f-29888715-bxdtr"]
Oct 30 01:15:22.129112 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:22.129080    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-01215f-29888715-bxdtr"]
Oct 30 01:15:22.159414 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:22.158577    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-beeba4-29888715-5t886"
Oct 30 01:15:22.230818 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:22.230738    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-11f521-29888715-fj8kn"]
Oct 30 01:15:22.265433 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:22.251940    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-11f521-29888715-fj8kn"]
Oct 30 01:15:22.338320 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:22.338273    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-7f92f6-29888715-d7v2x"]
Oct 30 01:15:22.339790 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:22.339754    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55f94866-dc95-4ee1-b634-a3c434226a03-kube-api-access-p2w79\" (UniqueName: \"kubernetes.io/projected/55f94866-dc95-4ee1-b634-a3c434226a03-kube-api-access-p2w79\") pod \"55f94866-dc95-4ee1-b634-a3c434226a03\" (UID: \"55f94866-dc95-4ee1-b634-a3c434226a03\") "
Oct 30 01:15:22.339923 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:22.339905    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55f94866-dc95-4ee1-b634-a3c434226a03-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/55f94866-dc95-4ee1-b634-a3c434226a03-maas-api-service-ca\") pod \"55f94866-dc95-4ee1-b634-a3c434226a03\" (UID: \"55f94866-dc95-4ee1-b634-a3c434226a03\") "
Oct 30 01:15:22.348492 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:22.348461    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-7f92f6-29888715-d7v2x"]
Oct 30 01:15:22.376404 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:22.375497    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55f94866-dc95-4ee1-b634-a3c434226a03-maas-api-service-ca" pod "55f94866-dc95-4ee1-b634-a3c434226a03" (UID: "55f94866-dc95-4ee1-b634-a3c434226a03"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:15:22.444146 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:22.442367    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/55f94866-dc95-4ee1-b634-a3c434226a03-maas-api-service-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:15:22.475129 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:22.472985    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55f94866-dc95-4ee1-b634-a3c434226a03-kube-api-access-p2w79" pod "55f94866-dc95-4ee1-b634-a3c434226a03" (UID: "55f94866-dc95-4ee1-b634-a3c434226a03"). InnerVolumeSpecName "kube-api-access-p2w79". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:15:22.480665 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:22.479087    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-48df96-29888715-nc8tn"]
Oct 30 01:15:22.482504 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:22.482443    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-48df96-29888715-nc8tn"]
Oct 30 01:15:22.515642 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:22.515613    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-00e07a-29888715-8dq5b"
Oct 30 01:15:22.517590 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:22.517511    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-29be73-29888715-p2ctj"
Oct 30 01:15:22.519292 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:22.519252    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="248fa0d9-fe68-4ba7-82c3-d7911c9b8070" path="/var/lib/kubelet/pods/248fa0d9-fe68-4ba7-82c3-d7911c9b8070/volumes"
Oct 30 01:15:22.520363 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:22.520316    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34ced72e-384a-4e43-90b7-2f907956c2c1" path="/var/lib/kubelet/pods/34ced72e-384a-4e43-90b7-2f907956c2c1/volumes"
Oct 30 01:15:22.521194 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:22.521149    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5834beef-f5d2-4d8f-94da-aa3a86d692cf" path="/var/lib/kubelet/pods/5834beef-f5d2-4d8f-94da-aa3a86d692cf/volumes"
Oct 30 01:15:22.524679 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:22.522441    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71d53e61-57bc-4d66-a029-965ea868d660" path="/var/lib/kubelet/pods/71d53e61-57bc-4d66-a029-965ea868d660/volumes"
Oct 30 01:15:22.524679 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:22.523024    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7568f0d2-0dfb-47b9-ab97-d06d3c985d90" path="/var/lib/kubelet/pods/7568f0d2-0dfb-47b9-ab97-d06d3c985d90/volumes"
Oct 30 01:15:22.526708 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:22.525238    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92776c04-7f5b-457a-be15-a5ce82a98a68" path="/var/lib/kubelet/pods/92776c04-7f5b-457a-be15-a5ce82a98a68/volumes"
Oct 30 01:15:22.526708 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:22.525711    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd232d4c-4d2b-4137-bf83-b3e140fba4a6" path="/var/lib/kubelet/pods/fd232d4c-4d2b-4137-bf83-b3e140fba4a6/volumes"
Oct 30 01:15:22.555131 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:22.554069    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d965e6c-4e56-4dff-918d-c93b9832b72f-kube-api-access-h59hc\" (UniqueName: \"kubernetes.io/projected/5d965e6c-4e56-4dff-918d-c93b9832b72f-kube-api-access-h59hc\") pod \"5d965e6c-4e56-4dff-918d-c93b9832b72f\" (UID: \"5d965e6c-4e56-4dff-918d-c93b9832b72f\") "
Oct 30 01:15:22.555131 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:22.555093    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p2w79\" (UniqueName: \"kubernetes.io/projected/55f94866-dc95-4ee1-b634-a3c434226a03-kube-api-access-p2w79\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:15:22.557802 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:22.557775    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d965e6c-4e56-4dff-918d-c93b9832b72f-kube-api-access-h59hc" pod "5d965e6c-4e56-4dff-918d-c93b9832b72f" (UID: "5d965e6c-4e56-4dff-918d-c93b9832b72f"). InnerVolumeSpecName "kube-api-access-h59hc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:15:22.628684 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:22.628656    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-6d44c2-29888715-d8zbn"
Oct 30 01:15:22.632454 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:22.632162    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a7ca1f-29888715-r8jhp"
Oct 30 01:15:22.655889 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:22.655850    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/663c6827-07ce-4de7-8d82-033f8b60c129-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/663c6827-07ce-4de7-8d82-033f8b60c129-maas-api-service-ca\") pod \"663c6827-07ce-4de7-8d82-033f8b60c129\" (UID: \"663c6827-07ce-4de7-8d82-033f8b60c129\") "
Oct 30 01:15:22.656069 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:22.655929    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07d6239e-5b00-4c4a-b8dd-d935c4c44ed8-kube-api-access-dznkn\" (UniqueName: \"kubernetes.io/projected/07d6239e-5b00-4c4a-b8dd-d935c4c44ed8-kube-api-access-dznkn\") pod \"07d6239e-5b00-4c4a-b8dd-d935c4c44ed8\" (UID: \"07d6239e-5b00-4c4a-b8dd-d935c4c44ed8\") "
Oct 30 01:15:22.656069 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:22.655975    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d965e6c-4e56-4dff-918d-c93b9832b72f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d965e6c-4e56-4dff-918d-c93b9832b72f-maas-api-service-ca\") pod \"5d965e6c-4e56-4dff-918d-c93b9832b72f\" (UID: \"5d965e6c-4e56-4dff-918d-c93b9832b72f\") "
Oct 30 01:15:22.656069 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:22.656010    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82b3d01a-515b-43fe-b05d-ae6d90c678a3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/82b3d01a-515b-43fe-b05d-ae6d90c678a3-maas-api-service-ca\") pod \"82b3d01a-515b-43fe-b05d-ae6d90c678a3\" (UID: \"82b3d01a-515b-43fe-b05d-ae6d90c678a3\") "
Oct 30 01:15:22.656261 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:22.656151    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07d6239e-5b00-4c4a-b8dd-d935c4c44ed8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/07d6239e-5b00-4c4a-b8dd-d935c4c44ed8-maas-api-service-ca\") pod \"07d6239e-5b00-4c4a-b8dd-d935c4c44ed8\" (UID: \"07d6239e-5b00-4c4a-b8dd-d935c4c44ed8\") "
Oct 30 01:15:22.656261 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:22.656188    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/663c6827-07ce-4de7-8d82-033f8b60c129-kube-api-access-lgshb\" (UniqueName: \"kubernetes.io/projected/663c6827-07ce-4de7-8d82-033f8b60c129-kube-api-access-lgshb\") pod \"663c6827-07ce-4de7-8d82-033f8b60c129\" (UID: \"663c6827-07ce-4de7-8d82-033f8b60c129\") "
Oct 30 01:15:22.656399 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:22.656343    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82b3d01a-515b-43fe-b05d-ae6d90c678a3-kube-api-access-skx8g\" (UniqueName: \"kubernetes.io/projected/82b3d01a-515b-43fe-b05d-ae6d90c678a3-kube-api-access-skx8g\") pod \"82b3d01a-515b-43fe-b05d-ae6d90c678a3\" (UID: \"82b3d01a-515b-43fe-b05d-ae6d90c678a3\") "
Oct 30 01:15:22.656893 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:22.656480    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82b3d01a-515b-43fe-b05d-ae6d90c678a3-maas-api-service-ca" pod "82b3d01a-515b-43fe-b05d-ae6d90c678a3" (UID: "82b3d01a-515b-43fe-b05d-ae6d90c678a3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:15:22.656893 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:22.656849    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07d6239e-5b00-4c4a-b8dd-d935c4c44ed8-maas-api-service-ca" pod "07d6239e-5b00-4c4a-b8dd-d935c4c44ed8" (UID: "07d6239e-5b00-4c4a-b8dd-d935c4c44ed8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:15:22.657073 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:22.656942    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/82b3d01a-515b-43fe-b05d-ae6d90c678a3-maas-api-service-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:15:22.657073 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:22.656966    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h59hc\" (UniqueName: \"kubernetes.io/projected/5d965e6c-4e56-4dff-918d-c93b9832b72f-kube-api-access-h59hc\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:15:22.657506 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:22.657475    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d965e6c-4e56-4dff-918d-c93b9832b72f-maas-api-service-ca" pod "5d965e6c-4e56-4dff-918d-c93b9832b72f" (UID: "5d965e6c-4e56-4dff-918d-c93b9832b72f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:15:22.657861 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:22.657841    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/663c6827-07ce-4de7-8d82-033f8b60c129-maas-api-service-ca" pod "663c6827-07ce-4de7-8d82-033f8b60c129" (UID: "663c6827-07ce-4de7-8d82-033f8b60c129"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:15:22.659506 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:22.659483    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/663c6827-07ce-4de7-8d82-033f8b60c129-kube-api-access-lgshb" pod "663c6827-07ce-4de7-8d82-033f8b60c129" (UID: "663c6827-07ce-4de7-8d82-033f8b60c129"). InnerVolumeSpecName "kube-api-access-lgshb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:15:22.662669 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:22.662642    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82b3d01a-515b-43fe-b05d-ae6d90c678a3-kube-api-access-skx8g" pod "82b3d01a-515b-43fe-b05d-ae6d90c678a3" (UID: "82b3d01a-515b-43fe-b05d-ae6d90c678a3"). InnerVolumeSpecName "kube-api-access-skx8g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:15:22.662800 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:22.662783    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07d6239e-5b00-4c4a-b8dd-d935c4c44ed8-kube-api-access-dznkn" pod "07d6239e-5b00-4c4a-b8dd-d935c4c44ed8" (UID: "07d6239e-5b00-4c4a-b8dd-d935c4c44ed8"). InnerVolumeSpecName "kube-api-access-dznkn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:15:22.758269 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:22.758161    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/07d6239e-5b00-4c4a-b8dd-d935c4c44ed8-maas-api-service-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:15:22.758269 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:22.758200    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lgshb\" (UniqueName: \"kubernetes.io/projected/663c6827-07ce-4de7-8d82-033f8b60c129-kube-api-access-lgshb\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:15:22.758269 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:22.758216    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-skx8g\" (UniqueName: \"kubernetes.io/projected/82b3d01a-515b-43fe-b05d-ae6d90c678a3-kube-api-access-skx8g\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:15:22.758269 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:22.758229    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/663c6827-07ce-4de7-8d82-033f8b60c129-maas-api-service-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:15:22.758269 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:22.758246    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dznkn\" (UniqueName: \"kubernetes.io/projected/07d6239e-5b00-4c4a-b8dd-d935c4c44ed8-kube-api-access-dznkn\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:15:22.758269 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:22.758260    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d965e6c-4e56-4dff-918d-c93b9832b72f-maas-api-service-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:15:22.779834 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:22.779802    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-beeba4-29888715-5t886"
Oct 30 01:15:22.780128 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:22.779833    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-beeba4-29888715-5t886" event={"ID":"55f94866-dc95-4ee1-b634-a3c434226a03","Type":"ContainerDied","Data":"47c109df8632bce8bce5b4257e38ed2e040696e9ce420a06bd8d3590b49e2478"}
Oct 30 01:15:22.780322 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:22.780307    5002 scope.go:122] "RemoveContainer" containerID="e6d8a59d8e01248217c97a89ef4ef1b930fc2179be1bf8514ab6049f3d906319"
Oct 30 01:15:22.802123 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:22.802068    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-6d44c2-29888715-d8zbn"
Oct 30 01:15:22.802123 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:22.802087    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-6d44c2-29888715-d8zbn" event={"ID":"07d6239e-5b00-4c4a-b8dd-d935c4c44ed8","Type":"ContainerDied","Data":"33b7081cfc0f0f76484d465f808326f485b12bbee65ea4c54fb76c88f19013b1"}
Oct 30 01:15:22.802348 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:22.802159    5002 scope.go:122] "RemoveContainer" containerID="73cbf10aa23091f2a934e7170773dc5e0d84204d35291a66cdcb6b9089b4ccab"
Oct 30 01:15:22.804592 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:22.804572    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-29be73-29888715-p2ctj"
Oct 30 01:15:22.804719 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:22.804608    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-29be73-29888715-p2ctj" event={"ID":"5d965e6c-4e56-4dff-918d-c93b9832b72f","Type":"ContainerDied","Data":"db7b74e9959f9257f1e0322aa9ad0910875293ffa18409f16c54e60d8e7fb1a0"}
Oct 30 01:15:22.812355 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:22.806311    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-00e07a-29888715-8dq5b" event={"ID":"82b3d01a-515b-43fe-b05d-ae6d90c678a3","Type":"ContainerDied","Data":"9c193892e89fed33c4a6390183cd7dad03e7ff5d7432a11ce3690beef07ea944"}
Oct 30 01:15:22.812355 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:22.806421    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-00e07a-29888715-8dq5b"
Oct 30 01:15:22.814235 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:22.814143    5002 generic.go:356] "Generic (PLEG): container finished" podID="aba84a8f-bfa1-4a2e-9569-8194bed6ec61" containerID="376f4a229095cb1c5991f71d395fa87a58bab69778ed6de44269592b3a867678" exitCode=60
Oct 30 01:15:22.814317 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:22.814231    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-021d6d-29888715-cdjpr" event={"ID":"aba84a8f-bfa1-4a2e-9569-8194bed6ec61","Type":"ContainerDied","Data":"376f4a229095cb1c5991f71d395fa87a58bab69778ed6de44269592b3a867678"}
Oct 30 01:15:22.814795 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:22.814769    5002 scope.go:122] "RemoveContainer" containerID="376f4a229095cb1c5991f71d395fa87a58bab69778ed6de44269592b3a867678"
Oct 30 01:15:22.815162 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:15:22.815133    5002 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-021d6d-29888715-cdjpr_odh-ai-gateway-infra(aba84a8f-bfa1-4a2e-9569-8194bed6ec61)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-021d6d-29888715-cdjpr" podUID="aba84a8f-bfa1-4a2e-9569-8194bed6ec61"
Oct 30 01:15:22.819125 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:22.819106    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a7ca1f-29888715-r8jhp"
Oct 30 01:15:22.819229 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:22.819130    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a7ca1f-29888715-r8jhp" event={"ID":"663c6827-07ce-4de7-8d82-033f8b60c129","Type":"ContainerDied","Data":"34e4f7315658b81f5cfe97e992e00503cb7b9299e7c348c6593a3d5b6ede24e0"}
Oct 30 01:15:22.819895 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:22.819877    5002 scope.go:122] "RemoveContainer" containerID="7f84e1267e79b985c7ed94248089f75ffe7fa5e911ff50fcf32343e00c6ab09e"
Oct 30 01:15:22.821213 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:22.821193    5002 generic.go:356] "Generic (PLEG): container finished" podID="4c140dcb-40bd-4cd2-ac16-df81ed774c16" containerID="d8efe79ab9d9d2cee14f023376facf07f6bf3d680db7bd4489893d8432f8bc3d" exitCode=0
Oct 30 01:15:22.821301 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:22.821286    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-01215f-openshift-default-95bfd666f-rkm78" event={"ID":"4c140dcb-40bd-4cd2-ac16-df81ed774c16","Type":"ContainerDied","Data":"d8efe79ab9d9d2cee14f023376facf07f6bf3d680db7bd4489893d8432f8bc3d"}
Oct 30 01:15:22.821823 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:22.821804    5002 scope.go:122] "RemoveContainer" containerID="d8efe79ab9d9d2cee14f023376facf07f6bf3d680db7bd4489893d8432f8bc3d"
Oct 30 01:15:22.824165 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:22.824141    5002 generic.go:356] "Generic (PLEG): container finished" podID="8dcf8782-f46c-4c0f-9133-793017d66a37" containerID="732f8571b2d6934413b47ae3888c8d1c3e55bcf67becf663728e99a0f5f60070" exitCode=60
Oct 30 01:15:22.824243 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:22.824194    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5f387c-29888715-qgmjp" event={"ID":"8dcf8782-f46c-4c0f-9133-793017d66a37","Type":"ContainerDied","Data":"732f8571b2d6934413b47ae3888c8d1c3e55bcf67becf663728e99a0f5f60070"}
Oct 30 01:15:22.824304 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:22.824245    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:15:22.824359 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:22.824339    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:15:22.824444 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:22.824415    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:15:22.826550 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:22.826504    5002 scope.go:122] "RemoveContainer" containerID="732f8571b2d6934413b47ae3888c8d1c3e55bcf67becf663728e99a0f5f60070"
Oct 30 01:15:22.833525 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:15:22.833500    5002 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-5f387c-29888715-qgmjp_odh-ai-gateway-infra(8dcf8782-f46c-4c0f-9133-793017d66a37)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5f387c-29888715-qgmjp" podUID="8dcf8782-f46c-4c0f-9133-793017d66a37"
Oct 30 01:15:22.837003 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:22.836977    5002 scope.go:122] "RemoveContainer" containerID="652a74c435342050fff5844397565f7de79f038ad2731f8535b33122fe1fef1f"
Oct 30 01:15:22.860262 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:22.860233    5002 scope.go:122] "RemoveContainer" containerID="fe7a45761c593f0a164c74318faf5ae1530d8823c3e8d4021d1dbf53be6a04ac"
Oct 30 01:15:22.891984 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:22.891961    5002 scope.go:122] "RemoveContainer" containerID="f80558127124592773fc7079a2faec9a66e7f1724ad0537746cb39dcd4049259"
Oct 30 01:15:22.904789 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:22.904763    5002 scope.go:122] "RemoveContainer" containerID="b4a475af6879a21a3dc26a89fdb0efe18a0a29c561d79be4c06cc7b9deee6190"
Oct 30 01:15:22.909907 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:22.908927    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a7ca1f-29888715-r8jhp"]
Oct 30 01:15:22.913481 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:22.913460    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a7ca1f-29888715-r8jhp"]
Oct 30 01:15:22.993477 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:22.993437    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-29be73-29888715-p2ctj"]
Oct 30 01:15:22.999784 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:22.999751    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-29be73-29888715-p2ctj"]
Oct 30 01:15:23.072188 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:23.072088    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-021d6d-29888715-cdjpr"]
Oct 30 01:15:23.084924 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:23.084878    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-beeba4-29888715-5t886"]
Oct 30 01:15:23.092036 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:23.091998    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-beeba4-29888715-5t886"]
Oct 30 01:15:23.177663 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:23.177632    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-00e07a-29888715-8dq5b"]
Oct 30 01:15:23.186291 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:23.186242    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-00e07a-29888715-8dq5b"]
Oct 30 01:15:23.220028 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:23.219995    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-6d44c2-29888715-d8zbn"]
Oct 30 01:15:23.228325 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:23.228283    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-6d44c2-29888715-d8zbn"]
Oct 30 01:15:23.511506 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:23.511467    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5f387c-29888715-qgmjp"]
Oct 30 01:15:23.836717 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:23.836616    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-01215f-openshift-default-95bfd666f-rkm78" event={"ID":"4c140dcb-40bd-4cd2-ac16-df81ed774c16","Type":"ContainerStarted","Data":"45fe47611a2d211358b789fd525622a10e810b445708603137038cbe3b5ccc34"}
Oct 30 01:15:24.023162 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:24.023130    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-01215f-openshift-default-95bfd666f-rkm78"
Oct 30 01:15:24.024321 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:24.024295    5002 patch_prober.go:29] interesting pod/e2e-ipp-a-01215f-openshift-default-95bfd666f-rkm78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" start-of-body=
Oct 30 01:15:24.024629 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:24.024463    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-01215f-openshift-default-95bfd666f-rkm78" podUID="4c140dcb-40bd-4cd2-ac16-df81ed774c16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused"
Oct 30 01:15:24.136180 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:24.136148    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-021d6d-29888715-cdjpr"
Oct 30 01:15:24.217725 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:24.214766    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aba84a8f-bfa1-4a2e-9569-8194bed6ec61-kube-api-access-v8mjg\" (UniqueName: \"kubernetes.io/projected/aba84a8f-bfa1-4a2e-9569-8194bed6ec61-kube-api-access-v8mjg\") pod \"aba84a8f-bfa1-4a2e-9569-8194bed6ec61\" (UID: \"aba84a8f-bfa1-4a2e-9569-8194bed6ec61\") "
Oct 30 01:15:24.217725 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:24.215007    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aba84a8f-bfa1-4a2e-9569-8194bed6ec61-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aba84a8f-bfa1-4a2e-9569-8194bed6ec61-maas-api-service-ca\") pod \"aba84a8f-bfa1-4a2e-9569-8194bed6ec61\" (UID: \"aba84a8f-bfa1-4a2e-9569-8194bed6ec61\") "
Oct 30 01:15:24.217725 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:24.215860    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aba84a8f-bfa1-4a2e-9569-8194bed6ec61-maas-api-service-ca" pod "aba84a8f-bfa1-4a2e-9569-8194bed6ec61" (UID: "aba84a8f-bfa1-4a2e-9569-8194bed6ec61"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:15:24.219363 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:24.219345    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5f387c-29888715-qgmjp"
Oct 30 01:15:24.219592 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:24.219477    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aba84a8f-bfa1-4a2e-9569-8194bed6ec61-kube-api-access-v8mjg" pod "aba84a8f-bfa1-4a2e-9569-8194bed6ec61" (UID: "aba84a8f-bfa1-4a2e-9569-8194bed6ec61"). InnerVolumeSpecName "kube-api-access-v8mjg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:15:24.316555 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:24.316516    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8dcf8782-f46c-4c0f-9133-793017d66a37-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8dcf8782-f46c-4c0f-9133-793017d66a37-maas-api-service-ca\") pod \"8dcf8782-f46c-4c0f-9133-793017d66a37\" (UID: \"8dcf8782-f46c-4c0f-9133-793017d66a37\") "
Oct 30 01:15:24.316740 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:24.316589    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8dcf8782-f46c-4c0f-9133-793017d66a37-kube-api-access-wxvs8\" (UniqueName: \"kubernetes.io/projected/8dcf8782-f46c-4c0f-9133-793017d66a37-kube-api-access-wxvs8\") pod \"8dcf8782-f46c-4c0f-9133-793017d66a37\" (UID: \"8dcf8782-f46c-4c0f-9133-793017d66a37\") "
Oct 30 01:15:24.317022 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:24.316991    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8dcf8782-f46c-4c0f-9133-793017d66a37-maas-api-service-ca" pod "8dcf8782-f46c-4c0f-9133-793017d66a37" (UID: "8dcf8782-f46c-4c0f-9133-793017d66a37"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:15:24.317186 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:24.317120    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aba84a8f-bfa1-4a2e-9569-8194bed6ec61-maas-api-service-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:15:24.317283 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:24.317271    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8mjg\" (UniqueName: \"kubernetes.io/projected/aba84a8f-bfa1-4a2e-9569-8194bed6ec61-kube-api-access-v8mjg\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:15:24.323920 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:24.323887    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8dcf8782-f46c-4c0f-9133-793017d66a37-kube-api-access-wxvs8" pod "8dcf8782-f46c-4c0f-9133-793017d66a37" (UID: "8dcf8782-f46c-4c0f-9133-793017d66a37"). InnerVolumeSpecName "kube-api-access-wxvs8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:15:24.419138 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:24.419044    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8dcf8782-f46c-4c0f-9133-793017d66a37-maas-api-service-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:15:24.419138 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:24.419078    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wxvs8\" (UniqueName: \"kubernetes.io/projected/8dcf8782-f46c-4c0f-9133-793017d66a37-kube-api-access-wxvs8\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:15:24.500211 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:24.500171    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07d6239e-5b00-4c4a-b8dd-d935c4c44ed8" path="/var/lib/kubelet/pods/07d6239e-5b00-4c4a-b8dd-d935c4c44ed8/volumes"
Oct 30 01:15:24.500760 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:24.500740    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55f94866-dc95-4ee1-b634-a3c434226a03" path="/var/lib/kubelet/pods/55f94866-dc95-4ee1-b634-a3c434226a03/volumes"
Oct 30 01:15:24.501165 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:24.501150    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d965e6c-4e56-4dff-918d-c93b9832b72f" path="/var/lib/kubelet/pods/5d965e6c-4e56-4dff-918d-c93b9832b72f/volumes"
Oct 30 01:15:24.501950 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:24.501930    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="663c6827-07ce-4de7-8d82-033f8b60c129" path="/var/lib/kubelet/pods/663c6827-07ce-4de7-8d82-033f8b60c129/volumes"
Oct 30 01:15:24.502360 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:24.502346    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82b3d01a-515b-43fe-b05d-ae6d90c678a3" path="/var/lib/kubelet/pods/82b3d01a-515b-43fe-b05d-ae6d90c678a3/volumes"
Oct 30 01:15:24.871019 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:24.870961    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-021d6d-29888715-cdjpr" event={"ID":"aba84a8f-bfa1-4a2e-9569-8194bed6ec61","Type":"ContainerDied","Data":"2d45cfec5ae49877c5e72b80fcebb0c08066c07ce0dcdebcecd39745cca5db7a"}
Oct 30 01:15:24.871661 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:24.871641    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-021d6d-29888715-cdjpr"
Oct 30 01:15:24.871797 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:24.871637    5002 scope.go:122] "RemoveContainer" containerID="376f4a229095cb1c5991f71d395fa87a58bab69778ed6de44269592b3a867678"
Oct 30 01:15:24.883715 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:24.883678    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5f387c-29888715-qgmjp"
Oct 30 01:15:24.895469 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:24.895426    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5f387c-29888715-qgmjp" event={"ID":"8dcf8782-f46c-4c0f-9133-793017d66a37","Type":"ContainerDied","Data":"3eed03255dbdbafc2d803d0bbe38c65c13b23a9f4e647518f797c3c2fd457e44"}
Oct 30 01:15:24.910036 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:24.910009    5002 scope.go:122] "RemoveContainer" containerID="732f8571b2d6934413b47ae3888c8d1c3e55bcf67becf663728e99a0f5f60070"
Oct 30 01:15:24.997692 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:24.997644    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-021d6d-29888715-cdjpr"]
Oct 30 01:15:25.023516 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:25.023478    5002 patch_prober.go:29] interesting pod/e2e-ipp-a-01215f-openshift-default-95bfd666f-rkm78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" start-of-body=
Oct 30 01:15:25.024066 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:25.024028    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-01215f-openshift-default-95bfd666f-rkm78" podUID="4c140dcb-40bd-4cd2-ac16-df81ed774c16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused"
Oct 30 01:15:25.026100 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:25.026069    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-021d6d-29888715-cdjpr"]
Oct 30 01:15:25.034486 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:25.034429    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5f387c-29888715-qgmjp"]
Oct 30 01:15:25.039893 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:25.039859    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5f387c-29888715-qgmjp"]
Oct 30 01:15:25.451917 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:25.451883    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-f6d5b8-865b864594-d5n25"
Oct 30 01:15:26.018263 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:26.018226    5002 patch_prober.go:29] interesting pod/e2e-ipp-a-01215f-openshift-default-95bfd666f-rkm78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" start-of-body=
Oct 30 01:15:26.018907 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:26.018864    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-01215f-openshift-default-95bfd666f-rkm78" podUID="4c140dcb-40bd-4cd2-ac16-df81ed774c16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused"
Oct 30 01:15:26.471522 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:26.470609    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-cf4567-8cf8db99b-rw6qt"]
Oct 30 01:15:26.654267 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:26.640511    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8dcf8782-f46c-4c0f-9133-793017d66a37" path="/var/lib/kubelet/pods/8dcf8782-f46c-4c0f-9133-793017d66a37/volumes"
Oct 30 01:15:26.654267 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:26.641364    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aba84a8f-bfa1-4a2e-9569-8194bed6ec61" path="/var/lib/kubelet/pods/aba84a8f-bfa1-4a2e-9569-8194bed6ec61/volumes"
Oct 30 01:15:26.654267 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:26.642157    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-cf4567-9f6f5f475-6wcvd"]
Oct 30 01:15:26.654267 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:26.642262    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f6d5b8-55c9ff76c5-xh6w2"
Oct 30 01:15:27.019326 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:27.019289    5002 patch_prober.go:29] interesting pod/e2e-ipp-a-01215f-openshift-default-95bfd666f-rkm78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" start-of-body=
Oct 30 01:15:27.019741 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:27.019345    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-01215f-openshift-default-95bfd666f-rkm78" podUID="4c140dcb-40bd-4cd2-ac16-df81ed774c16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused"
Oct 30 01:15:28.018780 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:28.018744    5002 patch_prober.go:29] interesting pod/e2e-ipp-a-01215f-openshift-default-95bfd666f-rkm78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" start-of-body=
Oct 30 01:15:28.018978 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:28.018801    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-01215f-openshift-default-95bfd666f-rkm78" podUID="4c140dcb-40bd-4cd2-ac16-df81ed774c16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused"
Oct 30 01:15:29.018003 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:29.017969    5002 patch_prober.go:29] interesting pod/e2e-ipp-a-01215f-openshift-default-95bfd666f-rkm78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" start-of-body=
Oct 30 01:15:29.018425 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:29.018034    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-01215f-openshift-default-95bfd666f-rkm78" podUID="4c140dcb-40bd-4cd2-ac16-df81ed774c16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused"
Oct 30 01:15:30.023741 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:30.023708    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-01215f-openshift-default-95bfd666f-rkm78"
Oct 30 01:15:30.024280 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:30.024255    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-01215f-openshift-default-95bfd666f-rkm78"
Oct 30 01:15:30.024977 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:30.024949    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-01215f-openshift-default-95bfd666f-rkm78"
Oct 30 01:15:31.471287 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:31.471243    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-cf4567-8cf8db99b-rw6qt" podUID="6ad27632-6ab9-4b4f-bf4f-3afa10eed73b" containerName="payload-processing" containerID="cri-o://1e68e98c9f174b88b569aaafc457d9be0f3cfb05c29e0b22e5019d0c745f7db6" gracePeriod=25
Oct 30 01:15:31.644000 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:31.643959    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cf4567-9f6f5f475-6wcvd" podUID="466ca974-a672-4f29-81c0-5e90b552b583" containerName="payload-pre-processing" containerID="cri-o://0d549f036e5aafd0e87361b3a9d9f96d62bd8ed3d27fbfca33df93a58919fd26" gracePeriod=25
Oct 30 01:15:31.801858 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:31.801822    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-cf4567-8cf8db99b-rw6qt"
Oct 30 01:15:31.803527 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:31.802609    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ad27632-6ab9-4b4f-bf4f-3afa10eed73b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ad27632-6ab9-4b4f-bf4f-3afa10eed73b-plugins-config-volume\") pod \"6ad27632-6ab9-4b4f-bf4f-3afa10eed73b\" (UID: \"6ad27632-6ab9-4b4f-bf4f-3afa10eed73b\") "
Oct 30 01:15:31.803527 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:31.802883    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ad27632-6ab9-4b4f-bf4f-3afa10eed73b-kube-api-access-nkdd9\" (UniqueName: \"kubernetes.io/projected/6ad27632-6ab9-4b4f-bf4f-3afa10eed73b-kube-api-access-nkdd9\") pod \"6ad27632-6ab9-4b4f-bf4f-3afa10eed73b\" (UID: \"6ad27632-6ab9-4b4f-bf4f-3afa10eed73b\") "
Oct 30 01:15:31.803527 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:31.802932    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ad27632-6ab9-4b4f-bf4f-3afa10eed73b-plugins-config-volume" pod "6ad27632-6ab9-4b4f-bf4f-3afa10eed73b" (UID: "6ad27632-6ab9-4b4f-bf4f-3afa10eed73b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:15:31.803527 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:31.803502    5002 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ad27632-6ab9-4b4f-bf4f-3afa10eed73b-plugins-config-volume\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:15:31.804941 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:31.804915    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ad27632-6ab9-4b4f-bf4f-3afa10eed73b-kube-api-access-nkdd9" pod "6ad27632-6ab9-4b4f-bf4f-3afa10eed73b" (UID: "6ad27632-6ab9-4b4f-bf4f-3afa10eed73b"). InnerVolumeSpecName "kube-api-access-nkdd9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:15:31.904367 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:31.904327    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nkdd9\" (UniqueName: \"kubernetes.io/projected/6ad27632-6ab9-4b4f-bf4f-3afa10eed73b-kube-api-access-nkdd9\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:15:31.913257 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:31.913231    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cf4567-9f6f5f475-6wcvd"
Oct 30 01:15:32.005347 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:32.005297    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/466ca974-a672-4f29-81c0-5e90b552b583-kube-api-access-8dgst\" (UniqueName: \"kubernetes.io/projected/466ca974-a672-4f29-81c0-5e90b552b583-kube-api-access-8dgst\") pod \"466ca974-a672-4f29-81c0-5e90b552b583\" (UID: \"466ca974-a672-4f29-81c0-5e90b552b583\") "
Oct 30 01:15:32.005567 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:32.005407    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/466ca974-a672-4f29-81c0-5e90b552b583-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/466ca974-a672-4f29-81c0-5e90b552b583-plugins-config-volume\") pod \"466ca974-a672-4f29-81c0-5e90b552b583\" (UID: \"466ca974-a672-4f29-81c0-5e90b552b583\") "
Oct 30 01:15:32.005908 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:32.005879    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/466ca974-a672-4f29-81c0-5e90b552b583-plugins-config-volume" pod "466ca974-a672-4f29-81c0-5e90b552b583" (UID: "466ca974-a672-4f29-81c0-5e90b552b583"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:15:32.006143 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:32.006124    5002 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/466ca974-a672-4f29-81c0-5e90b552b583-plugins-config-volume\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:15:32.008195 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:32.008128    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/466ca974-a672-4f29-81c0-5e90b552b583-kube-api-access-8dgst" pod "466ca974-a672-4f29-81c0-5e90b552b583" (UID: "466ca974-a672-4f29-81c0-5e90b552b583"). InnerVolumeSpecName "kube-api-access-8dgst". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:15:32.108042 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:32.108002    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8dgst\" (UniqueName: \"kubernetes.io/projected/466ca974-a672-4f29-81c0-5e90b552b583-kube-api-access-8dgst\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:15:32.204390 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:32.204339    5002 generic.go:356] "Generic (PLEG): container finished" podID="466ca974-a672-4f29-81c0-5e90b552b583" containerID="0d549f036e5aafd0e87361b3a9d9f96d62bd8ed3d27fbfca33df93a58919fd26" exitCode=0
Oct 30 01:15:32.204561 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:32.204423    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cf4567-9f6f5f475-6wcvd"
Oct 30 01:15:32.204561 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:32.204433    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cf4567-9f6f5f475-6wcvd" event={"ID":"466ca974-a672-4f29-81c0-5e90b552b583","Type":"ContainerDied","Data":"0d549f036e5aafd0e87361b3a9d9f96d62bd8ed3d27fbfca33df93a58919fd26"}
Oct 30 01:15:32.204561 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:32.204496    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cf4567-9f6f5f475-6wcvd" event={"ID":"466ca974-a672-4f29-81c0-5e90b552b583","Type":"ContainerDied","Data":"99ea81244f35423aef4e45cb663afafdf7ff47437f4fd7cd6e0f5f18cb15c9f1"}
Oct 30 01:15:32.204561 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:32.204520    5002 scope.go:122] "RemoveContainer" containerID="0d549f036e5aafd0e87361b3a9d9f96d62bd8ed3d27fbfca33df93a58919fd26"
Oct 30 01:15:32.206267 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:32.206236    5002 generic.go:356] "Generic (PLEG): container finished" podID="6ad27632-6ab9-4b4f-bf4f-3afa10eed73b" containerID="1e68e98c9f174b88b569aaafc457d9be0f3cfb05c29e0b22e5019d0c745f7db6" exitCode=0
Oct 30 01:15:32.206366 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:32.206278    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-cf4567-8cf8db99b-rw6qt"
Oct 30 01:15:32.206366 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:32.206309    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-cf4567-8cf8db99b-rw6qt" event={"ID":"6ad27632-6ab9-4b4f-bf4f-3afa10eed73b","Type":"ContainerDied","Data":"1e68e98c9f174b88b569aaafc457d9be0f3cfb05c29e0b22e5019d0c745f7db6"}
Oct 30 01:15:32.206475 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:32.206364    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-cf4567-8cf8db99b-rw6qt" event={"ID":"6ad27632-6ab9-4b4f-bf4f-3afa10eed73b","Type":"ContainerDied","Data":"7b200552bc6edfab0fb07909a8591ce57fc6559036a727dd14b27b2458a079b5"}
Oct 30 01:15:32.217330 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:32.217309    5002 scope.go:122] "RemoveContainer" containerID="0d549f036e5aafd0e87361b3a9d9f96d62bd8ed3d27fbfca33df93a58919fd26"
Oct 30 01:15:32.217673 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:15:32.217652    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d549f036e5aafd0e87361b3a9d9f96d62bd8ed3d27fbfca33df93a58919fd26\": container with ID starting with 0d549f036e5aafd0e87361b3a9d9f96d62bd8ed3d27fbfca33df93a58919fd26 not found: ID does not exist" containerID="0d549f036e5aafd0e87361b3a9d9f96d62bd8ed3d27fbfca33df93a58919fd26"
Oct 30 01:15:32.217751 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:32.217682    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d549f036e5aafd0e87361b3a9d9f96d62bd8ed3d27fbfca33df93a58919fd26"} err="failed to get container status \"0d549f036e5aafd0e87361b3a9d9f96d62bd8ed3d27fbfca33df93a58919fd26\": rpc error: code = NotFound desc = could not find container \"0d549f036e5aafd0e87361b3a9d9f96d62bd8ed3d27fbfca33df93a58919fd26\": container with ID starting with 0d549f036e5aafd0e87361b3a9d9f96d62bd8ed3d27fbfca33df93a58919fd26 not found: ID does not exist"
Oct 30 01:15:32.217751 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:32.217699    5002 scope.go:122] "RemoveContainer" containerID="1e68e98c9f174b88b569aaafc457d9be0f3cfb05c29e0b22e5019d0c745f7db6"
Oct 30 01:15:32.231490 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:32.231462    5002 scope.go:122] "RemoveContainer" containerID="1e68e98c9f174b88b569aaafc457d9be0f3cfb05c29e0b22e5019d0c745f7db6"
Oct 30 01:15:32.232058 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:15:32.232032    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e68e98c9f174b88b569aaafc457d9be0f3cfb05c29e0b22e5019d0c745f7db6\": container with ID starting with 1e68e98c9f174b88b569aaafc457d9be0f3cfb05c29e0b22e5019d0c745f7db6 not found: ID does not exist" containerID="1e68e98c9f174b88b569aaafc457d9be0f3cfb05c29e0b22e5019d0c745f7db6"
Oct 30 01:15:32.232188 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:32.232070    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e68e98c9f174b88b569aaafc457d9be0f3cfb05c29e0b22e5019d0c745f7db6"} err="failed to get container status \"1e68e98c9f174b88b569aaafc457d9be0f3cfb05c29e0b22e5019d0c745f7db6\": rpc error: code = NotFound desc = could not find container \"1e68e98c9f174b88b569aaafc457d9be0f3cfb05c29e0b22e5019d0c745f7db6\": container with ID starting with 1e68e98c9f174b88b569aaafc457d9be0f3cfb05c29e0b22e5019d0c745f7db6 not found: ID does not exist"
Oct 30 01:15:32.263846 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:32.263766    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-cf4567-8cf8db99b-rw6qt"]
Oct 30 01:15:32.269641 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:32.269610    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-cf4567-8cf8db99b-rw6qt"]
Oct 30 01:15:32.308602 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:32.308563    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-cf4567-9f6f5f475-6wcvd"]
Oct 30 01:15:32.312726 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:32.312698    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-cf4567-9f6f5f475-6wcvd"]
Oct 30 01:15:32.565532 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:32.553401    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="466ca974-a672-4f29-81c0-5e90b552b583" path="/var/lib/kubelet/pods/466ca974-a672-4f29-81c0-5e90b552b583/volumes"
Oct 30 01:15:32.565532 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:32.554116    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ad27632-6ab9-4b4f-bf4f-3afa10eed73b" path="/var/lib/kubelet/pods/6ad27632-6ab9-4b4f-bf4f-3afa10eed73b/volumes"
Oct 30 01:15:38.448909 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:38.448862    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-11f521-openshift-default-96689b67c-p9ptg"]
Oct 30 01:15:38.449284 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:38.449115    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-11f521-openshift-default-96689b67c-p9ptg" podUID="412d8cc8-d5e9-4d37-9021-baeda679711a" containerName="istio-proxy" containerID="cri-o://de66c75782cb4c00455584cea03af158133e50b7e124f6f29231156fb0cc75a9" gracePeriod=30
Oct 30 01:15:39.346450 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:39.346375    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-00e07a-d72x6"]
Oct 30 01:15:39.347032 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:39.347019    5002 cpu_manager.go:414] "containerMap: removing container" podUID="5834beef-f5d2-4d8f-94da-aa3a86d692cf" containerName="cleanup"
Oct 30 01:15:39.347073 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:39.347034    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5834beef-f5d2-4d8f-94da-aa3a86d692cf" containerName="cleanup"
Oct 30 01:15:39.347073 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:39.347043    5002 cpu_manager.go:414] "containerMap: removing container" podUID="71d53e61-57bc-4d66-a029-965ea868d660" containerName="cleanup"
Oct 30 01:15:39.347073 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:39.347048    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71d53e61-57bc-4d66-a029-965ea868d660" containerName="cleanup"
Oct 30 01:15:39.347073 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:39.347063    5002 cpu_manager.go:414] "containerMap: removing container" podUID="8dcf8782-f46c-4c0f-9133-793017d66a37" containerName="cleanup"
Oct 30 01:15:39.347073 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:39.347068    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8dcf8782-f46c-4c0f-9133-793017d66a37" containerName="cleanup"
Oct 30 01:15:39.347240 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:39.347075    5002 cpu_manager.go:414] "containerMap: removing container" podUID="7568f0d2-0dfb-47b9-ab97-d06d3c985d90" containerName="cleanup"
Oct 30 01:15:39.347240 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:39.347080    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7568f0d2-0dfb-47b9-ab97-d06d3c985d90" containerName="cleanup"
Oct 30 01:15:39.347240 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:39.347090    5002 cpu_manager.go:414] "containerMap: removing container" podUID="6b614db7-0a42-4251-a93b-80ee2f8d3fbe" containerName="cleanup"
Oct 30 01:15:39.347240 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:39.347095    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b614db7-0a42-4251-a93b-80ee2f8d3fbe" containerName="cleanup"
Oct 30 01:15:39.347240 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:39.347103    5002 cpu_manager.go:414] "containerMap: removing container" podUID="5834beef-f5d2-4d8f-94da-aa3a86d692cf" containerName="cleanup"
Oct 30 01:15:39.347240 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:39.347108    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5834beef-f5d2-4d8f-94da-aa3a86d692cf" containerName="cleanup"
Oct 30 01:15:39.347240 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:39.347114    5002 cpu_manager.go:414] "containerMap: removing container" podUID="92776c04-7f5b-457a-be15-a5ce82a98a68" containerName="cleanup"
Oct 30 01:15:39.347240 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:39.347118    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92776c04-7f5b-457a-be15-a5ce82a98a68" containerName="cleanup"
Oct 30 01:15:39.347240 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:39.347126    5002 cpu_manager.go:414] "containerMap: removing container" podUID="71d53e61-57bc-4d66-a029-965ea868d660" containerName="cleanup"
Oct 30 01:15:39.347240 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:39.347130    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71d53e61-57bc-4d66-a029-965ea868d660" containerName="cleanup"
Oct 30 01:15:39.347240 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:39.347137    5002 cpu_manager.go:414] "containerMap: removing container" podUID="82b3d01a-515b-43fe-b05d-ae6d90c678a3" containerName="cleanup"
Oct 30 01:15:39.347240 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:39.347141    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82b3d01a-515b-43fe-b05d-ae6d90c678a3" containerName="cleanup"
Oct 30 01:15:39.347240 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:39.347150    5002 cpu_manager.go:414] "containerMap: removing container" podUID="7568f0d2-0dfb-47b9-ab97-d06d3c985d90" containerName="cleanup"
Oct 30 01:15:39.347240 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:39.347156    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7568f0d2-0dfb-47b9-ab97-d06d3c985d90" containerName="cleanup"
Oct 30 01:15:39.347240 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:39.347163    5002 cpu_manager.go:414] "containerMap: removing container" podUID="55f94866-dc95-4ee1-b634-a3c434226a03" containerName="cleanup"
Oct 30 01:15:39.347240 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:39.347168    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55f94866-dc95-4ee1-b634-a3c434226a03" containerName="cleanup"
Oct 30 01:15:39.347240 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:39.347174    5002 cpu_manager.go:414] "containerMap: removing container" podUID="248fa0d9-fe68-4ba7-82c3-d7911c9b8070" containerName="cleanup"
Oct 30 01:15:39.347240 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:39.347178    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="248fa0d9-fe68-4ba7-82c3-d7911c9b8070" containerName="cleanup"
Oct 30 01:15:39.347240 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:39.347184    5002 cpu_manager.go:414] "containerMap: removing container" podUID="71d53e61-57bc-4d66-a029-965ea868d660" containerName="cleanup"
Oct 30 01:15:39.347240 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:39.347188    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71d53e61-57bc-4d66-a029-965ea868d660" containerName="cleanup"
Oct 30 01:15:39.347240 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:39.347196    5002 cpu_manager.go:414] "containerMap: removing container" podUID="6b614db7-0a42-4251-a93b-80ee2f8d3fbe" containerName="cleanup"
Oct 30 01:15:39.347240 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:39.347201    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b614db7-0a42-4251-a93b-80ee2f8d3fbe" containerName="cleanup"
Oct 30 01:15:39.347240 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:39.347211    5002 cpu_manager.go:414] "containerMap: removing container" podUID="7568f0d2-0dfb-47b9-ab97-d06d3c985d90" containerName="cleanup"
Oct 30 01:15:39.347240 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:39.347217    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7568f0d2-0dfb-47b9-ab97-d06d3c985d90" containerName="cleanup"
Oct 30 01:15:39.347240 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:39.347225    5002 cpu_manager.go:414] "containerMap: removing container" podUID="466ca974-a672-4f29-81c0-5e90b552b583" containerName="payload-pre-processing"
Oct 30 01:15:39.347240 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:39.347232    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="466ca974-a672-4f29-81c0-5e90b552b583" containerName="payload-pre-processing"
Oct 30 01:15:39.347240 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:39.347241    5002 cpu_manager.go:414] "containerMap: removing container" podUID="34ced72e-384a-4e43-90b7-2f907956c2c1" containerName="cleanup"
Oct 30 01:15:39.347240 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:39.347246    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34ced72e-384a-4e43-90b7-2f907956c2c1" containerName="cleanup"
Oct 30 01:15:39.347240 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:39.347254    5002 cpu_manager.go:414] "containerMap: removing container" podUID="34ced72e-384a-4e43-90b7-2f907956c2c1" containerName="cleanup"
Oct 30 01:15:39.348126 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:39.347260    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34ced72e-384a-4e43-90b7-2f907956c2c1" containerName="cleanup"
Oct 30 01:15:39.348126 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:39.347268    5002 cpu_manager.go:414] "containerMap: removing container" podUID="92776c04-7f5b-457a-be15-a5ce82a98a68" containerName="cleanup"
Oct 30 01:15:39.348126 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:39.347273    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92776c04-7f5b-457a-be15-a5ce82a98a68" containerName="cleanup"
Oct 30 01:15:39.348126 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:39.347279    5002 cpu_manager.go:414] "containerMap: removing container" podUID="aba84a8f-bfa1-4a2e-9569-8194bed6ec61" containerName="cleanup"
Oct 30 01:15:39.348126 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:39.347283    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aba84a8f-bfa1-4a2e-9569-8194bed6ec61" containerName="cleanup"
Oct 30 01:15:39.348126 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:39.347289    5002 cpu_manager.go:414] "containerMap: removing container" podUID="fd232d4c-4d2b-4137-bf83-b3e140fba4a6" containerName="cleanup"
Oct 30 01:15:39.348126 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:39.347294    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd232d4c-4d2b-4137-bf83-b3e140fba4a6" containerName="cleanup"
Oct 30 01:15:39.348126 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:39.347300    5002 cpu_manager.go:414] "containerMap: removing container" podUID="fd232d4c-4d2b-4137-bf83-b3e140fba4a6" containerName="cleanup"
Oct 30 01:15:39.348126 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:39.347305    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd232d4c-4d2b-4137-bf83-b3e140fba4a6" containerName="cleanup"
Oct 30 01:15:39.348126 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:39.347310    5002 cpu_manager.go:414] "containerMap: removing container" podUID="07d6239e-5b00-4c4a-b8dd-d935c4c44ed8" containerName="cleanup"
Oct 30 01:15:39.348126 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:39.347315    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07d6239e-5b00-4c4a-b8dd-d935c4c44ed8" containerName="cleanup"
Oct 30 01:15:39.348126 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:39.347325    5002 cpu_manager.go:414] "containerMap: removing container" podUID="248fa0d9-fe68-4ba7-82c3-d7911c9b8070" containerName="cleanup"
Oct 30 01:15:39.348126 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:39.347329    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="248fa0d9-fe68-4ba7-82c3-d7911c9b8070" containerName="cleanup"
Oct 30 01:15:39.348126 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:39.347336    5002 cpu_manager.go:414] "containerMap: removing container" podUID="663c6827-07ce-4de7-8d82-033f8b60c129" containerName="cleanup"
Oct 30 01:15:39.348126 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:39.347340    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="663c6827-07ce-4de7-8d82-033f8b60c129" containerName="cleanup"
Oct 30 01:15:39.348126 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:39.347348    5002 cpu_manager.go:414] "containerMap: removing container" podUID="fd232d4c-4d2b-4137-bf83-b3e140fba4a6" containerName="cleanup"
Oct 30 01:15:39.348126 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:39.347352    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd232d4c-4d2b-4137-bf83-b3e140fba4a6" containerName="cleanup"
Oct 30 01:15:39.348126 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:39.347360    5002 cpu_manager.go:414] "containerMap: removing container" podUID="248fa0d9-fe68-4ba7-82c3-d7911c9b8070" containerName="cleanup"
Oct 30 01:15:39.348126 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:39.347364    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="248fa0d9-fe68-4ba7-82c3-d7911c9b8070" containerName="cleanup"
Oct 30 01:15:39.348126 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:39.347371    5002 cpu_manager.go:414] "containerMap: removing container" podUID="07d6239e-5b00-4c4a-b8dd-d935c4c44ed8" containerName="cleanup"
Oct 30 01:15:39.348126 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:39.347375    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07d6239e-5b00-4c4a-b8dd-d935c4c44ed8" containerName="cleanup"
Oct 30 01:15:39.348126 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:39.347397    5002 cpu_manager.go:414] "containerMap: removing container" podUID="5d965e6c-4e56-4dff-918d-c93b9832b72f" containerName="cleanup"
Oct 30 01:15:39.348126 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:39.347404    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d965e6c-4e56-4dff-918d-c93b9832b72f" containerName="cleanup"
Oct 30 01:15:39.348126 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:39.347411    5002 cpu_manager.go:414] "containerMap: removing container" podUID="34ced72e-384a-4e43-90b7-2f907956c2c1" containerName="cleanup"
Oct 30 01:15:39.348126 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:39.347415    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34ced72e-384a-4e43-90b7-2f907956c2c1" containerName="cleanup"
Oct 30 01:15:39.348126 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:39.347423    5002 cpu_manager.go:414] "containerMap: removing container" podUID="663c6827-07ce-4de7-8d82-033f8b60c129" containerName="cleanup"
Oct 30 01:15:39.348126 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:39.347434    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="663c6827-07ce-4de7-8d82-033f8b60c129" containerName="cleanup"
Oct 30 01:15:39.348126 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:39.347441    5002 cpu_manager.go:414] "containerMap: removing container" podUID="55f94866-dc95-4ee1-b634-a3c434226a03" containerName="cleanup"
Oct 30 01:15:39.348126 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:39.347446    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55f94866-dc95-4ee1-b634-a3c434226a03" containerName="cleanup"
Oct 30 01:15:39.348126 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:39.347452    5002 cpu_manager.go:414] "containerMap: removing container" podUID="82b3d01a-515b-43fe-b05d-ae6d90c678a3" containerName="cleanup"
Oct 30 01:15:39.348126 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:39.347457    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82b3d01a-515b-43fe-b05d-ae6d90c678a3" containerName="cleanup"
Oct 30 01:15:39.348126 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:39.347463    5002 cpu_manager.go:414] "containerMap: removing container" podUID="8dcf8782-f46c-4c0f-9133-793017d66a37" containerName="cleanup"
Oct 30 01:15:39.348126 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:39.347467    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8dcf8782-f46c-4c0f-9133-793017d66a37" containerName="cleanup"
Oct 30 01:15:39.348126 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:39.347474    5002 cpu_manager.go:414] "containerMap: removing container" podUID="5d965e6c-4e56-4dff-918d-c93b9832b72f" containerName="cleanup"
Oct 30 01:15:39.348126 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:39.347478    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d965e6c-4e56-4dff-918d-c93b9832b72f" containerName="cleanup"
Oct 30 01:15:39.348126 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:39.347485    5002 cpu_manager.go:414] "containerMap: removing container" podUID="5d965e6c-4e56-4dff-918d-c93b9832b72f" containerName="cleanup"
Oct 30 01:15:39.348126 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:39.347490    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d965e6c-4e56-4dff-918d-c93b9832b72f" containerName="cleanup"
Oct 30 01:15:39.348126 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:39.347497    5002 cpu_manager.go:414] "containerMap: removing container" podUID="8dcf8782-f46c-4c0f-9133-793017d66a37" containerName="cleanup"
Oct 30 01:15:39.348126 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:39.347502    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8dcf8782-f46c-4c0f-9133-793017d66a37" containerName="cleanup"
Oct 30 01:15:39.348126 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:39.347513    5002 cpu_manager.go:414] "containerMap: removing container" podUID="07d6239e-5b00-4c4a-b8dd-d935c4c44ed8" containerName="cleanup"
Oct 30 01:15:39.348126 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:39.347518    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07d6239e-5b00-4c4a-b8dd-d935c4c44ed8" containerName="cleanup"
Oct 30 01:15:39.348126 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:39.347525    5002 cpu_manager.go:414] "containerMap: removing container" podUID="aba84a8f-bfa1-4a2e-9569-8194bed6ec61" containerName="cleanup"
Oct 30 01:15:39.348126 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:39.347529    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aba84a8f-bfa1-4a2e-9569-8194bed6ec61" containerName="cleanup"
Oct 30 01:15:39.348126 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:39.347543    5002 cpu_manager.go:414] "containerMap: removing container" podUID="6ad27632-6ab9-4b4f-bf4f-3afa10eed73b" containerName="payload-processing"
Oct 30 01:15:39.348126 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:39.347547    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ad27632-6ab9-4b4f-bf4f-3afa10eed73b" containerName="payload-processing"
Oct 30 01:15:39.348126 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:39.347653    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="8dcf8782-f46c-4c0f-9133-793017d66a37" containerName="cleanup"
Oct 30 01:15:39.348126 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:39.347661    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="7568f0d2-0dfb-47b9-ab97-d06d3c985d90" containerName="cleanup"
Oct 30 01:15:39.348126 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:39.347667    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b614db7-0a42-4251-a93b-80ee2f8d3fbe" containerName="cleanup"
Oct 30 01:15:39.348126 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:39.347674    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="5834beef-f5d2-4d8f-94da-aa3a86d692cf" containerName="cleanup"
Oct 30 01:15:39.348126 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:39.347679    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="92776c04-7f5b-457a-be15-a5ce82a98a68" containerName="cleanup"
Oct 30 01:15:39.348126 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:39.347685    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="71d53e61-57bc-4d66-a029-965ea868d660" containerName="cleanup"
Oct 30 01:15:39.348126 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:39.347691    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="82b3d01a-515b-43fe-b05d-ae6d90c678a3" containerName="cleanup"
Oct 30 01:15:39.348126 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:39.347697    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="55f94866-dc95-4ee1-b634-a3c434226a03" containerName="cleanup"
Oct 30 01:15:39.348126 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:39.347702    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="248fa0d9-fe68-4ba7-82c3-d7911c9b8070" containerName="cleanup"
Oct 30 01:15:39.348126 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:39.347708    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="71d53e61-57bc-4d66-a029-965ea868d660" containerName="cleanup"
Oct 30 01:15:39.348126 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:39.347715    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b614db7-0a42-4251-a93b-80ee2f8d3fbe" containerName="cleanup"
Oct 30 01:15:39.348126 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:39.347721    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="7568f0d2-0dfb-47b9-ab97-d06d3c985d90" containerName="cleanup"
Oct 30 01:15:39.348126 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:39.347726    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="466ca974-a672-4f29-81c0-5e90b552b583" containerName="payload-pre-processing"
Oct 30 01:15:39.348126 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:39.347733    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="34ced72e-384a-4e43-90b7-2f907956c2c1" containerName="cleanup"
Oct 30 01:15:39.348126 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:39.347738    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="34ced72e-384a-4e43-90b7-2f907956c2c1" containerName="cleanup"
Oct 30 01:15:39.348126 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:39.347744    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="92776c04-7f5b-457a-be15-a5ce82a98a68" containerName="cleanup"
Oct 30 01:15:39.348126 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:39.347749    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="aba84a8f-bfa1-4a2e-9569-8194bed6ec61" containerName="cleanup"
Oct 30 01:15:39.348126 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:39.347754    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd232d4c-4d2b-4137-bf83-b3e140fba4a6" containerName="cleanup"
Oct 30 01:15:39.348126 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:39.347759    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd232d4c-4d2b-4137-bf83-b3e140fba4a6" containerName="cleanup"
Oct 30 01:15:39.348126 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:39.347765    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="07d6239e-5b00-4c4a-b8dd-d935c4c44ed8" containerName="cleanup"
Oct 30 01:15:39.348126 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:39.347771    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="248fa0d9-fe68-4ba7-82c3-d7911c9b8070" containerName="cleanup"
Oct 30 01:15:39.348126 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:39.347777    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="663c6827-07ce-4de7-8d82-033f8b60c129" containerName="cleanup"
Oct 30 01:15:39.348126 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:39.347782    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd232d4c-4d2b-4137-bf83-b3e140fba4a6" containerName="cleanup"
Oct 30 01:15:39.348126 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:39.347788    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="248fa0d9-fe68-4ba7-82c3-d7911c9b8070" containerName="cleanup"
Oct 30 01:15:39.348126 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:39.347794    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="07d6239e-5b00-4c4a-b8dd-d935c4c44ed8" containerName="cleanup"
Oct 30 01:15:39.348126 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:39.347798    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d965e6c-4e56-4dff-918d-c93b9832b72f" containerName="cleanup"
Oct 30 01:15:39.348126 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:39.347803    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="34ced72e-384a-4e43-90b7-2f907956c2c1" containerName="cleanup"
Oct 30 01:15:39.348126 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:39.347808    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="55f94866-dc95-4ee1-b634-a3c434226a03" containerName="cleanup"
Oct 30 01:15:39.348126 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:39.347814    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="82b3d01a-515b-43fe-b05d-ae6d90c678a3" containerName="cleanup"
Oct 30 01:15:39.348126 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:39.347819    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d965e6c-4e56-4dff-918d-c93b9832b72f" containerName="cleanup"
Oct 30 01:15:39.348126 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:39.347824    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d965e6c-4e56-4dff-918d-c93b9832b72f" containerName="cleanup"
Oct 30 01:15:39.348126 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:39.347829    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="8dcf8782-f46c-4c0f-9133-793017d66a37" containerName="cleanup"
Oct 30 01:15:39.348126 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:39.347834    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="5834beef-f5d2-4d8f-94da-aa3a86d692cf" containerName="cleanup"
Oct 30 01:15:39.348126 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:39.347839    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="aba84a8f-bfa1-4a2e-9569-8194bed6ec61" containerName="cleanup"
Oct 30 01:15:39.348126 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:39.347845    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ad27632-6ab9-4b4f-bf4f-3afa10eed73b" containerName="payload-processing"
Oct 30 01:15:39.348126 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:39.347850    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="663c6827-07ce-4de7-8d82-033f8b60c129" containerName="cleanup"
Oct 30 01:15:39.348126 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:39.347857    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="5834beef-f5d2-4d8f-94da-aa3a86d692cf" containerName="cleanup"
Oct 30 01:15:39.351231 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:39.351215    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-00e07a-d72x6"
Oct 30 01:15:39.370352 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:39.370322    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-00e07a-d72x6"]
Oct 30 01:15:39.379557 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:39.379528    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:15:39.379719 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:39.379591    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:15:39.439906 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:39.439865    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/125ceb11-1fb3-4e4c-8bf9-fb376c3e1b13-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-00e07a-d72x6\" (UID: \"125ceb11-1fb3-4e4c-8bf9-fb376c3e1b13\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-00e07a-d72x6"
Oct 30 01:15:39.440058 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:39.439926    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ddlh\" (UniqueName: \"kubernetes.io/projected/125ceb11-1fb3-4e4c-8bf9-fb376c3e1b13-kube-api-access-5ddlh\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-00e07a-d72x6\" (UID: \"125ceb11-1fb3-4e4c-8bf9-fb376c3e1b13\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-00e07a-d72x6"
Oct 30 01:15:39.440058 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:39.440028    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/125ceb11-1fb3-4e4c-8bf9-fb376c3e1b13-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-00e07a-d72x6\" (UID: \"125ceb11-1fb3-4e4c-8bf9-fb376c3e1b13\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-00e07a-d72x6"
Oct 30 01:15:39.443370 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:39.443344    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:15:39.443497 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:39.443480    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5phsh\""
Oct 30 01:15:39.541503 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:39.541464    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/125ceb11-1fb3-4e4c-8bf9-fb376c3e1b13-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-00e07a-d72x6\" (UID: \"125ceb11-1fb3-4e4c-8bf9-fb376c3e1b13\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-00e07a-d72x6"
Oct 30 01:15:39.541931 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:39.541512    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ddlh\" (UniqueName: \"kubernetes.io/projected/125ceb11-1fb3-4e4c-8bf9-fb376c3e1b13-kube-api-access-5ddlh\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-00e07a-d72x6\" (UID: \"125ceb11-1fb3-4e4c-8bf9-fb376c3e1b13\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-00e07a-d72x6"
Oct 30 01:15:39.541931 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:39.541555    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/125ceb11-1fb3-4e4c-8bf9-fb376c3e1b13-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-00e07a-d72x6\" (UID: \"125ceb11-1fb3-4e4c-8bf9-fb376c3e1b13\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-00e07a-d72x6"
Oct 30 01:15:39.542368 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:39.542343    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/125ceb11-1fb3-4e4c-8bf9-fb376c3e1b13-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-00e07a-d72x6\" (UID: \"125ceb11-1fb3-4e4c-8bf9-fb376c3e1b13\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-00e07a-d72x6"
Oct 30 01:15:39.543967 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:39.543948    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/125ceb11-1fb3-4e4c-8bf9-fb376c3e1b13-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-00e07a-d72x6\" (UID: \"125ceb11-1fb3-4e4c-8bf9-fb376c3e1b13\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-00e07a-d72x6"
Oct 30 01:15:39.566615 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:39.566585    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ddlh\" (UniqueName: \"kubernetes.io/projected/125ceb11-1fb3-4e4c-8bf9-fb376c3e1b13-kube-api-access-5ddlh\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-00e07a-d72x6\" (UID: \"125ceb11-1fb3-4e4c-8bf9-fb376c3e1b13\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-00e07a-d72x6"
Oct 30 01:15:39.664084 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:39.663999    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-00e07a-d72x6"
Oct 30 01:15:39.971922 ip-10-0-128-7 kubenswrapper[5002]: W1030 01:15:39.971891    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod125ceb11_1fb3_4e4c_8bf9_fb376c3e1b13.slice/crio-16017cc7d1673fc6fe4c5f6e68c6664cc114b8041107d0e20291c18b278f67d3 WatchSource:0}: Error finding container 16017cc7d1673fc6fe4c5f6e68c6664cc114b8041107d0e20291c18b278f67d3: Status 404 returned error can't find the container with id 16017cc7d1673fc6fe4c5f6e68c6664cc114b8041107d0e20291c18b278f67d3
Oct 30 01:15:39.974444 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:39.974396    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-00e07a-d72x6"]
Oct 30 01:15:40.294419 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:40.294351    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-00e07a-d72x6" event={"ID":"125ceb11-1fb3-4e4c-8bf9-fb376c3e1b13","Type":"ContainerStarted","Data":"5ae3e9b5c539a28285ac7448bf74ad6b8fd23c1461d36884d436a32b87fe5f58"}
Oct 30 01:15:40.294591 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:40.294454    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-00e07a-d72x6" event={"ID":"125ceb11-1fb3-4e4c-8bf9-fb376c3e1b13","Type":"ContainerStarted","Data":"16017cc7d1673fc6fe4c5f6e68c6664cc114b8041107d0e20291c18b278f67d3"}
Oct 30 01:15:40.340884 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:40.340819    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-00e07a-d72x6" podStartSLOduration=1.340799685 podStartE2EDuration="1.340799685s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:15:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:15:40.338889532 +0000 UTC m=+2635.955322414" watchObservedRunningTime="2026-10-30 01:15:40.340799685 +0000 UTC m=+2635.957232565"
Oct 30 01:15:41.305740 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:41.305698    5002 generic.go:356] "Generic (PLEG): container finished" podID="125ceb11-1fb3-4e4c-8bf9-fb376c3e1b13" containerID="5ae3e9b5c539a28285ac7448bf74ad6b8fd23c1461d36884d436a32b87fe5f58" exitCode=0
Oct 30 01:15:41.306211 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:41.305750    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-00e07a-d72x6" event={"ID":"125ceb11-1fb3-4e4c-8bf9-fb376c3e1b13","Type":"ContainerDied","Data":"5ae3e9b5c539a28285ac7448bf74ad6b8fd23c1461d36884d436a32b87fe5f58"}
Oct 30 01:15:42.508954 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:42.508917    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-00e07a-d72x6"
Oct 30 01:15:42.587700 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:42.587602    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/125ceb11-1fb3-4e4c-8bf9-fb376c3e1b13-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/125ceb11-1fb3-4e4c-8bf9-fb376c3e1b13-maas-api-xks-ca\") pod \"125ceb11-1fb3-4e4c-8bf9-fb376c3e1b13\" (UID: \"125ceb11-1fb3-4e4c-8bf9-fb376c3e1b13\") "
Oct 30 01:15:42.587887 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:42.587751    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/125ceb11-1fb3-4e4c-8bf9-fb376c3e1b13-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/125ceb11-1fb3-4e4c-8bf9-fb376c3e1b13-maas-api-openshift-ca\") pod \"125ceb11-1fb3-4e4c-8bf9-fb376c3e1b13\" (UID: \"125ceb11-1fb3-4e4c-8bf9-fb376c3e1b13\") "
Oct 30 01:15:42.587887 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:42.587774    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/125ceb11-1fb3-4e4c-8bf9-fb376c3e1b13-kube-api-access-5ddlh\" (UniqueName: \"kubernetes.io/projected/125ceb11-1fb3-4e4c-8bf9-fb376c3e1b13-kube-api-access-5ddlh\") pod \"125ceb11-1fb3-4e4c-8bf9-fb376c3e1b13\" (UID: \"125ceb11-1fb3-4e4c-8bf9-fb376c3e1b13\") "
Oct 30 01:15:42.588702 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:42.588673    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/125ceb11-1fb3-4e4c-8bf9-fb376c3e1b13-maas-api-openshift-ca" pod "125ceb11-1fb3-4e4c-8bf9-fb376c3e1b13" (UID: "125ceb11-1fb3-4e4c-8bf9-fb376c3e1b13"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:15:42.589268 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:42.589240    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/125ceb11-1fb3-4e4c-8bf9-fb376c3e1b13-maas-api-openshift-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:15:42.590648 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:42.590618    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/125ceb11-1fb3-4e4c-8bf9-fb376c3e1b13-maas-api-xks-ca" pod "125ceb11-1fb3-4e4c-8bf9-fb376c3e1b13" (UID: "125ceb11-1fb3-4e4c-8bf9-fb376c3e1b13"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:15:42.590809 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:42.590788    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/125ceb11-1fb3-4e4c-8bf9-fb376c3e1b13-kube-api-access-5ddlh" pod "125ceb11-1fb3-4e4c-8bf9-fb376c3e1b13" (UID: "125ceb11-1fb3-4e4c-8bf9-fb376c3e1b13"). InnerVolumeSpecName "kube-api-access-5ddlh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:15:42.704564 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:42.704528    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/125ceb11-1fb3-4e4c-8bf9-fb376c3e1b13-maas-api-xks-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:15:42.704797 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:42.704772    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5ddlh\" (UniqueName: \"kubernetes.io/projected/125ceb11-1fb3-4e4c-8bf9-fb376c3e1b13-kube-api-access-5ddlh\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:15:42.868595 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:15:42.868500    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbjdq" containerName="extractor"
Oct 30 01:15:43.288163 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:43.288124    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6d44c2-lkp8m"]
Oct 30 01:15:43.323394 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:43.322724    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6d44c2-lkp8m"]
Oct 30 01:15:43.333121 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:43.333077    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-00e07a-d72x6" event={"ID":"125ceb11-1fb3-4e4c-8bf9-fb376c3e1b13","Type":"ContainerDied","Data":"16017cc7d1673fc6fe4c5f6e68c6664cc114b8041107d0e20291c18b278f67d3"}
Oct 30 01:15:43.333315 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:43.333142    5002 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="16017cc7d1673fc6fe4c5f6e68c6664cc114b8041107d0e20291c18b278f67d3"
Oct 30 01:15:43.333315 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:43.333206    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-00e07a-d72x6"
Oct 30 01:15:43.748648 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:43.748616    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-11f521-openshift-default-96689b67c-p9ptg"
Oct 30 01:15:43.832142 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:43.832048    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/412d8cc8-d5e9-4d37-9021-baeda679711a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/412d8cc8-d5e9-4d37-9021-baeda679711a-istio-podinfo\") pod \"412d8cc8-d5e9-4d37-9021-baeda679711a\" (UID: \"412d8cc8-d5e9-4d37-9021-baeda679711a\") "
Oct 30 01:15:43.832142 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:43.832109    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/412d8cc8-d5e9-4d37-9021-baeda679711a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/412d8cc8-d5e9-4d37-9021-baeda679711a-workload-certs\") pod \"412d8cc8-d5e9-4d37-9021-baeda679711a\" (UID: \"412d8cc8-d5e9-4d37-9021-baeda679711a\") "
Oct 30 01:15:43.832438 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:43.832150    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/412d8cc8-d5e9-4d37-9021-baeda679711a-kube-api-access-hfmfx\" (UniqueName: \"kubernetes.io/projected/412d8cc8-d5e9-4d37-9021-baeda679711a-kube-api-access-hfmfx\") pod \"412d8cc8-d5e9-4d37-9021-baeda679711a\" (UID: \"412d8cc8-d5e9-4d37-9021-baeda679711a\") "
Oct 30 01:15:43.832438 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:43.832232    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/412d8cc8-d5e9-4d37-9021-baeda679711a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/412d8cc8-d5e9-4d37-9021-baeda679711a-istio-envoy\") pod \"412d8cc8-d5e9-4d37-9021-baeda679711a\" (UID: \"412d8cc8-d5e9-4d37-9021-baeda679711a\") "
Oct 30 01:15:43.832438 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:43.832298    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/412d8cc8-d5e9-4d37-9021-baeda679711a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/412d8cc8-d5e9-4d37-9021-baeda679711a-workload-socket\") pod \"412d8cc8-d5e9-4d37-9021-baeda679711a\" (UID: \"412d8cc8-d5e9-4d37-9021-baeda679711a\") "
Oct 30 01:15:43.832438 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:43.832347    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/412d8cc8-d5e9-4d37-9021-baeda679711a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/412d8cc8-d5e9-4d37-9021-baeda679711a-istio-data\") pod \"412d8cc8-d5e9-4d37-9021-baeda679711a\" (UID: \"412d8cc8-d5e9-4d37-9021-baeda679711a\") "
Oct 30 01:15:43.832438 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:43.832432    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/412d8cc8-d5e9-4d37-9021-baeda679711a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/412d8cc8-d5e9-4d37-9021-baeda679711a-istiod-ca-cert\") pod \"412d8cc8-d5e9-4d37-9021-baeda679711a\" (UID: \"412d8cc8-d5e9-4d37-9021-baeda679711a\") "
Oct 30 01:15:43.832763 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:43.832484    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/412d8cc8-d5e9-4d37-9021-baeda679711a-istio-token\" (UniqueName: \"kubernetes.io/projected/412d8cc8-d5e9-4d37-9021-baeda679711a-istio-token\") pod \"412d8cc8-d5e9-4d37-9021-baeda679711a\" (UID: \"412d8cc8-d5e9-4d37-9021-baeda679711a\") "
Oct 30 01:15:43.832763 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:43.832487    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/412d8cc8-d5e9-4d37-9021-baeda679711a-workload-certs" pod "412d8cc8-d5e9-4d37-9021-baeda679711a" (UID: "412d8cc8-d5e9-4d37-9021-baeda679711a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:15:43.832763 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:43.832606    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/412d8cc8-d5e9-4d37-9021-baeda679711a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/412d8cc8-d5e9-4d37-9021-baeda679711a-credential-socket\") pod \"412d8cc8-d5e9-4d37-9021-baeda679711a\" (UID: \"412d8cc8-d5e9-4d37-9021-baeda679711a\") "
Oct 30 01:15:43.832763 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:43.832711    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/412d8cc8-d5e9-4d37-9021-baeda679711a-workload-socket" pod "412d8cc8-d5e9-4d37-9021-baeda679711a" (UID: "412d8cc8-d5e9-4d37-9021-baeda679711a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:15:43.833290 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:43.833260    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/412d8cc8-d5e9-4d37-9021-baeda679711a-credential-socket" pod "412d8cc8-d5e9-4d37-9021-baeda679711a" (UID: "412d8cc8-d5e9-4d37-9021-baeda679711a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:15:43.833436 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:43.833300    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/412d8cc8-d5e9-4d37-9021-baeda679711a-istiod-ca-cert" pod "412d8cc8-d5e9-4d37-9021-baeda679711a" (UID: "412d8cc8-d5e9-4d37-9021-baeda679711a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:15:43.833436 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:43.833357    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/412d8cc8-d5e9-4d37-9021-baeda679711a-istio-data" pod "412d8cc8-d5e9-4d37-9021-baeda679711a" (UID: "412d8cc8-d5e9-4d37-9021-baeda679711a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:15:43.833793 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:43.833767    5002 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/412d8cc8-d5e9-4d37-9021-baeda679711a-workload-socket\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:15:43.833874 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:43.833798    5002 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/412d8cc8-d5e9-4d37-9021-baeda679711a-istio-data\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:15:43.833874 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:43.833812    5002 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/412d8cc8-d5e9-4d37-9021-baeda679711a-istiod-ca-cert\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:15:43.833874 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:43.833826    5002 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/412d8cc8-d5e9-4d37-9021-baeda679711a-credential-socket\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:15:43.833874 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:43.833838    5002 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/412d8cc8-d5e9-4d37-9021-baeda679711a-workload-certs\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:15:43.837064 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:43.837027    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/412d8cc8-d5e9-4d37-9021-baeda679711a-istio-podinfo" pod "412d8cc8-d5e9-4d37-9021-baeda679711a" (UID: "412d8cc8-d5e9-4d37-9021-baeda679711a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:15:43.837210 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:43.837185    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/412d8cc8-d5e9-4d37-9021-baeda679711a-istio-token" pod "412d8cc8-d5e9-4d37-9021-baeda679711a" (UID: "412d8cc8-d5e9-4d37-9021-baeda679711a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:15:43.837633 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:43.837596    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/412d8cc8-d5e9-4d37-9021-baeda679711a-istio-envoy" pod "412d8cc8-d5e9-4d37-9021-baeda679711a" (UID: "412d8cc8-d5e9-4d37-9021-baeda679711a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:15:43.838132 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:43.838102    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/412d8cc8-d5e9-4d37-9021-baeda679711a-kube-api-access-hfmfx" pod "412d8cc8-d5e9-4d37-9021-baeda679711a" (UID: "412d8cc8-d5e9-4d37-9021-baeda679711a"). InnerVolumeSpecName "kube-api-access-hfmfx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:15:43.935090 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:43.935056    5002 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/412d8cc8-d5e9-4d37-9021-baeda679711a-istio-podinfo\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:15:43.935090 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:43.935089    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hfmfx\" (UniqueName: \"kubernetes.io/projected/412d8cc8-d5e9-4d37-9021-baeda679711a-kube-api-access-hfmfx\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:15:43.935269 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:43.935099    5002 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/412d8cc8-d5e9-4d37-9021-baeda679711a-istio-envoy\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:15:43.935269 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:43.935109    5002 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/412d8cc8-d5e9-4d37-9021-baeda679711a-istio-token\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:15:44.342891 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:44.342858    5002 generic.go:356] "Generic (PLEG): container finished" podID="412d8cc8-d5e9-4d37-9021-baeda679711a" containerID="de66c75782cb4c00455584cea03af158133e50b7e124f6f29231156fb0cc75a9" exitCode=0
Oct 30 01:15:44.343063 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:44.342910    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-11f521-openshift-default-96689b67c-p9ptg"
Oct 30 01:15:44.343063 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:44.342920    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-11f521-openshift-default-96689b67c-p9ptg" event={"ID":"412d8cc8-d5e9-4d37-9021-baeda679711a","Type":"ContainerDied","Data":"de66c75782cb4c00455584cea03af158133e50b7e124f6f29231156fb0cc75a9"}
Oct 30 01:15:44.343063 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:44.342964    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-11f521-openshift-default-96689b67c-p9ptg" event={"ID":"412d8cc8-d5e9-4d37-9021-baeda679711a","Type":"ContainerDied","Data":"477cf24986e054a54c80fa25c9ad2692f18b6380aac2baf80c199f92caf53c0f"}
Oct 30 01:15:44.343063 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:44.342985    5002 scope.go:122] "RemoveContainer" containerID="de66c75782cb4c00455584cea03af158133e50b7e124f6f29231156fb0cc75a9"
Oct 30 01:15:44.355032 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:44.355011    5002 scope.go:122] "RemoveContainer" containerID="de66c75782cb4c00455584cea03af158133e50b7e124f6f29231156fb0cc75a9"
Oct 30 01:15:44.355341 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:15:44.355311    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de66c75782cb4c00455584cea03af158133e50b7e124f6f29231156fb0cc75a9\": container with ID starting with de66c75782cb4c00455584cea03af158133e50b7e124f6f29231156fb0cc75a9 not found: ID does not exist" containerID="de66c75782cb4c00455584cea03af158133e50b7e124f6f29231156fb0cc75a9"
Oct 30 01:15:44.355434 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:44.355348    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de66c75782cb4c00455584cea03af158133e50b7e124f6f29231156fb0cc75a9"} err="failed to get container status \"de66c75782cb4c00455584cea03af158133e50b7e124f6f29231156fb0cc75a9\": rpc error: code = NotFound desc = could not find container \"de66c75782cb4c00455584cea03af158133e50b7e124f6f29231156fb0cc75a9\": container with ID starting with de66c75782cb4c00455584cea03af158133e50b7e124f6f29231156fb0cc75a9 not found: ID does not exist"
Oct 30 01:15:44.400849 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:44.400818    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-11f521-openshift-default-96689b67c-p9ptg"]
Oct 30 01:15:44.417258 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:44.417222    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-11f521-openshift-default-96689b67c-p9ptg"]
Oct 30 01:15:44.498561 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:44.498520    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="412d8cc8-d5e9-4d37-9021-baeda679711a" path="/var/lib/kubelet/pods/412d8cc8-d5e9-4d37-9021-baeda679711a/volumes"
Oct 30 01:15:44.499290 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:44.499265    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59b4b452-11f0-4885-8b7b-d7a249340941" path="/var/lib/kubelet/pods/59b4b452-11f0-4885-8b7b-d7a249340941/volumes"
Oct 30 01:15:48.789296 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:48.789250    5002 scope.go:122] "RemoveContainer" containerID="3b5761da010b184e6034f5db5338864a803f372d99a7e16a4d5bd3c498358def"
Oct 30 01:15:48.816772 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:48.816744    5002 scope.go:122] "RemoveContainer" containerID="fe4a875ed48a56d8c739283d5856f58dbb4235f6424e3315bfd8cfcd4ccb23aa"
Oct 30 01:15:50.915769 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:50.879227    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-6d44c2-openshift-default-69fb7659f-snvvd"]
Oct 30 01:15:50.915769 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:50.879511    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-6d44c2-openshift-default-69fb7659f-snvvd" podUID="f3f9e906-5594-4cdf-a620-efc9cbc28a44" containerName="istio-proxy" containerID="cri-o://61d2d21bb52878d7b3ccbd7517bca54723c0ec9066044bf1b77313d20d46cd63" gracePeriod=30
Oct 30 01:15:51.017404 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:51.014977    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-48df96-kpdrg"]
Oct 30 01:15:51.017404 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:51.016031    5002 cpu_manager.go:414] "containerMap: removing container" podUID="92776c04-7f5b-457a-be15-a5ce82a98a68" containerName="cleanup"
Oct 30 01:15:51.017404 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:51.016052    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92776c04-7f5b-457a-be15-a5ce82a98a68" containerName="cleanup"
Oct 30 01:15:51.017404 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:51.016100    5002 cpu_manager.go:414] "containerMap: removing container" podUID="412d8cc8-d5e9-4d37-9021-baeda679711a" containerName="istio-proxy"
Oct 30 01:15:51.017404 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:51.016109    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="412d8cc8-d5e9-4d37-9021-baeda679711a" containerName="istio-proxy"
Oct 30 01:15:51.017404 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:51.016123    5002 cpu_manager.go:414] "containerMap: removing container" podUID="125ceb11-1fb3-4e4c-8bf9-fb376c3e1b13" containerName="revoke-keys"
Oct 30 01:15:51.017404 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:51.016131    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="125ceb11-1fb3-4e4c-8bf9-fb376c3e1b13" containerName="revoke-keys"
Oct 30 01:15:51.017404 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:51.016150    5002 cpu_manager.go:414] "containerMap: removing container" podUID="663c6827-07ce-4de7-8d82-033f8b60c129" containerName="cleanup"
Oct 30 01:15:51.017404 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:51.016158    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="663c6827-07ce-4de7-8d82-033f8b60c129" containerName="cleanup"
Oct 30 01:15:51.017404 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:51.016169    5002 cpu_manager.go:414] "containerMap: removing container" podUID="55f94866-dc95-4ee1-b634-a3c434226a03" containerName="cleanup"
Oct 30 01:15:51.017404 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:51.016177    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55f94866-dc95-4ee1-b634-a3c434226a03" containerName="cleanup"
Oct 30 01:15:51.017404 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:51.016202    5002 cpu_manager.go:414] "containerMap: removing container" podUID="aba84a8f-bfa1-4a2e-9569-8194bed6ec61" containerName="cleanup"
Oct 30 01:15:51.017404 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:51.016210    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aba84a8f-bfa1-4a2e-9569-8194bed6ec61" containerName="cleanup"
Oct 30 01:15:51.017404 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:51.016246    5002 cpu_manager.go:414] "containerMap: removing container" podUID="5834beef-f5d2-4d8f-94da-aa3a86d692cf" containerName="cleanup"
Oct 30 01:15:51.017404 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:51.016254    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5834beef-f5d2-4d8f-94da-aa3a86d692cf" containerName="cleanup"
Oct 30 01:15:51.017404 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:51.016271    5002 cpu_manager.go:414] "containerMap: removing container" podUID="82b3d01a-515b-43fe-b05d-ae6d90c678a3" containerName="cleanup"
Oct 30 01:15:51.017404 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:51.016280    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82b3d01a-515b-43fe-b05d-ae6d90c678a3" containerName="cleanup"
Oct 30 01:15:51.017404 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:51.016299    5002 cpu_manager.go:414] "containerMap: removing container" podUID="6b614db7-0a42-4251-a93b-80ee2f8d3fbe" containerName="cleanup"
Oct 30 01:15:51.017404 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:51.016307    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b614db7-0a42-4251-a93b-80ee2f8d3fbe" containerName="cleanup"
Oct 30 01:15:51.017404 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:51.016471    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="412d8cc8-d5e9-4d37-9021-baeda679711a" containerName="istio-proxy"
Oct 30 01:15:51.017404 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:51.016489    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="07d6239e-5b00-4c4a-b8dd-d935c4c44ed8" containerName="cleanup"
Oct 30 01:15:51.017404 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:51.016500    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="125ceb11-1fb3-4e4c-8bf9-fb376c3e1b13" containerName="revoke-keys"
Oct 30 01:15:51.017404 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:51.016511    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="55f94866-dc95-4ee1-b634-a3c434226a03" containerName="cleanup"
Oct 30 01:15:51.017404 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:51.016524    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="aba84a8f-bfa1-4a2e-9569-8194bed6ec61" containerName="cleanup"
Oct 30 01:15:51.017404 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:51.016539    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="8dcf8782-f46c-4c0f-9133-793017d66a37" containerName="cleanup"
Oct 30 01:15:51.017404 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:51.016550    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="82b3d01a-515b-43fe-b05d-ae6d90c678a3" containerName="cleanup"
Oct 30 01:15:51.017404 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:51.016561    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="71d53e61-57bc-4d66-a029-965ea868d660" containerName="cleanup"
Oct 30 01:15:51.017404 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:51.016570    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b614db7-0a42-4251-a93b-80ee2f8d3fbe" containerName="cleanup"
Oct 30 01:15:51.017404 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:51.016577    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="7568f0d2-0dfb-47b9-ab97-d06d3c985d90" containerName="cleanup"
Oct 30 01:15:51.017404 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:51.016586    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="92776c04-7f5b-457a-be15-a5ce82a98a68" containerName="cleanup"
Oct 30 01:15:51.017404 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:51.016598    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="663c6827-07ce-4de7-8d82-033f8b60c129" containerName="cleanup"
Oct 30 01:15:51.026751 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:51.026702    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-48df96-kpdrg"
Oct 30 01:15:51.046359 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:51.046315    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-48df96-kpdrg"]
Oct 30 01:15:51.046899 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:51.046866    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:15:51.047022 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:51.046959    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:15:51.047335 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:51.047312    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5phsh\""
Oct 30 01:15:51.047450 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:51.047411    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:15:51.059843 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:51.057757    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlhjm\" (UniqueName: \"kubernetes.io/projected/f2fc0346-89e4-4049-b6c5-6fd11e53f02b-kube-api-access-jlhjm\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-48df96-kpdrg\" (UID: \"f2fc0346-89e4-4049-b6c5-6fd11e53f02b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-48df96-kpdrg"
Oct 30 01:15:51.059843 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:51.057886    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2fc0346-89e4-4049-b6c5-6fd11e53f02b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-48df96-kpdrg\" (UID: \"f2fc0346-89e4-4049-b6c5-6fd11e53f02b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-48df96-kpdrg"
Oct 30 01:15:51.059843 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:51.057927    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2fc0346-89e4-4049-b6c5-6fd11e53f02b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-48df96-kpdrg\" (UID: \"f2fc0346-89e4-4049-b6c5-6fd11e53f02b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-48df96-kpdrg"
Oct 30 01:15:51.185329 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:51.185281    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlhjm\" (UniqueName: \"kubernetes.io/projected/f2fc0346-89e4-4049-b6c5-6fd11e53f02b-kube-api-access-jlhjm\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-48df96-kpdrg\" (UID: \"f2fc0346-89e4-4049-b6c5-6fd11e53f02b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-48df96-kpdrg"
Oct 30 01:15:51.185534 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:51.185443    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2fc0346-89e4-4049-b6c5-6fd11e53f02b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-48df96-kpdrg\" (UID: \"f2fc0346-89e4-4049-b6c5-6fd11e53f02b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-48df96-kpdrg"
Oct 30 01:15:51.185534 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:51.185489    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2fc0346-89e4-4049-b6c5-6fd11e53f02b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-48df96-kpdrg\" (UID: \"f2fc0346-89e4-4049-b6c5-6fd11e53f02b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-48df96-kpdrg"
Oct 30 01:15:51.233403 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:51.222288    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2fc0346-89e4-4049-b6c5-6fd11e53f02b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-48df96-kpdrg\" (UID: \"f2fc0346-89e4-4049-b6c5-6fd11e53f02b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-48df96-kpdrg"
Oct 30 01:15:51.233403 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:51.222735    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2fc0346-89e4-4049-b6c5-6fd11e53f02b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-48df96-kpdrg\" (UID: \"f2fc0346-89e4-4049-b6c5-6fd11e53f02b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-48df96-kpdrg"
Oct 30 01:15:51.453489 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:51.449143    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlhjm\" (UniqueName: \"kubernetes.io/projected/f2fc0346-89e4-4049-b6c5-6fd11e53f02b-kube-api-access-jlhjm\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-48df96-kpdrg\" (UID: \"f2fc0346-89e4-4049-b6c5-6fd11e53f02b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-48df96-kpdrg"
Oct 30 01:15:51.509401 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:51.504785    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-48df96-kpdrg"
Oct 30 01:15:51.679282 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:51.679221    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-00e07a-d72x6"]
Oct 30 01:15:51.685691 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:51.685653    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-00e07a-d72x6"]
Oct 30 01:15:51.834783 ip-10-0-128-7 kubenswrapper[5002]: W1030 01:15:51.834747    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2fc0346_89e4_4049_b6c5_6fd11e53f02b.slice/crio-e986730d63da8033b41dfa0d72e9870346c8c6954503e6cf449cc3aba28fa2b3 WatchSource:0}: Error finding container e986730d63da8033b41dfa0d72e9870346c8c6954503e6cf449cc3aba28fa2b3: Status 404 returned error can't find the container with id e986730d63da8033b41dfa0d72e9870346c8c6954503e6cf449cc3aba28fa2b3
Oct 30 01:15:51.836100 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:51.836073    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-48df96-kpdrg"]
Oct 30 01:15:52.496718 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:52.496680    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="125ceb11-1fb3-4e4c-8bf9-fb376c3e1b13" path="/var/lib/kubelet/pods/125ceb11-1fb3-4e4c-8bf9-fb376c3e1b13/volumes"
Oct 30 01:15:52.514509 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:52.514469    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-48df96-kpdrg" event={"ID":"f2fc0346-89e4-4049-b6c5-6fd11e53f02b","Type":"ContainerStarted","Data":"ed25f2124508554d400dfa6c85427fdf4aad342c3c1e9c7318f002e95fbf2408"}
Oct 30 01:15:52.514702 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:52.514540    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-48df96-kpdrg" event={"ID":"f2fc0346-89e4-4049-b6c5-6fd11e53f02b","Type":"ContainerStarted","Data":"e986730d63da8033b41dfa0d72e9870346c8c6954503e6cf449cc3aba28fa2b3"}
Oct 30 01:15:52.559705 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:52.559641    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-48df96-kpdrg" podStartSLOduration=2.559625402 podStartE2EDuration="2.559625402s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:15:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:15:52.557298402 +0000 UTC m=+2648.173731283" watchObservedRunningTime="2026-10-30 01:15:52.559625402 +0000 UTC m=+2648.176058333"
Oct 30 01:15:53.532210 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:53.532168    5002 generic.go:356] "Generic (PLEG): container finished" podID="f2fc0346-89e4-4049-b6c5-6fd11e53f02b" containerID="ed25f2124508554d400dfa6c85427fdf4aad342c3c1e9c7318f002e95fbf2408" exitCode=0
Oct 30 01:15:53.532637 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:53.532256    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-48df96-kpdrg" event={"ID":"f2fc0346-89e4-4049-b6c5-6fd11e53f02b","Type":"ContainerDied","Data":"ed25f2124508554d400dfa6c85427fdf4aad342c3c1e9c7318f002e95fbf2408"}
Oct 30 01:15:54.144137 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:54.144098    5002 patch_prober.go:29] interesting pod/e2e-ipp-b-6d44c2-openshift-default-69fb7659f-snvvd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 01:15:54.144347 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:54.144201    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-6d44c2-openshift-default-69fb7659f-snvvd" podUID="f3f9e906-5594-4cdf-a620-efc9cbc28a44" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 01:15:54.752416 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:54.752346    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-48df96-kpdrg"
Oct 30 01:15:54.825266 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:54.825156    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2fc0346-89e4-4049-b6c5-6fd11e53f02b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2fc0346-89e4-4049-b6c5-6fd11e53f02b-maas-api-openshift-ca\") pod \"f2fc0346-89e4-4049-b6c5-6fd11e53f02b\" (UID: \"f2fc0346-89e4-4049-b6c5-6fd11e53f02b\") "
Oct 30 01:15:54.825500 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:54.825317    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f2fc0346-89e4-4049-b6c5-6fd11e53f02b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2fc0346-89e4-4049-b6c5-6fd11e53f02b-maas-api-xks-ca\") pod \"f2fc0346-89e4-4049-b6c5-6fd11e53f02b\" (UID: \"f2fc0346-89e4-4049-b6c5-6fd11e53f02b\") "
Oct 30 01:15:54.825500 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:54.825451    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2fc0346-89e4-4049-b6c5-6fd11e53f02b-kube-api-access-jlhjm\" (UniqueName: \"kubernetes.io/projected/f2fc0346-89e4-4049-b6c5-6fd11e53f02b-kube-api-access-jlhjm\") pod \"f2fc0346-89e4-4049-b6c5-6fd11e53f02b\" (UID: \"f2fc0346-89e4-4049-b6c5-6fd11e53f02b\") "
Oct 30 01:15:54.825665 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:54.825600    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2fc0346-89e4-4049-b6c5-6fd11e53f02b-maas-api-openshift-ca" pod "f2fc0346-89e4-4049-b6c5-6fd11e53f02b" (UID: "f2fc0346-89e4-4049-b6c5-6fd11e53f02b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:15:54.826028 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:54.826002    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2fc0346-89e4-4049-b6c5-6fd11e53f02b-maas-api-openshift-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:15:54.827509 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:54.827484    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2fc0346-89e4-4049-b6c5-6fd11e53f02b-maas-api-xks-ca" pod "f2fc0346-89e4-4049-b6c5-6fd11e53f02b" (UID: "f2fc0346-89e4-4049-b6c5-6fd11e53f02b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:15:54.827938 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:54.827910    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2fc0346-89e4-4049-b6c5-6fd11e53f02b-kube-api-access-jlhjm" pod "f2fc0346-89e4-4049-b6c5-6fd11e53f02b" (UID: "f2fc0346-89e4-4049-b6c5-6fd11e53f02b"). InnerVolumeSpecName "kube-api-access-jlhjm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:15:54.927987 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:54.927944    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jlhjm\" (UniqueName: \"kubernetes.io/projected/f2fc0346-89e4-4049-b6c5-6fd11e53f02b-kube-api-access-jlhjm\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:15:54.927987 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:54.927988    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2fc0346-89e4-4049-b6c5-6fd11e53f02b-maas-api-xks-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:15:55.548994 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:55.548961    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-48df96-kpdrg"
Oct 30 01:15:55.549190 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:55.548990    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-48df96-kpdrg" event={"ID":"f2fc0346-89e4-4049-b6c5-6fd11e53f02b","Type":"ContainerDied","Data":"e986730d63da8033b41dfa0d72e9870346c8c6954503e6cf449cc3aba28fa2b3"}
Oct 30 01:15:55.549190 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:55.549043    5002 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e986730d63da8033b41dfa0d72e9870346c8c6954503e6cf449cc3aba28fa2b3"
Oct 30 01:15:56.208605 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:56.208578    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-6d44c2-openshift-default-69fb7659f-snvvd"
Oct 30 01:15:56.259982 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:56.259886    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3f9e906-5594-4cdf-a620-efc9cbc28a44-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3f9e906-5594-4cdf-a620-efc9cbc28a44-credential-socket\") pod \"f3f9e906-5594-4cdf-a620-efc9cbc28a44\" (UID: \"f3f9e906-5594-4cdf-a620-efc9cbc28a44\") "
Oct 30 01:15:56.259982 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:56.259934    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3f9e906-5594-4cdf-a620-efc9cbc28a44-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3f9e906-5594-4cdf-a620-efc9cbc28a44-istiod-ca-cert\") pod \"f3f9e906-5594-4cdf-a620-efc9cbc28a44\" (UID: \"f3f9e906-5594-4cdf-a620-efc9cbc28a44\") "
Oct 30 01:15:56.259982 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:56.259978    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f3f9e906-5594-4cdf-a620-efc9cbc28a44-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3f9e906-5594-4cdf-a620-efc9cbc28a44-istio-podinfo\") pod \"f3f9e906-5594-4cdf-a620-efc9cbc28a44\" (UID: \"f3f9e906-5594-4cdf-a620-efc9cbc28a44\") "
Oct 30 01:15:56.260291 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:56.260045    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3f9e906-5594-4cdf-a620-efc9cbc28a44-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3f9e906-5594-4cdf-a620-efc9cbc28a44-istio-data\") pod \"f3f9e906-5594-4cdf-a620-efc9cbc28a44\" (UID: \"f3f9e906-5594-4cdf-a620-efc9cbc28a44\") "
Oct 30 01:15:56.260291 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:56.260126    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3f9e906-5594-4cdf-a620-efc9cbc28a44-kube-api-access-qbhqz\" (UniqueName: \"kubernetes.io/projected/f3f9e906-5594-4cdf-a620-efc9cbc28a44-kube-api-access-qbhqz\") pod \"f3f9e906-5594-4cdf-a620-efc9cbc28a44\" (UID: \"f3f9e906-5594-4cdf-a620-efc9cbc28a44\") "
Oct 30 01:15:56.260291 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:56.260176    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3f9e906-5594-4cdf-a620-efc9cbc28a44-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3f9e906-5594-4cdf-a620-efc9cbc28a44-istio-envoy\") pod \"f3f9e906-5594-4cdf-a620-efc9cbc28a44\" (UID: \"f3f9e906-5594-4cdf-a620-efc9cbc28a44\") "
Oct 30 01:15:56.260291 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:56.260188    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3f9e906-5594-4cdf-a620-efc9cbc28a44-credential-socket" pod "f3f9e906-5594-4cdf-a620-efc9cbc28a44" (UID: "f3f9e906-5594-4cdf-a620-efc9cbc28a44"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:15:56.260291 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:56.260213    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3f9e906-5594-4cdf-a620-efc9cbc28a44-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3f9e906-5594-4cdf-a620-efc9cbc28a44-workload-socket\") pod \"f3f9e906-5594-4cdf-a620-efc9cbc28a44\" (UID: \"f3f9e906-5594-4cdf-a620-efc9cbc28a44\") "
Oct 30 01:15:56.260291 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:56.260272    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3f9e906-5594-4cdf-a620-efc9cbc28a44-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3f9e906-5594-4cdf-a620-efc9cbc28a44-workload-certs\") pod \"f3f9e906-5594-4cdf-a620-efc9cbc28a44\" (UID: \"f3f9e906-5594-4cdf-a620-efc9cbc28a44\") "
Oct 30 01:15:56.260669 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:56.260364    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3f9e906-5594-4cdf-a620-efc9cbc28a44-istiod-ca-cert" pod "f3f9e906-5594-4cdf-a620-efc9cbc28a44" (UID: "f3f9e906-5594-4cdf-a620-efc9cbc28a44"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:15:56.260669 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:56.260415    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3f9e906-5594-4cdf-a620-efc9cbc28a44-istio-data" pod "f3f9e906-5594-4cdf-a620-efc9cbc28a44" (UID: "f3f9e906-5594-4cdf-a620-efc9cbc28a44"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:15:56.260669 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:56.260431    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3f9e906-5594-4cdf-a620-efc9cbc28a44-istio-token\" (UniqueName: \"kubernetes.io/projected/f3f9e906-5594-4cdf-a620-efc9cbc28a44-istio-token\") pod \"f3f9e906-5594-4cdf-a620-efc9cbc28a44\" (UID: \"f3f9e906-5594-4cdf-a620-efc9cbc28a44\") "
Oct 30 01:15:56.260861 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:56.260836    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3f9e906-5594-4cdf-a620-efc9cbc28a44-workload-socket" pod "f3f9e906-5594-4cdf-a620-efc9cbc28a44" (UID: "f3f9e906-5594-4cdf-a620-efc9cbc28a44"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:15:56.260986 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:56.260966    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3f9e906-5594-4cdf-a620-efc9cbc28a44-workload-certs" pod "f3f9e906-5594-4cdf-a620-efc9cbc28a44" (UID: "f3f9e906-5594-4cdf-a620-efc9cbc28a44"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:15:56.261304 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:56.261279    5002 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3f9e906-5594-4cdf-a620-efc9cbc28a44-credential-socket\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:15:56.261304 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:56.261304    5002 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3f9e906-5594-4cdf-a620-efc9cbc28a44-istiod-ca-cert\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:15:56.261559 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:56.261318    5002 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3f9e906-5594-4cdf-a620-efc9cbc28a44-istio-data\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:15:56.261559 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:56.261330    5002 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3f9e906-5594-4cdf-a620-efc9cbc28a44-workload-socket\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:15:56.261559 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:56.261341    5002 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3f9e906-5594-4cdf-a620-efc9cbc28a44-workload-certs\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:15:56.263114 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:56.263079    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3f9e906-5594-4cdf-a620-efc9cbc28a44-istio-envoy" pod "f3f9e906-5594-4cdf-a620-efc9cbc28a44" (UID: "f3f9e906-5594-4cdf-a620-efc9cbc28a44"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:15:56.263240 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:56.263181    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f3f9e906-5594-4cdf-a620-efc9cbc28a44-istio-podinfo" pod "f3f9e906-5594-4cdf-a620-efc9cbc28a44" (UID: "f3f9e906-5594-4cdf-a620-efc9cbc28a44"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:15:56.263487 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:56.263458    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3f9e906-5594-4cdf-a620-efc9cbc28a44-kube-api-access-qbhqz" pod "f3f9e906-5594-4cdf-a620-efc9cbc28a44" (UID: "f3f9e906-5594-4cdf-a620-efc9cbc28a44"). InnerVolumeSpecName "kube-api-access-qbhqz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:15:56.263610 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:56.263495    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3f9e906-5594-4cdf-a620-efc9cbc28a44-istio-token" pod "f3f9e906-5594-4cdf-a620-efc9cbc28a44" (UID: "f3f9e906-5594-4cdf-a620-efc9cbc28a44"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:15:56.363764 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:56.363724    5002 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3f9e906-5594-4cdf-a620-efc9cbc28a44-istio-envoy\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:15:56.363764 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:56.363760    5002 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3f9e906-5594-4cdf-a620-efc9cbc28a44-istio-token\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:15:56.364026 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:56.363776    5002 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3f9e906-5594-4cdf-a620-efc9cbc28a44-istio-podinfo\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:15:56.364026 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:56.363791    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qbhqz\" (UniqueName: \"kubernetes.io/projected/f3f9e906-5594-4cdf-a620-efc9cbc28a44-kube-api-access-qbhqz\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:15:56.558798 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:56.558701    5002 generic.go:356] "Generic (PLEG): container finished" podID="f3f9e906-5594-4cdf-a620-efc9cbc28a44" containerID="61d2d21bb52878d7b3ccbd7517bca54723c0ec9066044bf1b77313d20d46cd63" exitCode=0
Oct 30 01:15:56.558798 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:56.558764    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-6d44c2-openshift-default-69fb7659f-snvvd"
Oct 30 01:15:56.558798 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:56.558788    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-6d44c2-openshift-default-69fb7659f-snvvd" event={"ID":"f3f9e906-5594-4cdf-a620-efc9cbc28a44","Type":"ContainerDied","Data":"61d2d21bb52878d7b3ccbd7517bca54723c0ec9066044bf1b77313d20d46cd63"}
Oct 30 01:15:56.559085 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:56.558843    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-6d44c2-openshift-default-69fb7659f-snvvd" event={"ID":"f3f9e906-5594-4cdf-a620-efc9cbc28a44","Type":"ContainerDied","Data":"ee9990e89d0b13463b6865740de3fc8535fd4812e1e7efc553a2dbbc90e74c19"}
Oct 30 01:15:56.559085 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:56.558867    5002 scope.go:122] "RemoveContainer" containerID="61d2d21bb52878d7b3ccbd7517bca54723c0ec9066044bf1b77313d20d46cd63"
Oct 30 01:15:56.572429 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:56.572365    5002 scope.go:122] "RemoveContainer" containerID="670c9acd81e92e790db801790362f50310923577a102c2934548080f2d5fd45e"
Oct 30 01:15:56.583224 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:56.583197    5002 scope.go:122] "RemoveContainer" containerID="61d2d21bb52878d7b3ccbd7517bca54723c0ec9066044bf1b77313d20d46cd63"
Oct 30 01:15:56.583624 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:15:56.583592    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61d2d21bb52878d7b3ccbd7517bca54723c0ec9066044bf1b77313d20d46cd63\": container with ID starting with 61d2d21bb52878d7b3ccbd7517bca54723c0ec9066044bf1b77313d20d46cd63 not found: ID does not exist" containerID="61d2d21bb52878d7b3ccbd7517bca54723c0ec9066044bf1b77313d20d46cd63"
Oct 30 01:15:56.583755 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:56.583634    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61d2d21bb52878d7b3ccbd7517bca54723c0ec9066044bf1b77313d20d46cd63"} err="failed to get container status \"61d2d21bb52878d7b3ccbd7517bca54723c0ec9066044bf1b77313d20d46cd63\": rpc error: code = NotFound desc = could not find container \"61d2d21bb52878d7b3ccbd7517bca54723c0ec9066044bf1b77313d20d46cd63\": container with ID starting with 61d2d21bb52878d7b3ccbd7517bca54723c0ec9066044bf1b77313d20d46cd63 not found: ID does not exist"
Oct 30 01:15:56.583755 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:56.583662    5002 scope.go:122] "RemoveContainer" containerID="670c9acd81e92e790db801790362f50310923577a102c2934548080f2d5fd45e"
Oct 30 01:15:56.584000 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:15:56.583978    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"670c9acd81e92e790db801790362f50310923577a102c2934548080f2d5fd45e\": container with ID starting with 670c9acd81e92e790db801790362f50310923577a102c2934548080f2d5fd45e not found: ID does not exist" containerID="670c9acd81e92e790db801790362f50310923577a102c2934548080f2d5fd45e"
Oct 30 01:15:56.584046 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:56.584007    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"670c9acd81e92e790db801790362f50310923577a102c2934548080f2d5fd45e"} err="failed to get container status \"670c9acd81e92e790db801790362f50310923577a102c2934548080f2d5fd45e\": rpc error: code = NotFound desc = could not find container \"670c9acd81e92e790db801790362f50310923577a102c2934548080f2d5fd45e\": container with ID starting with 670c9acd81e92e790db801790362f50310923577a102c2934548080f2d5fd45e not found: ID does not exist"
Oct 30 01:15:56.617816 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:56.617784    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-6d44c2-openshift-default-69fb7659f-snvvd"]
Oct 30 01:15:56.631588 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:56.631555    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-6d44c2-openshift-default-69fb7659f-snvvd"]
Oct 30 01:15:58.500326 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:58.500289    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3f9e906-5594-4cdf-a620-efc9cbc28a44" path="/var/lib/kubelet/pods/f3f9e906-5594-4cdf-a620-efc9cbc28a44/volumes"
Oct 30 01:15:59.407433 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:59.407366    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-52650f-x22xt"]
Oct 30 01:15:59.408084 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:59.408066    5002 cpu_manager.go:414] "containerMap: removing container" podUID="f3f9e906-5594-4cdf-a620-efc9cbc28a44" containerName="istio-proxy"
Oct 30 01:15:59.408128 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:59.408086    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3f9e906-5594-4cdf-a620-efc9cbc28a44" containerName="istio-proxy"
Oct 30 01:15:59.408128 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:59.408106    5002 cpu_manager.go:414] "containerMap: removing container" podUID="f3f9e906-5594-4cdf-a620-efc9cbc28a44" containerName="istio-proxy"
Oct 30 01:15:59.408128 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:59.408111    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3f9e906-5594-4cdf-a620-efc9cbc28a44" containerName="istio-proxy"
Oct 30 01:15:59.408128 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:59.408126    5002 cpu_manager.go:414] "containerMap: removing container" podUID="f2fc0346-89e4-4049-b6c5-6fd11e53f02b" containerName="revoke-keys"
Oct 30 01:15:59.408256 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:59.408132    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2fc0346-89e4-4049-b6c5-6fd11e53f02b" containerName="revoke-keys"
Oct 30 01:15:59.408256 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:59.408227    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3f9e906-5594-4cdf-a620-efc9cbc28a44" containerName="istio-proxy"
Oct 30 01:15:59.408256 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:59.408237    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2fc0346-89e4-4049-b6c5-6fd11e53f02b" containerName="revoke-keys"
Oct 30 01:15:59.413588 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:59.413568    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-52650f-x22xt"
Oct 30 01:15:59.432226 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:59.432197    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-52650f-x22xt"]
Oct 30 01:15:59.440493 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:59.440455    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:15:59.440631 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:59.440511    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:15:59.441740 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:59.441717    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:15:59.443939 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:59.443913    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5phsh\""
Oct 30 01:15:59.536448 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:59.536414    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc526934-1cca-481c-bfe8-1b9bce696916-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-52650f-x22xt\" (UID: \"cc526934-1cca-481c-bfe8-1b9bce696916\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-52650f-x22xt"
Oct 30 01:15:59.536839 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:59.536574    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gl682\" (UniqueName: \"kubernetes.io/projected/cc526934-1cca-481c-bfe8-1b9bce696916-kube-api-access-gl682\") pod \"maas-api-revoke-keys-e2e-worker-w0-52650f-x22xt\" (UID: \"cc526934-1cca-481c-bfe8-1b9bce696916\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-52650f-x22xt"
Oct 30 01:15:59.536839 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:59.536620    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc526934-1cca-481c-bfe8-1b9bce696916-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-52650f-x22xt\" (UID: \"cc526934-1cca-481c-bfe8-1b9bce696916\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-52650f-x22xt"
Oct 30 01:15:59.638545 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:59.638511    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc526934-1cca-481c-bfe8-1b9bce696916-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-52650f-x22xt\" (UID: \"cc526934-1cca-481c-bfe8-1b9bce696916\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-52650f-x22xt"
Oct 30 01:15:59.638721 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:59.638608    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gl682\" (UniqueName: \"kubernetes.io/projected/cc526934-1cca-481c-bfe8-1b9bce696916-kube-api-access-gl682\") pod \"maas-api-revoke-keys-e2e-worker-w0-52650f-x22xt\" (UID: \"cc526934-1cca-481c-bfe8-1b9bce696916\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-52650f-x22xt"
Oct 30 01:15:59.638721 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:59.638639    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc526934-1cca-481c-bfe8-1b9bce696916-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-52650f-x22xt\" (UID: \"cc526934-1cca-481c-bfe8-1b9bce696916\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-52650f-x22xt"
Oct 30 01:15:59.639179 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:59.639152    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc526934-1cca-481c-bfe8-1b9bce696916-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-52650f-x22xt\" (UID: \"cc526934-1cca-481c-bfe8-1b9bce696916\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-52650f-x22xt"
Oct 30 01:15:59.641025 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:59.640998    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc526934-1cca-481c-bfe8-1b9bce696916-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-52650f-x22xt\" (UID: \"cc526934-1cca-481c-bfe8-1b9bce696916\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-52650f-x22xt"
Oct 30 01:15:59.663914 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:59.663853    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gl682\" (UniqueName: \"kubernetes.io/projected/cc526934-1cca-481c-bfe8-1b9bce696916-kube-api-access-gl682\") pod \"maas-api-revoke-keys-e2e-worker-w0-52650f-x22xt\" (UID: \"cc526934-1cca-481c-bfe8-1b9bce696916\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-52650f-x22xt"
Oct 30 01:15:59.727840 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:15:59.727808    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-52650f-x22xt"
Oct 30 01:16:00.043760 ip-10-0-128-7 kubenswrapper[5002]: W1030 01:16:00.043730    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc526934_1cca_481c_bfe8_1b9bce696916.slice/crio-fd0e427cefd0e5ce6958d0824e93c6ab9a29974dd60b1e2bc903f084ef9e9eff WatchSource:0}: Error finding container fd0e427cefd0e5ce6958d0824e93c6ab9a29974dd60b1e2bc903f084ef9e9eff: Status 404 returned error can't find the container with id fd0e427cefd0e5ce6958d0824e93c6ab9a29974dd60b1e2bc903f084ef9e9eff
Oct 30 01:16:00.047044 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:00.047011    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-52650f-x22xt"]
Oct 30 01:16:00.600180 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:00.600141    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-52650f-x22xt" event={"ID":"cc526934-1cca-481c-bfe8-1b9bce696916","Type":"ContainerStarted","Data":"1d07332934936c02743528807ce7ca35a0c42fe2afac4e47ccb2628f6bd16524"}
Oct 30 01:16:00.600661 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:00.600199    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-52650f-x22xt" event={"ID":"cc526934-1cca-481c-bfe8-1b9bce696916","Type":"ContainerStarted","Data":"fd0e427cefd0e5ce6958d0824e93c6ab9a29974dd60b1e2bc903f084ef9e9eff"}
Oct 30 01:16:00.648997 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:00.648945    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-52650f-x22xt" podStartSLOduration=1.64892975 podStartE2EDuration="1.64892975s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:15:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:16:00.645216176 +0000 UTC m=+2656.261649056" watchObservedRunningTime="2026-10-30 01:16:00.64892975 +0000 UTC m=+2656.265362629"
Oct 30 01:16:01.609326 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:01.609289    5002 generic.go:356] "Generic (PLEG): container finished" podID="cc526934-1cca-481c-bfe8-1b9bce696916" containerID="1d07332934936c02743528807ce7ca35a0c42fe2afac4e47ccb2628f6bd16524" exitCode=0
Oct 30 01:16:01.609760 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:01.609374    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-52650f-x22xt" event={"ID":"cc526934-1cca-481c-bfe8-1b9bce696916","Type":"ContainerDied","Data":"1d07332934936c02743528807ce7ca35a0c42fe2afac4e47ccb2628f6bd16524"}
Oct 30 01:16:03.427160 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:03.408880    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-52650f-x22xt"
Oct 30 01:16:03.545586 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:03.545533    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc526934-1cca-481c-bfe8-1b9bce696916-maas-api-openshift-ca" pod "cc526934-1cca-481c-bfe8-1b9bce696916" (UID: "cc526934-1cca-481c-bfe8-1b9bce696916"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:16:03.545797 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:03.545125    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc526934-1cca-481c-bfe8-1b9bce696916-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc526934-1cca-481c-bfe8-1b9bce696916-maas-api-openshift-ca\") pod \"cc526934-1cca-481c-bfe8-1b9bce696916\" (UID: \"cc526934-1cca-481c-bfe8-1b9bce696916\") "
Oct 30 01:16:03.545919 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:03.545883    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc526934-1cca-481c-bfe8-1b9bce696916-kube-api-access-gl682\" (UniqueName: \"kubernetes.io/projected/cc526934-1cca-481c-bfe8-1b9bce696916-kube-api-access-gl682\") pod \"cc526934-1cca-481c-bfe8-1b9bce696916\" (UID: \"cc526934-1cca-481c-bfe8-1b9bce696916\") "
Oct 30 01:16:03.546349 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:03.545974    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cc526934-1cca-481c-bfe8-1b9bce696916-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc526934-1cca-481c-bfe8-1b9bce696916-maas-api-xks-ca\") pod \"cc526934-1cca-481c-bfe8-1b9bce696916\" (UID: \"cc526934-1cca-481c-bfe8-1b9bce696916\") "
Oct 30 01:16:03.548349 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:03.546890    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc526934-1cca-481c-bfe8-1b9bce696916-maas-api-openshift-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:16:03.549470 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:03.549357    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc526934-1cca-481c-bfe8-1b9bce696916-kube-api-access-gl682" pod "cc526934-1cca-481c-bfe8-1b9bce696916" (UID: "cc526934-1cca-481c-bfe8-1b9bce696916"). InnerVolumeSpecName "kube-api-access-gl682". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:16:03.606716 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:03.605565    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc526934-1cca-481c-bfe8-1b9bce696916-maas-api-xks-ca" pod "cc526934-1cca-481c-bfe8-1b9bce696916" (UID: "cc526934-1cca-481c-bfe8-1b9bce696916"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:16:03.646688 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:03.644730    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-52650f-x22xt"
Oct 30 01:16:03.646688 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:03.645087    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-52650f-x22xt" event={"ID":"cc526934-1cca-481c-bfe8-1b9bce696916","Type":"ContainerDied","Data":"fd0e427cefd0e5ce6958d0824e93c6ab9a29974dd60b1e2bc903f084ef9e9eff"}
Oct 30 01:16:03.646688 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:03.645141    5002 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fd0e427cefd0e5ce6958d0824e93c6ab9a29974dd60b1e2bc903f084ef9e9eff"
Oct 30 01:16:03.660445 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:03.649222    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gl682\" (UniqueName: \"kubernetes.io/projected/cc526934-1cca-481c-bfe8-1b9bce696916-kube-api-access-gl682\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:16:03.660445 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:03.649259    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc526934-1cca-481c-bfe8-1b9bce696916-maas-api-xks-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:16:04.196250 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:04.196212    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-52650f/distinct-w0-52650f-kserve-d85899b5-vlhps"]
Oct 30 01:16:04.204418 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:04.204328    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-52650f/premium-w0-52650f-kserve-c5bf66f9d-qq5dm"]
Oct 30 01:16:04.214089 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:04.214042    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-52650f/simulator-w0-52650f-kserve-58b64766c-94qxx"]
Oct 30 01:16:05.748005 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:05.747967    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f2b534c-9xvkw"]
Oct 30 01:16:05.748638 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:05.748621    5002 cpu_manager.go:414] "containerMap: removing container" podUID="cc526934-1cca-481c-bfe8-1b9bce696916" containerName="revoke-keys"
Oct 30 01:16:05.748727 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:05.748638    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc526934-1cca-481c-bfe8-1b9bce696916" containerName="revoke-keys"
Oct 30 01:16:05.748785 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:05.748760    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc526934-1cca-481c-bfe8-1b9bce696916" containerName="revoke-keys"
Oct 30 01:16:05.748785 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:05.748775    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3f9e906-5594-4cdf-a620-efc9cbc28a44" containerName="istio-proxy"
Oct 30 01:16:05.752878 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:05.752854    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f2b534c-9xvkw"
Oct 30 01:16:05.773712 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:05.773678    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f2b534c-9xvkw"]
Oct 30 01:16:05.782676 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:05.782648    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:16:05.782836 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:05.782765    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:16:05.782836 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:05.782776    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5phsh\""
Oct 30 01:16:05.785525 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:05.785497    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:16:05.827759 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:05.827720    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5525c91f-ca9d-41b2-b2eb-4cf53262541c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8f2b534c-9xvkw\" (UID: \"5525c91f-ca9d-41b2-b2eb-4cf53262541c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f2b534c-9xvkw"
Oct 30 01:16:05.827935 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:05.827879    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kk27c\" (UniqueName: \"kubernetes.io/projected/5525c91f-ca9d-41b2-b2eb-4cf53262541c-kube-api-access-kk27c\") pod \"maas-api-revoke-sub-8f2b534c-9xvkw\" (UID: \"5525c91f-ca9d-41b2-b2eb-4cf53262541c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f2b534c-9xvkw"
Oct 30 01:16:05.828104 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:05.828080    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5525c91f-ca9d-41b2-b2eb-4cf53262541c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8f2b534c-9xvkw\" (UID: \"5525c91f-ca9d-41b2-b2eb-4cf53262541c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f2b534c-9xvkw"
Oct 30 01:16:05.929658 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:05.929608    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5525c91f-ca9d-41b2-b2eb-4cf53262541c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8f2b534c-9xvkw\" (UID: \"5525c91f-ca9d-41b2-b2eb-4cf53262541c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f2b534c-9xvkw"
Oct 30 01:16:05.929880 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:05.929710    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kk27c\" (UniqueName: \"kubernetes.io/projected/5525c91f-ca9d-41b2-b2eb-4cf53262541c-kube-api-access-kk27c\") pod \"maas-api-revoke-sub-8f2b534c-9xvkw\" (UID: \"5525c91f-ca9d-41b2-b2eb-4cf53262541c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f2b534c-9xvkw"
Oct 30 01:16:05.929944 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:05.929913    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5525c91f-ca9d-41b2-b2eb-4cf53262541c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8f2b534c-9xvkw\" (UID: \"5525c91f-ca9d-41b2-b2eb-4cf53262541c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f2b534c-9xvkw"
Oct 30 01:16:05.930824 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:05.930793    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5525c91f-ca9d-41b2-b2eb-4cf53262541c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8f2b534c-9xvkw\" (UID: \"5525c91f-ca9d-41b2-b2eb-4cf53262541c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f2b534c-9xvkw"
Oct 30 01:16:05.932257 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:05.932232    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5525c91f-ca9d-41b2-b2eb-4cf53262541c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8f2b534c-9xvkw\" (UID: \"5525c91f-ca9d-41b2-b2eb-4cf53262541c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f2b534c-9xvkw"
Oct 30 01:16:05.973403 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:05.960099    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kk27c\" (UniqueName: \"kubernetes.io/projected/5525c91f-ca9d-41b2-b2eb-4cf53262541c-kube-api-access-kk27c\") pod \"maas-api-revoke-sub-8f2b534c-9xvkw\" (UID: \"5525c91f-ca9d-41b2-b2eb-4cf53262541c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f2b534c-9xvkw"
Oct 30 01:16:06.070445 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:06.070078    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f2b534c-9xvkw"
Oct 30 01:16:06.645885 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:06.645847    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f2b534c-9xvkw"]
Oct 30 01:16:06.676323 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:06.676235    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f2b534c-9xvkw" event={"ID":"5525c91f-ca9d-41b2-b2eb-4cf53262541c","Type":"ContainerStarted","Data":"80bfc02bf819afbae9fa74c55f7f26ba6a63e62d6301d7b892607afff22f3003"}
Oct 30 01:16:07.688156 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:07.688115    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f2b534c-9xvkw" event={"ID":"5525c91f-ca9d-41b2-b2eb-4cf53262541c","Type":"ContainerStarted","Data":"ba5ceaa053f661f3ae62cc6771ba6eca78089c96823f62396a13bd50c999cc95"}
Oct 30 01:16:07.734575 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:07.734516    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f2b534c-9xvkw" podStartSLOduration=2.7345009620000003 podStartE2EDuration="2.734500962s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:16:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:16:07.732351689 +0000 UTC m=+2663.348784588" watchObservedRunningTime="2026-10-30 01:16:07.734500962 +0000 UTC m=+2663.350933841"
Oct 30 01:16:08.698980 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:08.698944    5002 generic.go:356] "Generic (PLEG): container finished" podID="5525c91f-ca9d-41b2-b2eb-4cf53262541c" containerID="ba5ceaa053f661f3ae62cc6771ba6eca78089c96823f62396a13bd50c999cc95" exitCode=0
Oct 30 01:16:08.699418 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:08.699032    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f2b534c-9xvkw" event={"ID":"5525c91f-ca9d-41b2-b2eb-4cf53262541c","Type":"ContainerDied","Data":"ba5ceaa053f661f3ae62cc6771ba6eca78089c96823f62396a13bd50c999cc95"}
Oct 30 01:16:09.875908 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:09.875882    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f2b534c-9xvkw"
Oct 30 01:16:09.923166 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:09.923134    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5525c91f-ca9d-41b2-b2eb-4cf53262541c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5525c91f-ca9d-41b2-b2eb-4cf53262541c-maas-api-xks-ca\") pod \"5525c91f-ca9d-41b2-b2eb-4cf53262541c\" (UID: \"5525c91f-ca9d-41b2-b2eb-4cf53262541c\") "
Oct 30 01:16:09.923351 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:09.923218    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5525c91f-ca9d-41b2-b2eb-4cf53262541c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5525c91f-ca9d-41b2-b2eb-4cf53262541c-maas-api-openshift-ca\") pod \"5525c91f-ca9d-41b2-b2eb-4cf53262541c\" (UID: \"5525c91f-ca9d-41b2-b2eb-4cf53262541c\") "
Oct 30 01:16:09.923351 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:09.923333    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5525c91f-ca9d-41b2-b2eb-4cf53262541c-kube-api-access-kk27c\" (UniqueName: \"kubernetes.io/projected/5525c91f-ca9d-41b2-b2eb-4cf53262541c-kube-api-access-kk27c\") pod \"5525c91f-ca9d-41b2-b2eb-4cf53262541c\" (UID: \"5525c91f-ca9d-41b2-b2eb-4cf53262541c\") "
Oct 30 01:16:09.927661 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:09.927613    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5525c91f-ca9d-41b2-b2eb-4cf53262541c-maas-api-xks-ca" pod "5525c91f-ca9d-41b2-b2eb-4cf53262541c" (UID: "5525c91f-ca9d-41b2-b2eb-4cf53262541c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:16:09.927871 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:09.927742    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5525c91f-ca9d-41b2-b2eb-4cf53262541c-kube-api-access-kk27c" pod "5525c91f-ca9d-41b2-b2eb-4cf53262541c" (UID: "5525c91f-ca9d-41b2-b2eb-4cf53262541c"). InnerVolumeSpecName "kube-api-access-kk27c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:16:09.929359 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:09.929329    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5525c91f-ca9d-41b2-b2eb-4cf53262541c-maas-api-openshift-ca" pod "5525c91f-ca9d-41b2-b2eb-4cf53262541c" (UID: "5525c91f-ca9d-41b2-b2eb-4cf53262541c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:16:10.025487 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:10.025405    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5525c91f-ca9d-41b2-b2eb-4cf53262541c-maas-api-openshift-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:16:10.025487 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:10.025436    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kk27c\" (UniqueName: \"kubernetes.io/projected/5525c91f-ca9d-41b2-b2eb-4cf53262541c-kube-api-access-kk27c\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:16:10.025487 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:10.025447    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5525c91f-ca9d-41b2-b2eb-4cf53262541c-maas-api-xks-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:16:10.717245 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:10.717209    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f2b534c-9xvkw"
Oct 30 01:16:10.717445 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:10.717243    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f2b534c-9xvkw" event={"ID":"5525c91f-ca9d-41b2-b2eb-4cf53262541c","Type":"ContainerDied","Data":"80bfc02bf819afbae9fa74c55f7f26ba6a63e62d6301d7b892607afff22f3003"}
Oct 30 01:16:10.717445 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:10.717290    5002 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="80bfc02bf819afbae9fa74c55f7f26ba6a63e62d6301d7b892607afff22f3003"
Oct 30 01:16:12.863827 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:16:12.863783    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbjdq" containerName="extractor"
Oct 30 01:16:16.089539 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:16.089500    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6e506fd-kv6tc"]
Oct 30 01:16:16.090527 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:16.090503    5002 cpu_manager.go:414] "containerMap: removing container" podUID="5525c91f-ca9d-41b2-b2eb-4cf53262541c" containerName="revoke-keys"
Oct 30 01:16:16.090584 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:16.090529    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5525c91f-ca9d-41b2-b2eb-4cf53262541c" containerName="revoke-keys"
Oct 30 01:16:16.090701 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:16.090689    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="5525c91f-ca9d-41b2-b2eb-4cf53262541c" containerName="revoke-keys"
Oct 30 01:16:16.094987 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:16.094966    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6e506fd-kv6tc"
Oct 30 01:16:16.111308 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:16.111272    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6e506fd-kv6tc"]
Oct 30 01:16:16.120683 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:16.120623    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:16:16.120860 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:16.120645    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:16:16.120948 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:16.120671    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:16:16.120948 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:16.120728    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5phsh\""
Oct 30 01:16:16.213497 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:16.213462    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44d5bcd2-3029-4d1d-b163-f90ee946d53b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d6e506fd-kv6tc\" (UID: \"44d5bcd2-3029-4d1d-b163-f90ee946d53b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6e506fd-kv6tc"
Oct 30 01:16:16.213690 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:16.213676    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dn5xg\" (UniqueName: \"kubernetes.io/projected/44d5bcd2-3029-4d1d-b163-f90ee946d53b-kube-api-access-dn5xg\") pod \"maas-api-revoke-sub-d6e506fd-kv6tc\" (UID: \"44d5bcd2-3029-4d1d-b163-f90ee946d53b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6e506fd-kv6tc"
Oct 30 01:16:16.213754 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:16.213732    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44d5bcd2-3029-4d1d-b163-f90ee946d53b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d6e506fd-kv6tc\" (UID: \"44d5bcd2-3029-4d1d-b163-f90ee946d53b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6e506fd-kv6tc"
Oct 30 01:16:16.315545 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:16.315499    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dn5xg\" (UniqueName: \"kubernetes.io/projected/44d5bcd2-3029-4d1d-b163-f90ee946d53b-kube-api-access-dn5xg\") pod \"maas-api-revoke-sub-d6e506fd-kv6tc\" (UID: \"44d5bcd2-3029-4d1d-b163-f90ee946d53b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6e506fd-kv6tc"
Oct 30 01:16:16.315760 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:16.315585    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44d5bcd2-3029-4d1d-b163-f90ee946d53b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d6e506fd-kv6tc\" (UID: \"44d5bcd2-3029-4d1d-b163-f90ee946d53b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6e506fd-kv6tc"
Oct 30 01:16:16.315760 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:16.315676    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44d5bcd2-3029-4d1d-b163-f90ee946d53b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d6e506fd-kv6tc\" (UID: \"44d5bcd2-3029-4d1d-b163-f90ee946d53b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6e506fd-kv6tc"
Oct 30 01:16:16.316372 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:16.316352    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44d5bcd2-3029-4d1d-b163-f90ee946d53b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d6e506fd-kv6tc\" (UID: \"44d5bcd2-3029-4d1d-b163-f90ee946d53b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6e506fd-kv6tc"
Oct 30 01:16:16.318282 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:16.318260    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44d5bcd2-3029-4d1d-b163-f90ee946d53b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d6e506fd-kv6tc\" (UID: \"44d5bcd2-3029-4d1d-b163-f90ee946d53b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6e506fd-kv6tc"
Oct 30 01:16:16.340716 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:16.340650    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dn5xg\" (UniqueName: \"kubernetes.io/projected/44d5bcd2-3029-4d1d-b163-f90ee946d53b-kube-api-access-dn5xg\") pod \"maas-api-revoke-sub-d6e506fd-kv6tc\" (UID: \"44d5bcd2-3029-4d1d-b163-f90ee946d53b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6e506fd-kv6tc"
Oct 30 01:16:16.411706 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:16.411663    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6e506fd-kv6tc"
Oct 30 01:16:16.773565 ip-10-0-128-7 kubenswrapper[5002]: W1030 01:16:16.773529    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44d5bcd2_3029_4d1d_b163_f90ee946d53b.slice/crio-6757a0bf7607f741dbff05381f063c73a434f068c9a1255a6e94467d8cbb1d98 WatchSource:0}: Error finding container 6757a0bf7607f741dbff05381f063c73a434f068c9a1255a6e94467d8cbb1d98: Status 404 returned error can't find the container with id 6757a0bf7607f741dbff05381f063c73a434f068c9a1255a6e94467d8cbb1d98
Oct 30 01:16:16.774509 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:16.774481    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6e506fd-kv6tc"]
Oct 30 01:16:17.778117 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:17.778081    5002 generic.go:356] "Generic (PLEG): container finished" podID="44d5bcd2-3029-4d1d-b163-f90ee946d53b" containerID="bcf9afd51c11fb718dd6a70d15ada064c566d06c34fc8a6790a9a7562ac7f613" exitCode=0
Oct 30 01:16:17.778671 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:17.778173    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6e506fd-kv6tc" event={"ID":"44d5bcd2-3029-4d1d-b163-f90ee946d53b","Type":"ContainerDied","Data":"bcf9afd51c11fb718dd6a70d15ada064c566d06c34fc8a6790a9a7562ac7f613"}
Oct 30 01:16:17.778671 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:17.778230    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6e506fd-kv6tc" event={"ID":"44d5bcd2-3029-4d1d-b163-f90ee946d53b","Type":"ContainerStarted","Data":"6757a0bf7607f741dbff05381f063c73a434f068c9a1255a6e94467d8cbb1d98"}
Oct 30 01:16:18.967810 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:18.967782    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6e506fd-kv6tc"
Oct 30 01:16:19.056758 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:19.056656    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44d5bcd2-3029-4d1d-b163-f90ee946d53b-kube-api-access-dn5xg\" (UniqueName: \"kubernetes.io/projected/44d5bcd2-3029-4d1d-b163-f90ee946d53b-kube-api-access-dn5xg\") pod \"44d5bcd2-3029-4d1d-b163-f90ee946d53b\" (UID: \"44d5bcd2-3029-4d1d-b163-f90ee946d53b\") "
Oct 30 01:16:19.056947 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:19.056831    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44d5bcd2-3029-4d1d-b163-f90ee946d53b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44d5bcd2-3029-4d1d-b163-f90ee946d53b-maas-api-openshift-ca\") pod \"44d5bcd2-3029-4d1d-b163-f90ee946d53b\" (UID: \"44d5bcd2-3029-4d1d-b163-f90ee946d53b\") "
Oct 30 01:16:19.056947 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:19.056907    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/44d5bcd2-3029-4d1d-b163-f90ee946d53b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44d5bcd2-3029-4d1d-b163-f90ee946d53b-maas-api-xks-ca\") pod \"44d5bcd2-3029-4d1d-b163-f90ee946d53b\" (UID: \"44d5bcd2-3029-4d1d-b163-f90ee946d53b\") "
Oct 30 01:16:19.057844 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:19.057810    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44d5bcd2-3029-4d1d-b163-f90ee946d53b-maas-api-openshift-ca" pod "44d5bcd2-3029-4d1d-b163-f90ee946d53b" (UID: "44d5bcd2-3029-4d1d-b163-f90ee946d53b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:16:19.060151 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:19.060124    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44d5bcd2-3029-4d1d-b163-f90ee946d53b-maas-api-xks-ca" pod "44d5bcd2-3029-4d1d-b163-f90ee946d53b" (UID: "44d5bcd2-3029-4d1d-b163-f90ee946d53b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:16:19.071099 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:19.071068    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44d5bcd2-3029-4d1d-b163-f90ee946d53b-kube-api-access-dn5xg" pod "44d5bcd2-3029-4d1d-b163-f90ee946d53b" (UID: "44d5bcd2-3029-4d1d-b163-f90ee946d53b"). InnerVolumeSpecName "kube-api-access-dn5xg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:16:19.158834 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:19.158797    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44d5bcd2-3029-4d1d-b163-f90ee946d53b-maas-api-openshift-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:16:19.158834 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:19.158831    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44d5bcd2-3029-4d1d-b163-f90ee946d53b-maas-api-xks-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:16:19.159050 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:19.158843    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dn5xg\" (UniqueName: \"kubernetes.io/projected/44d5bcd2-3029-4d1d-b163-f90ee946d53b-kube-api-access-dn5xg\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:16:19.222215 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:19.222167    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-52650f/distinct-w0-52650f-kserve-d85899b5-vlhps" podUID="ea1850d9-abba-4977-a540-f6e0eaefb17d" containerName="main" containerID="cri-o://11f466b38ab95d14798e8bdccca6fbc6dbe5350bec4ca0c79a57bf6d8942c6c4" gracePeriod=45
Oct 30 01:16:19.227611 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:19.227563    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-52650f/premium-w0-52650f-kserve-c5bf66f9d-qq5dm" podUID="3455d983-b11b-4cee-bfce-d1f399cf9679" containerName="main" containerID="cri-o://1ecfd3f6f656034f89b9864941c49a69ad3a47309acb1fc74491cc34c84cf548" gracePeriod=45
Oct 30 01:16:19.237108 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:19.237069    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-52650f/simulator-w0-52650f-kserve-58b64766c-94qxx" podUID="659c1425-5b67-4d01-bfc2-f1ba56b9a59b" containerName="main" containerID="cri-o://d1ec256d75183565f2cd8e1b1a95e9be6778c4671262e46758e7e8e46e506a3e" gracePeriod=45
Oct 30 01:16:19.548834 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:19.548804    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-52650f/premium-w0-52650f-kserve-c5bf66f9d-qq5dm"
Oct 30 01:16:19.562458 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:19.562433    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-52650f/distinct-w0-52650f-kserve-d85899b5-vlhps"
Oct 30 01:16:19.605133 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:19.605055    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-52650f/simulator-w0-52650f-kserve-58b64766c-94qxx"
Oct 30 01:16:19.671066 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:19.671027    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ea1850d9-abba-4977-a540-f6e0eaefb17d-tls-certs\" (UniqueName: \"kubernetes.io/secret/ea1850d9-abba-4977-a540-f6e0eaefb17d-tls-certs\") pod \"ea1850d9-abba-4977-a540-f6e0eaefb17d\" (UID: \"ea1850d9-abba-4977-a540-f6e0eaefb17d\") "
Oct 30 01:16:19.671323 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:19.671296    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3455d983-b11b-4cee-bfce-d1f399cf9679-home\" (UniqueName: \"kubernetes.io/empty-dir/3455d983-b11b-4cee-bfce-d1f399cf9679-home\") pod \"3455d983-b11b-4cee-bfce-d1f399cf9679\" (UID: \"3455d983-b11b-4cee-bfce-d1f399cf9679\") "
Oct 30 01:16:19.671608 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:19.671589    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea1850d9-abba-4977-a540-f6e0eaefb17d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea1850d9-abba-4977-a540-f6e0eaefb17d-model-cache\") pod \"ea1850d9-abba-4977-a540-f6e0eaefb17d\" (UID: \"ea1850d9-abba-4977-a540-f6e0eaefb17d\") "
Oct 30 01:16:19.671787 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:19.671764    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3455d983-b11b-4cee-bfce-d1f399cf9679-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3455d983-b11b-4cee-bfce-d1f399cf9679-model-cache\") pod \"3455d983-b11b-4cee-bfce-d1f399cf9679\" (UID: \"3455d983-b11b-4cee-bfce-d1f399cf9679\") "
Oct 30 01:16:19.671910 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:19.671895    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3455d983-b11b-4cee-bfce-d1f399cf9679-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3455d983-b11b-4cee-bfce-d1f399cf9679-dshm\") pod \"3455d983-b11b-4cee-bfce-d1f399cf9679\" (UID: \"3455d983-b11b-4cee-bfce-d1f399cf9679\") "
Oct 30 01:16:19.672088 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:19.672072    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea1850d9-abba-4977-a540-f6e0eaefb17d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea1850d9-abba-4977-a540-f6e0eaefb17d-tmp-dir\") pod \"ea1850d9-abba-4977-a540-f6e0eaefb17d\" (UID: \"ea1850d9-abba-4977-a540-f6e0eaefb17d\") "
Oct 30 01:16:19.672207 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:19.672193    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3455d983-b11b-4cee-bfce-d1f399cf9679-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3455d983-b11b-4cee-bfce-d1f399cf9679-tmp-dir\") pod \"3455d983-b11b-4cee-bfce-d1f399cf9679\" (UID: \"3455d983-b11b-4cee-bfce-d1f399cf9679\") "
Oct 30 01:16:19.672328 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:19.672314    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea1850d9-abba-4977-a540-f6e0eaefb17d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea1850d9-abba-4977-a540-f6e0eaefb17d-dshm\") pod \"ea1850d9-abba-4977-a540-f6e0eaefb17d\" (UID: \"ea1850d9-abba-4977-a540-f6e0eaefb17d\") "
Oct 30 01:16:19.672545 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:19.672527    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3455d983-b11b-4cee-bfce-d1f399cf9679-tls-certs\" (UniqueName: \"kubernetes.io/secret/3455d983-b11b-4cee-bfce-d1f399cf9679-tls-certs\") pod \"3455d983-b11b-4cee-bfce-d1f399cf9679\" (UID: \"3455d983-b11b-4cee-bfce-d1f399cf9679\") "
Oct 30 01:16:19.672881 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:19.672861    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3455d983-b11b-4cee-bfce-d1f399cf9679-kube-api-access-82sbm\" (UniqueName: \"kubernetes.io/projected/3455d983-b11b-4cee-bfce-d1f399cf9679-kube-api-access-82sbm\") pod \"3455d983-b11b-4cee-bfce-d1f399cf9679\" (UID: \"3455d983-b11b-4cee-bfce-d1f399cf9679\") "
Oct 30 01:16:19.673031 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:19.673017    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea1850d9-abba-4977-a540-f6e0eaefb17d-kube-api-access-8hf95\" (UniqueName: \"kubernetes.io/projected/ea1850d9-abba-4977-a540-f6e0eaefb17d-kube-api-access-8hf95\") pod \"ea1850d9-abba-4977-a540-f6e0eaefb17d\" (UID: \"ea1850d9-abba-4977-a540-f6e0eaefb17d\") "
Oct 30 01:16:19.673238 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:19.673224    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea1850d9-abba-4977-a540-f6e0eaefb17d-home\" (UniqueName: \"kubernetes.io/empty-dir/ea1850d9-abba-4977-a540-f6e0eaefb17d-home\") pod \"ea1850d9-abba-4977-a540-f6e0eaefb17d\" (UID: \"ea1850d9-abba-4977-a540-f6e0eaefb17d\") "
Oct 30 01:16:19.674281 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:19.674258    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea1850d9-abba-4977-a540-f6e0eaefb17d-home" pod "ea1850d9-abba-4977-a540-f6e0eaefb17d" (UID: "ea1850d9-abba-4977-a540-f6e0eaefb17d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:16:19.675492 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:19.675109    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea1850d9-abba-4977-a540-f6e0eaefb17d-model-cache" pod "ea1850d9-abba-4977-a540-f6e0eaefb17d" (UID: "ea1850d9-abba-4977-a540-f6e0eaefb17d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:16:19.675631 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:19.675570    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3455d983-b11b-4cee-bfce-d1f399cf9679-model-cache" pod "3455d983-b11b-4cee-bfce-d1f399cf9679" (UID: "3455d983-b11b-4cee-bfce-d1f399cf9679"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:16:19.678984 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:19.677357    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3455d983-b11b-4cee-bfce-d1f399cf9679-tmp-dir" pod "3455d983-b11b-4cee-bfce-d1f399cf9679" (UID: "3455d983-b11b-4cee-bfce-d1f399cf9679"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:16:19.678984 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:19.677430    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3455d983-b11b-4cee-bfce-d1f399cf9679-home" pod "3455d983-b11b-4cee-bfce-d1f399cf9679" (UID: "3455d983-b11b-4cee-bfce-d1f399cf9679"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:16:19.678984 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:19.677468    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea1850d9-abba-4977-a540-f6e0eaefb17d-tmp-dir" pod "ea1850d9-abba-4977-a540-f6e0eaefb17d" (UID: "ea1850d9-abba-4977-a540-f6e0eaefb17d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:16:19.680997 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:19.680976    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3455d983-b11b-4cee-bfce-d1f399cf9679-dshm" pod "3455d983-b11b-4cee-bfce-d1f399cf9679" (UID: "3455d983-b11b-4cee-bfce-d1f399cf9679"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:16:19.681182 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:19.681165    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea1850d9-abba-4977-a540-f6e0eaefb17d-dshm" pod "ea1850d9-abba-4977-a540-f6e0eaefb17d" (UID: "ea1850d9-abba-4977-a540-f6e0eaefb17d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:16:19.681523 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:19.681503    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea1850d9-abba-4977-a540-f6e0eaefb17d-kube-api-access-8hf95" pod "ea1850d9-abba-4977-a540-f6e0eaefb17d" (UID: "ea1850d9-abba-4977-a540-f6e0eaefb17d"). InnerVolumeSpecName "kube-api-access-8hf95". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:16:19.681650 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:19.681634    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea1850d9-abba-4977-a540-f6e0eaefb17d-tls-certs" pod "ea1850d9-abba-4977-a540-f6e0eaefb17d" (UID: "ea1850d9-abba-4977-a540-f6e0eaefb17d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:16:19.681965 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:19.681945    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3455d983-b11b-4cee-bfce-d1f399cf9679-kube-api-access-82sbm" pod "3455d983-b11b-4cee-bfce-d1f399cf9679" (UID: "3455d983-b11b-4cee-bfce-d1f399cf9679"). InnerVolumeSpecName "kube-api-access-82sbm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:16:19.683050 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:19.682997    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3455d983-b11b-4cee-bfce-d1f399cf9679-tls-certs" pod "3455d983-b11b-4cee-bfce-d1f399cf9679" (UID: "3455d983-b11b-4cee-bfce-d1f399cf9679"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:16:19.774686 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:19.774647    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/659c1425-5b67-4d01-bfc2-f1ba56b9a59b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/659c1425-5b67-4d01-bfc2-f1ba56b9a59b-tmp-dir\") pod \"659c1425-5b67-4d01-bfc2-f1ba56b9a59b\" (UID: \"659c1425-5b67-4d01-bfc2-f1ba56b9a59b\") "
Oct 30 01:16:19.774893 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:19.774744    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/659c1425-5b67-4d01-bfc2-f1ba56b9a59b-kube-api-access-5qtkz\" (UniqueName: \"kubernetes.io/projected/659c1425-5b67-4d01-bfc2-f1ba56b9a59b-kube-api-access-5qtkz\") pod \"659c1425-5b67-4d01-bfc2-f1ba56b9a59b\" (UID: \"659c1425-5b67-4d01-bfc2-f1ba56b9a59b\") "
Oct 30 01:16:19.774893 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:19.774800    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/659c1425-5b67-4d01-bfc2-f1ba56b9a59b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/659c1425-5b67-4d01-bfc2-f1ba56b9a59b-model-cache\") pod \"659c1425-5b67-4d01-bfc2-f1ba56b9a59b\" (UID: \"659c1425-5b67-4d01-bfc2-f1ba56b9a59b\") "
Oct 30 01:16:19.774893 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:19.774889    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/659c1425-5b67-4d01-bfc2-f1ba56b9a59b-home\" (UniqueName: \"kubernetes.io/empty-dir/659c1425-5b67-4d01-bfc2-f1ba56b9a59b-home\") pod \"659c1425-5b67-4d01-bfc2-f1ba56b9a59b\" (UID: \"659c1425-5b67-4d01-bfc2-f1ba56b9a59b\") "
Oct 30 01:16:19.775084 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:19.774960    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/659c1425-5b67-4d01-bfc2-f1ba56b9a59b-tls-certs\" (UniqueName: \"kubernetes.io/secret/659c1425-5b67-4d01-bfc2-f1ba56b9a59b-tls-certs\") pod \"659c1425-5b67-4d01-bfc2-f1ba56b9a59b\" (UID: \"659c1425-5b67-4d01-bfc2-f1ba56b9a59b\") "
Oct 30 01:16:19.775084 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:19.774957    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/659c1425-5b67-4d01-bfc2-f1ba56b9a59b-tmp-dir" pod "659c1425-5b67-4d01-bfc2-f1ba56b9a59b" (UID: "659c1425-5b67-4d01-bfc2-f1ba56b9a59b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:16:19.775084 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:19.775003    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/659c1425-5b67-4d01-bfc2-f1ba56b9a59b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/659c1425-5b67-4d01-bfc2-f1ba56b9a59b-dshm\") pod \"659c1425-5b67-4d01-bfc2-f1ba56b9a59b\" (UID: \"659c1425-5b67-4d01-bfc2-f1ba56b9a59b\") "
Oct 30 01:16:19.775084 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:19.775032    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/659c1425-5b67-4d01-bfc2-f1ba56b9a59b-model-cache" pod "659c1425-5b67-4d01-bfc2-f1ba56b9a59b" (UID: "659c1425-5b67-4d01-bfc2-f1ba56b9a59b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:16:19.775294 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:19.775164    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/659c1425-5b67-4d01-bfc2-f1ba56b9a59b-home" pod "659c1425-5b67-4d01-bfc2-f1ba56b9a59b" (UID: "659c1425-5b67-4d01-bfc2-f1ba56b9a59b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:16:19.775800 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:19.775776    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-82sbm\" (UniqueName: \"kubernetes.io/projected/3455d983-b11b-4cee-bfce-d1f399cf9679-kube-api-access-82sbm\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:16:19.776002 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:19.775977    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8hf95\" (UniqueName: \"kubernetes.io/projected/ea1850d9-abba-4977-a540-f6e0eaefb17d-kube-api-access-8hf95\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:16:19.776110 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:19.776088    5002 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea1850d9-abba-4977-a540-f6e0eaefb17d-home\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:16:19.776110 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:19.776110    5002 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea1850d9-abba-4977-a540-f6e0eaefb17d-tls-certs\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:16:19.776486 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:19.776124    5002 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3455d983-b11b-4cee-bfce-d1f399cf9679-home\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:16:19.776486 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:19.776137    5002 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/659c1425-5b67-4d01-bfc2-f1ba56b9a59b-tmp-dir\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:16:19.776486 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:19.776149    5002 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea1850d9-abba-4977-a540-f6e0eaefb17d-model-cache\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:16:19.776486 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:19.776164    5002 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3455d983-b11b-4cee-bfce-d1f399cf9679-model-cache\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:16:19.776486 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:19.776177    5002 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3455d983-b11b-4cee-bfce-d1f399cf9679-dshm\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:16:19.776486 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:19.776189    5002 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/659c1425-5b67-4d01-bfc2-f1ba56b9a59b-model-cache\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:16:19.776486 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:19.776201    5002 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea1850d9-abba-4977-a540-f6e0eaefb17d-tmp-dir\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:16:19.776486 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:19.776214    5002 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3455d983-b11b-4cee-bfce-d1f399cf9679-tmp-dir\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:16:19.776486 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:19.776226    5002 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea1850d9-abba-4977-a540-f6e0eaefb17d-dshm\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:16:19.776486 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:19.776237    5002 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/659c1425-5b67-4d01-bfc2-f1ba56b9a59b-home\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:16:19.776486 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:19.776250    5002 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3455d983-b11b-4cee-bfce-d1f399cf9679-tls-certs\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:16:19.777731 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:19.777691    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/659c1425-5b67-4d01-bfc2-f1ba56b9a59b-tls-certs" pod "659c1425-5b67-4d01-bfc2-f1ba56b9a59b" (UID: "659c1425-5b67-4d01-bfc2-f1ba56b9a59b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:16:19.777959 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:19.777939    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/659c1425-5b67-4d01-bfc2-f1ba56b9a59b-kube-api-access-5qtkz" pod "659c1425-5b67-4d01-bfc2-f1ba56b9a59b" (UID: "659c1425-5b67-4d01-bfc2-f1ba56b9a59b"). InnerVolumeSpecName "kube-api-access-5qtkz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:16:19.778026 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:19.778004    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/659c1425-5b67-4d01-bfc2-f1ba56b9a59b-dshm" pod "659c1425-5b67-4d01-bfc2-f1ba56b9a59b" (UID: "659c1425-5b67-4d01-bfc2-f1ba56b9a59b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:16:19.793337 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:19.793313    5002 generic.go:356] "Generic (PLEG): container finished" podID="ea1850d9-abba-4977-a540-f6e0eaefb17d" containerID="11f466b38ab95d14798e8bdccca6fbc6dbe5350bec4ca0c79a57bf6d8942c6c4" exitCode=0
Oct 30 01:16:19.793503 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:19.793392    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-52650f/distinct-w0-52650f-kserve-d85899b5-vlhps"
Oct 30 01:16:19.793503 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:19.793434    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-52650f/distinct-w0-52650f-kserve-d85899b5-vlhps" event={"ID":"ea1850d9-abba-4977-a540-f6e0eaefb17d","Type":"ContainerDied","Data":"11f466b38ab95d14798e8bdccca6fbc6dbe5350bec4ca0c79a57bf6d8942c6c4"}
Oct 30 01:16:19.793503 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:19.793490    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-52650f/distinct-w0-52650f-kserve-d85899b5-vlhps" event={"ID":"ea1850d9-abba-4977-a540-f6e0eaefb17d","Type":"ContainerDied","Data":"7344b1bfd00b82bbb1533f91c8b6f8a119d3010d2f8876fed4aacf2a5fd683c3"}
Oct 30 01:16:19.793654 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:19.793514    5002 scope.go:122] "RemoveContainer" containerID="11f466b38ab95d14798e8bdccca6fbc6dbe5350bec4ca0c79a57bf6d8942c6c4"
Oct 30 01:16:19.795005 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:19.794990    5002 generic.go:356] "Generic (PLEG): container finished" podID="3455d983-b11b-4cee-bfce-d1f399cf9679" containerID="1ecfd3f6f656034f89b9864941c49a69ad3a47309acb1fc74491cc34c84cf548" exitCode=0
Oct 30 01:16:19.795150 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:19.795128    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-52650f/premium-w0-52650f-kserve-c5bf66f9d-qq5dm" event={"ID":"3455d983-b11b-4cee-bfce-d1f399cf9679","Type":"ContainerDied","Data":"1ecfd3f6f656034f89b9864941c49a69ad3a47309acb1fc74491cc34c84cf548"}
Oct 30 01:16:19.795230 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:19.795168    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-52650f/premium-w0-52650f-kserve-c5bf66f9d-qq5dm" event={"ID":"3455d983-b11b-4cee-bfce-d1f399cf9679","Type":"ContainerDied","Data":"3e66fa365375faab474fa4216f5ff0b2982a4671ca16caa4c456af5ddcb7df64"}
Oct 30 01:16:19.795316 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:19.795299    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-52650f/premium-w0-52650f-kserve-c5bf66f9d-qq5dm"
Oct 30 01:16:19.796782 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:19.796754    5002 generic.go:356] "Generic (PLEG): container finished" podID="659c1425-5b67-4d01-bfc2-f1ba56b9a59b" containerID="d1ec256d75183565f2cd8e1b1a95e9be6778c4671262e46758e7e8e46e506a3e" exitCode=0
Oct 30 01:16:19.796891 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:19.796836    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-52650f/simulator-w0-52650f-kserve-58b64766c-94qxx" event={"ID":"659c1425-5b67-4d01-bfc2-f1ba56b9a59b","Type":"ContainerDied","Data":"d1ec256d75183565f2cd8e1b1a95e9be6778c4671262e46758e7e8e46e506a3e"}
Oct 30 01:16:19.796891 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:19.796862    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-52650f/simulator-w0-52650f-kserve-58b64766c-94qxx" event={"ID":"659c1425-5b67-4d01-bfc2-f1ba56b9a59b","Type":"ContainerDied","Data":"63fc43ac43e6fbfa4ccb102ca3cb2e40f5c7930c14a74374bf0312b7cc7e5375"}
Oct 30 01:16:19.797014 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:19.796914    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-52650f/simulator-w0-52650f-kserve-58b64766c-94qxx"
Oct 30 01:16:19.800317 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:19.800294    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6e506fd-kv6tc" event={"ID":"44d5bcd2-3029-4d1d-b163-f90ee946d53b","Type":"ContainerDied","Data":"6757a0bf7607f741dbff05381f063c73a434f068c9a1255a6e94467d8cbb1d98"}
Oct 30 01:16:19.800427 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:19.800337    5002 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6757a0bf7607f741dbff05381f063c73a434f068c9a1255a6e94467d8cbb1d98"
Oct 30 01:16:19.800427 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:19.800405    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6e506fd-kv6tc"
Oct 30 01:16:19.814552 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:19.812934    5002 scope.go:122] "RemoveContainer" containerID="11f466b38ab95d14798e8bdccca6fbc6dbe5350bec4ca0c79a57bf6d8942c6c4"
Oct 30 01:16:19.815631 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:16:19.815598    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11f466b38ab95d14798e8bdccca6fbc6dbe5350bec4ca0c79a57bf6d8942c6c4\": container with ID starting with 11f466b38ab95d14798e8bdccca6fbc6dbe5350bec4ca0c79a57bf6d8942c6c4 not found: ID does not exist" containerID="11f466b38ab95d14798e8bdccca6fbc6dbe5350bec4ca0c79a57bf6d8942c6c4"
Oct 30 01:16:19.815729 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:19.815643    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11f466b38ab95d14798e8bdccca6fbc6dbe5350bec4ca0c79a57bf6d8942c6c4"} err="failed to get container status \"11f466b38ab95d14798e8bdccca6fbc6dbe5350bec4ca0c79a57bf6d8942c6c4\": rpc error: code = NotFound desc = could not find container \"11f466b38ab95d14798e8bdccca6fbc6dbe5350bec4ca0c79a57bf6d8942c6c4\": container with ID starting with 11f466b38ab95d14798e8bdccca6fbc6dbe5350bec4ca0c79a57bf6d8942c6c4 not found: ID does not exist"
Oct 30 01:16:19.815729 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:19.815669    5002 scope.go:122] "RemoveContainer" containerID="1ecfd3f6f656034f89b9864941c49a69ad3a47309acb1fc74491cc34c84cf548"
Oct 30 01:16:19.827555 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:19.827531    5002 scope.go:122] "RemoveContainer" containerID="1ecfd3f6f656034f89b9864941c49a69ad3a47309acb1fc74491cc34c84cf548"
Oct 30 01:16:19.827861 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:16:19.827836    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ecfd3f6f656034f89b9864941c49a69ad3a47309acb1fc74491cc34c84cf548\": container with ID starting with 1ecfd3f6f656034f89b9864941c49a69ad3a47309acb1fc74491cc34c84cf548 not found: ID does not exist" containerID="1ecfd3f6f656034f89b9864941c49a69ad3a47309acb1fc74491cc34c84cf548"
Oct 30 01:16:19.827911 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:19.827874    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ecfd3f6f656034f89b9864941c49a69ad3a47309acb1fc74491cc34c84cf548"} err="failed to get container status \"1ecfd3f6f656034f89b9864941c49a69ad3a47309acb1fc74491cc34c84cf548\": rpc error: code = NotFound desc = could not find container \"1ecfd3f6f656034f89b9864941c49a69ad3a47309acb1fc74491cc34c84cf548\": container with ID starting with 1ecfd3f6f656034f89b9864941c49a69ad3a47309acb1fc74491cc34c84cf548 not found: ID does not exist"
Oct 30 01:16:19.827911 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:19.827898    5002 scope.go:122] "RemoveContainer" containerID="d1ec256d75183565f2cd8e1b1a95e9be6778c4671262e46758e7e8e46e506a3e"
Oct 30 01:16:19.838100 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:19.838081    5002 scope.go:122] "RemoveContainer" containerID="d1ec256d75183565f2cd8e1b1a95e9be6778c4671262e46758e7e8e46e506a3e"
Oct 30 01:16:19.838434 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:16:19.838414    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1ec256d75183565f2cd8e1b1a95e9be6778c4671262e46758e7e8e46e506a3e\": container with ID starting with d1ec256d75183565f2cd8e1b1a95e9be6778c4671262e46758e7e8e46e506a3e not found: ID does not exist" containerID="d1ec256d75183565f2cd8e1b1a95e9be6778c4671262e46758e7e8e46e506a3e"
Oct 30 01:16:19.838503 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:19.838446    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1ec256d75183565f2cd8e1b1a95e9be6778c4671262e46758e7e8e46e506a3e"} err="failed to get container status \"d1ec256d75183565f2cd8e1b1a95e9be6778c4671262e46758e7e8e46e506a3e\": rpc error: code = NotFound desc = could not find container \"d1ec256d75183565f2cd8e1b1a95e9be6778c4671262e46758e7e8e46e506a3e\": container with ID starting with d1ec256d75183565f2cd8e1b1a95e9be6778c4671262e46758e7e8e46e506a3e not found: ID does not exist"
Oct 30 01:16:19.877683 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:19.877613    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5qtkz\" (UniqueName: \"kubernetes.io/projected/659c1425-5b67-4d01-bfc2-f1ba56b9a59b-kube-api-access-5qtkz\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:16:19.877683 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:19.877641    5002 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/659c1425-5b67-4d01-bfc2-f1ba56b9a59b-tls-certs\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:16:19.877683 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:19.877652    5002 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/659c1425-5b67-4d01-bfc2-f1ba56b9a59b-dshm\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:16:19.887717 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:19.887678    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-52650f/distinct-w0-52650f-kserve-d85899b5-vlhps"]
Oct 30 01:16:19.891203 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:19.891181    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-52650f/distinct-w0-52650f-kserve-d85899b5-vlhps"]
Oct 30 01:16:19.935103 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:19.935065    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-52650f/premium-w0-52650f-kserve-c5bf66f9d-qq5dm"]
Oct 30 01:16:19.935281 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:19.935123    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-52650f/premium-w0-52650f-kserve-c5bf66f9d-qq5dm"]
Oct 30 01:16:19.987182 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:19.987146    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-52650f/simulator-w0-52650f-kserve-58b64766c-94qxx"]
Oct 30 01:16:19.990819 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:19.990793    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-52650f/simulator-w0-52650f-kserve-58b64766c-94qxx"]
Oct 30 01:16:20.502044 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:20.502005    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3455d983-b11b-4cee-bfce-d1f399cf9679" path="/var/lib/kubelet/pods/3455d983-b11b-4cee-bfce-d1f399cf9679/volumes"
Oct 30 01:16:20.502671 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:20.502653    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="659c1425-5b67-4d01-bfc2-f1ba56b9a59b" path="/var/lib/kubelet/pods/659c1425-5b67-4d01-bfc2-f1ba56b9a59b/volumes"
Oct 30 01:16:20.503081 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:20.503067    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea1850d9-abba-4977-a540-f6e0eaefb17d" path="/var/lib/kubelet/pods/ea1850d9-abba-4977-a540-f6e0eaefb17d/volumes"
Oct 30 01:16:22.240541 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:22.240510    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-48df96-kpdrg"]
Oct 30 01:16:22.255517 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:22.255478    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-48df96-kpdrg"]
Oct 30 01:16:22.496018 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:22.495926    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2fc0346-89e4-4049-b6c5-6fd11e53f02b" path="/var/lib/kubelet/pods/f2fc0346-89e4-4049-b6c5-6fd11e53f02b/volumes"
Oct 30 01:16:25.030771 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:25.030733    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0aabe7a-6qthl"]
Oct 30 01:16:25.044141 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:25.044109    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0aabe7a-6qthl"]
Oct 30 01:16:25.522917 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:25.522886    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-9a8001-openshift-default-d885f8f7f-dt8r5"]
Oct 30 01:16:25.523571 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:25.523554    5002 cpu_manager.go:414] "containerMap: removing container" podUID="659c1425-5b67-4d01-bfc2-f1ba56b9a59b" containerName="main"
Oct 30 01:16:25.523641 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:25.523572    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="659c1425-5b67-4d01-bfc2-f1ba56b9a59b" containerName="main"
Oct 30 01:16:25.523641 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:25.523587    5002 cpu_manager.go:414] "containerMap: removing container" podUID="ea1850d9-abba-4977-a540-f6e0eaefb17d" containerName="main"
Oct 30 01:16:25.523641 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:25.523593    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea1850d9-abba-4977-a540-f6e0eaefb17d" containerName="main"
Oct 30 01:16:25.523641 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:25.523610    5002 cpu_manager.go:414] "containerMap: removing container" podUID="3455d983-b11b-4cee-bfce-d1f399cf9679" containerName="main"
Oct 30 01:16:25.523641 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:25.523616    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3455d983-b11b-4cee-bfce-d1f399cf9679" containerName="main"
Oct 30 01:16:25.523641 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:25.523622    5002 cpu_manager.go:414] "containerMap: removing container" podUID="44d5bcd2-3029-4d1d-b163-f90ee946d53b" containerName="revoke-keys"
Oct 30 01:16:25.523641 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:25.523627    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44d5bcd2-3029-4d1d-b163-f90ee946d53b" containerName="revoke-keys"
Oct 30 01:16:25.523900 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:25.523757    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="659c1425-5b67-4d01-bfc2-f1ba56b9a59b" containerName="main"
Oct 30 01:16:25.523900 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:25.523773    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea1850d9-abba-4977-a540-f6e0eaefb17d" containerName="main"
Oct 30 01:16:25.523900 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:25.523780    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="3455d983-b11b-4cee-bfce-d1f399cf9679" containerName="main"
Oct 30 01:16:25.523900 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:25.523786    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="44d5bcd2-3029-4d1d-b163-f90ee946d53b" containerName="revoke-keys"
Oct 30 01:16:25.529228 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:25.529202    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-9a8001-openshift-default-d885f8f7f-dt8r5"
Oct 30 01:16:25.556460 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:25.556317    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-9a8001-openshift-default-d885f8f7f-dt8r5"]
Oct 30 01:16:25.564196 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:25.564149    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-9a8001-openshift-default-dockercfg-lkxm8\""
Oct 30 01:16:25.650416 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:25.647444    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/423cd7a9-8a7c-4b26-b7c2-ba0e4053c042-istio-token\") pod \"e2e-ipp-b-9a8001-openshift-default-d885f8f7f-dt8r5\" (UID: \"423cd7a9-8a7c-4b26-b7c2-ba0e4053c042\") " pod="openshift-ingress/e2e-ipp-b-9a8001-openshift-default-d885f8f7f-dt8r5"
Oct 30 01:16:25.650416 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:25.647553    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rsb4\" (UniqueName: \"kubernetes.io/projected/423cd7a9-8a7c-4b26-b7c2-ba0e4053c042-kube-api-access-5rsb4\") pod \"e2e-ipp-b-9a8001-openshift-default-d885f8f7f-dt8r5\" (UID: \"423cd7a9-8a7c-4b26-b7c2-ba0e4053c042\") " pod="openshift-ingress/e2e-ipp-b-9a8001-openshift-default-d885f8f7f-dt8r5"
Oct 30 01:16:25.650416 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:25.647629    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/423cd7a9-8a7c-4b26-b7c2-ba0e4053c042-istio-podinfo\") pod \"e2e-ipp-b-9a8001-openshift-default-d885f8f7f-dt8r5\" (UID: \"423cd7a9-8a7c-4b26-b7c2-ba0e4053c042\") " pod="openshift-ingress/e2e-ipp-b-9a8001-openshift-default-d885f8f7f-dt8r5"
Oct 30 01:16:25.650416 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:25.647724    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/423cd7a9-8a7c-4b26-b7c2-ba0e4053c042-istio-data\") pod \"e2e-ipp-b-9a8001-openshift-default-d885f8f7f-dt8r5\" (UID: \"423cd7a9-8a7c-4b26-b7c2-ba0e4053c042\") " pod="openshift-ingress/e2e-ipp-b-9a8001-openshift-default-d885f8f7f-dt8r5"
Oct 30 01:16:25.650416 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:25.647759    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/423cd7a9-8a7c-4b26-b7c2-ba0e4053c042-workload-socket\") pod \"e2e-ipp-b-9a8001-openshift-default-d885f8f7f-dt8r5\" (UID: \"423cd7a9-8a7c-4b26-b7c2-ba0e4053c042\") " pod="openshift-ingress/e2e-ipp-b-9a8001-openshift-default-d885f8f7f-dt8r5"
Oct 30 01:16:25.650416 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:25.647803    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/423cd7a9-8a7c-4b26-b7c2-ba0e4053c042-istio-envoy\") pod \"e2e-ipp-b-9a8001-openshift-default-d885f8f7f-dt8r5\" (UID: \"423cd7a9-8a7c-4b26-b7c2-ba0e4053c042\") " pod="openshift-ingress/e2e-ipp-b-9a8001-openshift-default-d885f8f7f-dt8r5"
Oct 30 01:16:25.650416 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:25.647868    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/423cd7a9-8a7c-4b26-b7c2-ba0e4053c042-credential-socket\") pod \"e2e-ipp-b-9a8001-openshift-default-d885f8f7f-dt8r5\" (UID: \"423cd7a9-8a7c-4b26-b7c2-ba0e4053c042\") " pod="openshift-ingress/e2e-ipp-b-9a8001-openshift-default-d885f8f7f-dt8r5"
Oct 30 01:16:25.650416 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:25.647935    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/423cd7a9-8a7c-4b26-b7c2-ba0e4053c042-istiod-ca-cert\") pod \"e2e-ipp-b-9a8001-openshift-default-d885f8f7f-dt8r5\" (UID: \"423cd7a9-8a7c-4b26-b7c2-ba0e4053c042\") " pod="openshift-ingress/e2e-ipp-b-9a8001-openshift-default-d885f8f7f-dt8r5"
Oct 30 01:16:25.650416 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:25.648042    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/423cd7a9-8a7c-4b26-b7c2-ba0e4053c042-workload-certs\") pod \"e2e-ipp-b-9a8001-openshift-default-d885f8f7f-dt8r5\" (UID: \"423cd7a9-8a7c-4b26-b7c2-ba0e4053c042\") " pod="openshift-ingress/e2e-ipp-b-9a8001-openshift-default-d885f8f7f-dt8r5"
Oct 30 01:16:25.752405 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:25.752271    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rsb4\" (UniqueName: \"kubernetes.io/projected/423cd7a9-8a7c-4b26-b7c2-ba0e4053c042-kube-api-access-5rsb4\") pod \"e2e-ipp-b-9a8001-openshift-default-d885f8f7f-dt8r5\" (UID: \"423cd7a9-8a7c-4b26-b7c2-ba0e4053c042\") " pod="openshift-ingress/e2e-ipp-b-9a8001-openshift-default-d885f8f7f-dt8r5"
Oct 30 01:16:25.752405 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:25.752360    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/423cd7a9-8a7c-4b26-b7c2-ba0e4053c042-istio-podinfo\") pod \"e2e-ipp-b-9a8001-openshift-default-d885f8f7f-dt8r5\" (UID: \"423cd7a9-8a7c-4b26-b7c2-ba0e4053c042\") " pod="openshift-ingress/e2e-ipp-b-9a8001-openshift-default-d885f8f7f-dt8r5"
Oct 30 01:16:25.752668 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:25.752465    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/423cd7a9-8a7c-4b26-b7c2-ba0e4053c042-istio-data\") pod \"e2e-ipp-b-9a8001-openshift-default-d885f8f7f-dt8r5\" (UID: \"423cd7a9-8a7c-4b26-b7c2-ba0e4053c042\") " pod="openshift-ingress/e2e-ipp-b-9a8001-openshift-default-d885f8f7f-dt8r5"
Oct 30 01:16:25.752668 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:25.752494    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/423cd7a9-8a7c-4b26-b7c2-ba0e4053c042-workload-socket\") pod \"e2e-ipp-b-9a8001-openshift-default-d885f8f7f-dt8r5\" (UID: \"423cd7a9-8a7c-4b26-b7c2-ba0e4053c042\") " pod="openshift-ingress/e2e-ipp-b-9a8001-openshift-default-d885f8f7f-dt8r5"
Oct 30 01:16:25.752668 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:25.752536    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/423cd7a9-8a7c-4b26-b7c2-ba0e4053c042-istio-envoy\") pod \"e2e-ipp-b-9a8001-openshift-default-d885f8f7f-dt8r5\" (UID: \"423cd7a9-8a7c-4b26-b7c2-ba0e4053c042\") " pod="openshift-ingress/e2e-ipp-b-9a8001-openshift-default-d885f8f7f-dt8r5"
Oct 30 01:16:25.752668 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:25.752596    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/423cd7a9-8a7c-4b26-b7c2-ba0e4053c042-credential-socket\") pod \"e2e-ipp-b-9a8001-openshift-default-d885f8f7f-dt8r5\" (UID: \"423cd7a9-8a7c-4b26-b7c2-ba0e4053c042\") " pod="openshift-ingress/e2e-ipp-b-9a8001-openshift-default-d885f8f7f-dt8r5"
Oct 30 01:16:25.752668 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:25.752637    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/423cd7a9-8a7c-4b26-b7c2-ba0e4053c042-istiod-ca-cert\") pod \"e2e-ipp-b-9a8001-openshift-default-d885f8f7f-dt8r5\" (UID: \"423cd7a9-8a7c-4b26-b7c2-ba0e4053c042\") " pod="openshift-ingress/e2e-ipp-b-9a8001-openshift-default-d885f8f7f-dt8r5"
Oct 30 01:16:25.752899 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:25.752710    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/423cd7a9-8a7c-4b26-b7c2-ba0e4053c042-workload-certs\") pod \"e2e-ipp-b-9a8001-openshift-default-d885f8f7f-dt8r5\" (UID: \"423cd7a9-8a7c-4b26-b7c2-ba0e4053c042\") " pod="openshift-ingress/e2e-ipp-b-9a8001-openshift-default-d885f8f7f-dt8r5"
Oct 30 01:16:25.752899 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:25.752801    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/423cd7a9-8a7c-4b26-b7c2-ba0e4053c042-istio-token\") pod \"e2e-ipp-b-9a8001-openshift-default-d885f8f7f-dt8r5\" (UID: \"423cd7a9-8a7c-4b26-b7c2-ba0e4053c042\") " pod="openshift-ingress/e2e-ipp-b-9a8001-openshift-default-d885f8f7f-dt8r5"
Oct 30 01:16:25.762403 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:25.753272    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/423cd7a9-8a7c-4b26-b7c2-ba0e4053c042-workload-socket\") pod \"e2e-ipp-b-9a8001-openshift-default-d885f8f7f-dt8r5\" (UID: \"423cd7a9-8a7c-4b26-b7c2-ba0e4053c042\") " pod="openshift-ingress/e2e-ipp-b-9a8001-openshift-default-d885f8f7f-dt8r5"
Oct 30 01:16:25.762403 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:25.754698    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/423cd7a9-8a7c-4b26-b7c2-ba0e4053c042-credential-socket\") pod \"e2e-ipp-b-9a8001-openshift-default-d885f8f7f-dt8r5\" (UID: \"423cd7a9-8a7c-4b26-b7c2-ba0e4053c042\") " pod="openshift-ingress/e2e-ipp-b-9a8001-openshift-default-d885f8f7f-dt8r5"
Oct 30 01:16:25.762403 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:25.754910    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/423cd7a9-8a7c-4b26-b7c2-ba0e4053c042-workload-certs\") pod \"e2e-ipp-b-9a8001-openshift-default-d885f8f7f-dt8r5\" (UID: \"423cd7a9-8a7c-4b26-b7c2-ba0e4053c042\") " pod="openshift-ingress/e2e-ipp-b-9a8001-openshift-default-d885f8f7f-dt8r5"
Oct 30 01:16:25.762403 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:25.755118    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/423cd7a9-8a7c-4b26-b7c2-ba0e4053c042-istio-data\") pod \"e2e-ipp-b-9a8001-openshift-default-d885f8f7f-dt8r5\" (UID: \"423cd7a9-8a7c-4b26-b7c2-ba0e4053c042\") " pod="openshift-ingress/e2e-ipp-b-9a8001-openshift-default-d885f8f7f-dt8r5"
Oct 30 01:16:25.762403 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:25.755164    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/423cd7a9-8a7c-4b26-b7c2-ba0e4053c042-istiod-ca-cert\") pod \"e2e-ipp-b-9a8001-openshift-default-d885f8f7f-dt8r5\" (UID: \"423cd7a9-8a7c-4b26-b7c2-ba0e4053c042\") " pod="openshift-ingress/e2e-ipp-b-9a8001-openshift-default-d885f8f7f-dt8r5"
Oct 30 01:16:25.817456 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:25.801065    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/423cd7a9-8a7c-4b26-b7c2-ba0e4053c042-istio-podinfo\") pod \"e2e-ipp-b-9a8001-openshift-default-d885f8f7f-dt8r5\" (UID: \"423cd7a9-8a7c-4b26-b7c2-ba0e4053c042\") " pod="openshift-ingress/e2e-ipp-b-9a8001-openshift-default-d885f8f7f-dt8r5"
Oct 30 01:16:25.817456 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:25.801574    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/423cd7a9-8a7c-4b26-b7c2-ba0e4053c042-istio-token\") pod \"e2e-ipp-b-9a8001-openshift-default-d885f8f7f-dt8r5\" (UID: \"423cd7a9-8a7c-4b26-b7c2-ba0e4053c042\") " pod="openshift-ingress/e2e-ipp-b-9a8001-openshift-default-d885f8f7f-dt8r5"
Oct 30 01:16:25.817456 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:25.802272    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rsb4\" (UniqueName: \"kubernetes.io/projected/423cd7a9-8a7c-4b26-b7c2-ba0e4053c042-kube-api-access-5rsb4\") pod \"e2e-ipp-b-9a8001-openshift-default-d885f8f7f-dt8r5\" (UID: \"423cd7a9-8a7c-4b26-b7c2-ba0e4053c042\") " pod="openshift-ingress/e2e-ipp-b-9a8001-openshift-default-d885f8f7f-dt8r5"
Oct 30 01:16:25.817456 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:25.802588    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/423cd7a9-8a7c-4b26-b7c2-ba0e4053c042-istio-envoy\") pod \"e2e-ipp-b-9a8001-openshift-default-d885f8f7f-dt8r5\" (UID: \"423cd7a9-8a7c-4b26-b7c2-ba0e4053c042\") " pod="openshift-ingress/e2e-ipp-b-9a8001-openshift-default-d885f8f7f-dt8r5"
Oct 30 01:16:25.859405 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:25.850776    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-9a8001-openshift-default-d885f8f7f-dt8r5"
Oct 30 01:16:26.263313 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:26.263219    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-9a8001-openshift-default-d885f8f7f-dt8r5"]
Oct 30 01:16:26.263789 ip-10-0-128-7 kubenswrapper[5002]: W1030 01:16:26.263540    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod423cd7a9_8a7c_4b26_b7c2_ba0e4053c042.slice/crio-f0c81b38a38643a44df636e1313ac85aac2babb0ed20be04ad28707cc73aae67 WatchSource:0}: Error finding container f0c81b38a38643a44df636e1313ac85aac2babb0ed20be04ad28707cc73aae67: Status 404 returned error can't find the container with id f0c81b38a38643a44df636e1313ac85aac2babb0ed20be04ad28707cc73aae67
Oct 30 01:16:26.266847 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:26.266803    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:16:26.266974 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:26.266897    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:16:26.266974 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:26.266947    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:16:26.501205 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:26.501169    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d930a049-cee8-4106-a988-0032fcaf229e" path="/var/lib/kubelet/pods/d930a049-cee8-4106-a988-0032fcaf229e/volumes"
Oct 30 01:16:26.952896 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:26.952853    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-9a8001-openshift-default-d885f8f7f-dt8r5" event={"ID":"423cd7a9-8a7c-4b26-b7c2-ba0e4053c042","Type":"ContainerStarted","Data":"f0c46b72fed307bda296524757f0270115e4ab84456c2250aa124fbb87458199"}
Oct 30 01:16:26.953096 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:26.952926    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-9a8001-openshift-default-d885f8f7f-dt8r5" event={"ID":"423cd7a9-8a7c-4b26-b7c2-ba0e4053c042","Type":"ContainerStarted","Data":"f0c81b38a38643a44df636e1313ac85aac2babb0ed20be04ad28707cc73aae67"}
Oct 30 01:16:27.018470 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:27.018413    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-9a8001-openshift-default-d885f8f7f-dt8r5" podStartSLOduration=2.018370703 podStartE2EDuration="2.018370703s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:16:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:16:27.014580816 +0000 UTC m=+2682.631013695" watchObservedRunningTime="2026-10-30 01:16:27.018370703 +0000 UTC m=+2682.634803581"
Oct 30 01:16:27.856313 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:27.856283    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-9a8001-openshift-default-d885f8f7f-dt8r5"
Oct 30 01:16:27.866269 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:27.866222    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-9a8001-openshift-default-d885f8f7f-dt8r5"
Oct 30 01:16:27.961745 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:27.961714    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-9a8001-openshift-default-d885f8f7f-dt8r5"
Oct 30 01:16:27.962916 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:27.962889    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-9a8001-openshift-default-d885f8f7f-dt8r5"
Oct 30 01:16:28.502307 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:28.502273    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-f64dc0/auth-test-model-f64dc0-kserve-7d44468c6f-xf5xp"]
Oct 30 01:16:30.635234 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:30.635198    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-8s7j6"]
Oct 30 01:16:30.635762 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:30.635471    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-8s7j6" podUID="70b897af-fd89-4e91-b0f4-e6adff33cfdd" containerName="istio-proxy" containerID="cri-o://a60c000933a68b8e5f964cea4522e277d96b81275d5d99ed30aa5453c39c37b0" gracePeriod=30
Oct 30 01:16:31.381416 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:31.381367    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-cce513-plbvz"]
Oct 30 01:16:31.387047 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:31.387026    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-cce513-plbvz"
Oct 30 01:16:31.408447 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:31.408409    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-cce513-plbvz"]
Oct 30 01:16:31.416109 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:31.416081    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:16:31.416313 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:31.416108    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:16:31.416586 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:31.416561    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5phsh\""
Oct 30 01:16:31.418021 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:31.417986    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:16:31.494591 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:31.494558    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d22b0df-6481-4817-b64d-88466a69bceb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-cce513-plbvz\" (UID: \"3d22b0df-6481-4817-b64d-88466a69bceb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-cce513-plbvz"
Oct 30 01:16:31.494752 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:31.494638    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfm8x\" (UniqueName: \"kubernetes.io/projected/3d22b0df-6481-4817-b64d-88466a69bceb-kube-api-access-bfm8x\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-cce513-plbvz\" (UID: \"3d22b0df-6481-4817-b64d-88466a69bceb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-cce513-plbvz"
Oct 30 01:16:31.494752 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:31.494739    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d22b0df-6481-4817-b64d-88466a69bceb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-cce513-plbvz\" (UID: \"3d22b0df-6481-4817-b64d-88466a69bceb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-cce513-plbvz"
Oct 30 01:16:31.615406 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:31.602373    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d22b0df-6481-4817-b64d-88466a69bceb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-cce513-plbvz\" (UID: \"3d22b0df-6481-4817-b64d-88466a69bceb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-cce513-plbvz"
Oct 30 01:16:31.663257 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:31.663162    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfm8x\" (UniqueName: \"kubernetes.io/projected/3d22b0df-6481-4817-b64d-88466a69bceb-kube-api-access-bfm8x\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-cce513-plbvz\" (UID: \"3d22b0df-6481-4817-b64d-88466a69bceb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-cce513-plbvz"
Oct 30 01:16:31.684674 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:31.684633    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d22b0df-6481-4817-b64d-88466a69bceb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-cce513-plbvz\" (UID: \"3d22b0df-6481-4817-b64d-88466a69bceb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-cce513-plbvz"
Oct 30 01:16:31.740413 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:31.616479    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d22b0df-6481-4817-b64d-88466a69bceb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-cce513-plbvz\" (UID: \"3d22b0df-6481-4817-b64d-88466a69bceb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-cce513-plbvz"
Oct 30 01:16:31.771169 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:31.771048    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d22b0df-6481-4817-b64d-88466a69bceb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-cce513-plbvz\" (UID: \"3d22b0df-6481-4817-b64d-88466a69bceb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-cce513-plbvz"
Oct 30 01:16:31.805745 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:31.783104    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfm8x\" (UniqueName: \"kubernetes.io/projected/3d22b0df-6481-4817-b64d-88466a69bceb-kube-api-access-bfm8x\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-cce513-plbvz\" (UID: \"3d22b0df-6481-4817-b64d-88466a69bceb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-cce513-plbvz"
Oct 30 01:16:32.004520 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:32.004486    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-cce513-plbvz"
Oct 30 01:16:32.371555 ip-10-0-128-7 kubenswrapper[5002]: W1030 01:16:32.371519    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d22b0df_6481_4817_b64d_88466a69bceb.slice/crio-aca06918efdbea42b159bc51cd35f2cfa563141cf1b6d120e39d060bb0c56c0a WatchSource:0}: Error finding container aca06918efdbea42b159bc51cd35f2cfa563141cf1b6d120e39d060bb0c56c0a: Status 404 returned error can't find the container with id aca06918efdbea42b159bc51cd35f2cfa563141cf1b6d120e39d060bb0c56c0a
Oct 30 01:16:32.372597 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:32.372563    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-cce513-plbvz"]
Oct 30 01:16:33.011238 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:33.011192    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-cce513-plbvz" event={"ID":"3d22b0df-6481-4817-b64d-88466a69bceb","Type":"ContainerStarted","Data":"341ca5dd074ba6de591068dce98d9ffe848e8250a9c42b3f56211db52426d38e"}
Oct 30 01:16:33.011757 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:33.011300    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-cce513-plbvz" event={"ID":"3d22b0df-6481-4817-b64d-88466a69bceb","Type":"ContainerStarted","Data":"aca06918efdbea42b159bc51cd35f2cfa563141cf1b6d120e39d060bb0c56c0a"}
Oct 30 01:16:33.055810 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:33.055756    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-cce513-plbvz" podStartSLOduration=2.055742734 podStartE2EDuration="2.055742734s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:16:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:16:33.054066739 +0000 UTC m=+2688.670499619" watchObservedRunningTime="2026-10-30 01:16:33.055742734 +0000 UTC m=+2688.672175613"
Oct 30 01:16:34.020678 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:34.020642    5002 generic.go:356] "Generic (PLEG): container finished" podID="3d22b0df-6481-4817-b64d-88466a69bceb" containerID="341ca5dd074ba6de591068dce98d9ffe848e8250a9c42b3f56211db52426d38e" exitCode=0
Oct 30 01:16:34.021173 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:34.020732    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-cce513-plbvz" event={"ID":"3d22b0df-6481-4817-b64d-88466a69bceb","Type":"ContainerDied","Data":"341ca5dd074ba6de591068dce98d9ffe848e8250a9c42b3f56211db52426d38e"}
Oct 30 01:16:35.239290 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:35.239263    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-cce513-plbvz"
Oct 30 01:16:35.377227 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:35.377148    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d22b0df-6481-4817-b64d-88466a69bceb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d22b0df-6481-4817-b64d-88466a69bceb-maas-api-openshift-ca\") pod \"3d22b0df-6481-4817-b64d-88466a69bceb\" (UID: \"3d22b0df-6481-4817-b64d-88466a69bceb\") "
Oct 30 01:16:35.377227 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:35.377212    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3d22b0df-6481-4817-b64d-88466a69bceb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d22b0df-6481-4817-b64d-88466a69bceb-maas-api-xks-ca\") pod \"3d22b0df-6481-4817-b64d-88466a69bceb\" (UID: \"3d22b0df-6481-4817-b64d-88466a69bceb\") "
Oct 30 01:16:35.377498 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:35.377250    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d22b0df-6481-4817-b64d-88466a69bceb-kube-api-access-bfm8x\" (UniqueName: \"kubernetes.io/projected/3d22b0df-6481-4817-b64d-88466a69bceb-kube-api-access-bfm8x\") pod \"3d22b0df-6481-4817-b64d-88466a69bceb\" (UID: \"3d22b0df-6481-4817-b64d-88466a69bceb\") "
Oct 30 01:16:35.377592 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:35.377562    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d22b0df-6481-4817-b64d-88466a69bceb-maas-api-openshift-ca" pod "3d22b0df-6481-4817-b64d-88466a69bceb" (UID: "3d22b0df-6481-4817-b64d-88466a69bceb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:16:35.377917 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:35.377896    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d22b0df-6481-4817-b64d-88466a69bceb-maas-api-openshift-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:16:35.380007 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:35.379980    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d22b0df-6481-4817-b64d-88466a69bceb-maas-api-xks-ca" pod "3d22b0df-6481-4817-b64d-88466a69bceb" (UID: "3d22b0df-6481-4817-b64d-88466a69bceb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:16:35.380556 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:35.380525    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d22b0df-6481-4817-b64d-88466a69bceb-kube-api-access-bfm8x" pod "3d22b0df-6481-4817-b64d-88466a69bceb" (UID: "3d22b0df-6481-4817-b64d-88466a69bceb"). InnerVolumeSpecName "kube-api-access-bfm8x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:16:35.479070 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:35.479027    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d22b0df-6481-4817-b64d-88466a69bceb-maas-api-xks-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:16:35.479070 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:35.479062    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bfm8x\" (UniqueName: \"kubernetes.io/projected/3d22b0df-6481-4817-b64d-88466a69bceb-kube-api-access-bfm8x\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:16:35.915700 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:35.915669    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-8s7j6"
Oct 30 01:16:36.036196 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:36.036162    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4b8027ba-bz2kg"]
Oct 30 01:16:36.040978 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:36.040950    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4b8027ba-bz2kg"]
Oct 30 01:16:36.042183 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:36.042153    5002 generic.go:356] "Generic (PLEG): container finished" podID="70b897af-fd89-4e91-b0f4-e6adff33cfdd" containerID="a60c000933a68b8e5f964cea4522e277d96b81275d5d99ed30aa5453c39c37b0" exitCode=0
Oct 30 01:16:36.042335 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:36.042206    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-8s7j6"
Oct 30 01:16:36.042335 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:36.042239    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-8s7j6" event={"ID":"70b897af-fd89-4e91-b0f4-e6adff33cfdd","Type":"ContainerDied","Data":"a60c000933a68b8e5f964cea4522e277d96b81275d5d99ed30aa5453c39c37b0"}
Oct 30 01:16:36.042335 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:36.042308    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-8s7j6" event={"ID":"70b897af-fd89-4e91-b0f4-e6adff33cfdd","Type":"ContainerDied","Data":"e1563650df956a017d0166dafa3833fc87751dc6a52719e44956d050e9380571"}
Oct 30 01:16:36.042554 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:36.042345    5002 scope.go:122] "RemoveContainer" containerID="a60c000933a68b8e5f964cea4522e277d96b81275d5d99ed30aa5453c39c37b0"
Oct 30 01:16:36.044320 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:36.044299    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-cce513-plbvz" event={"ID":"3d22b0df-6481-4817-b64d-88466a69bceb","Type":"ContainerDied","Data":"aca06918efdbea42b159bc51cd35f2cfa563141cf1b6d120e39d060bb0c56c0a"}
Oct 30 01:16:36.044461 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:36.044340    5002 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aca06918efdbea42b159bc51cd35f2cfa563141cf1b6d120e39d060bb0c56c0a"
Oct 30 01:16:36.044461 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:36.044411    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-cce513-plbvz"
Oct 30 01:16:36.063219 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:36.063198    5002 scope.go:122] "RemoveContainer" containerID="a60c000933a68b8e5f964cea4522e277d96b81275d5d99ed30aa5453c39c37b0"
Oct 30 01:16:36.063901 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:16:36.063873    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a60c000933a68b8e5f964cea4522e277d96b81275d5d99ed30aa5453c39c37b0\": container with ID starting with a60c000933a68b8e5f964cea4522e277d96b81275d5d99ed30aa5453c39c37b0 not found: ID does not exist" containerID="a60c000933a68b8e5f964cea4522e277d96b81275d5d99ed30aa5453c39c37b0"
Oct 30 01:16:36.064013 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:36.063908    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a60c000933a68b8e5f964cea4522e277d96b81275d5d99ed30aa5453c39c37b0"} err="failed to get container status \"a60c000933a68b8e5f964cea4522e277d96b81275d5d99ed30aa5453c39c37b0\": rpc error: code = NotFound desc = could not find container \"a60c000933a68b8e5f964cea4522e277d96b81275d5d99ed30aa5453c39c37b0\": container with ID starting with a60c000933a68b8e5f964cea4522e277d96b81275d5d99ed30aa5453c39c37b0 not found: ID does not exist"
Oct 30 01:16:36.088129 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:36.088092    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70b897af-fd89-4e91-b0f4-e6adff33cfdd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70b897af-fd89-4e91-b0f4-e6adff33cfdd-istio-data\") pod \"70b897af-fd89-4e91-b0f4-e6adff33cfdd\" (UID: \"70b897af-fd89-4e91-b0f4-e6adff33cfdd\") "
Oct 30 01:16:36.088308 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:36.088181    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70b897af-fd89-4e91-b0f4-e6adff33cfdd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70b897af-fd89-4e91-b0f4-e6adff33cfdd-istio-envoy\") pod \"70b897af-fd89-4e91-b0f4-e6adff33cfdd\" (UID: \"70b897af-fd89-4e91-b0f4-e6adff33cfdd\") "
Oct 30 01:16:36.088308 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:36.088245    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70b897af-fd89-4e91-b0f4-e6adff33cfdd-istio-token\" (UniqueName: \"kubernetes.io/projected/70b897af-fd89-4e91-b0f4-e6adff33cfdd-istio-token\") pod \"70b897af-fd89-4e91-b0f4-e6adff33cfdd\" (UID: \"70b897af-fd89-4e91-b0f4-e6adff33cfdd\") "
Oct 30 01:16:36.088492 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:36.088342    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/70b897af-fd89-4e91-b0f4-e6adff33cfdd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70b897af-fd89-4e91-b0f4-e6adff33cfdd-istio-podinfo\") pod \"70b897af-fd89-4e91-b0f4-e6adff33cfdd\" (UID: \"70b897af-fd89-4e91-b0f4-e6adff33cfdd\") "
Oct 30 01:16:36.088492 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:36.088396    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70b897af-fd89-4e91-b0f4-e6adff33cfdd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70b897af-fd89-4e91-b0f4-e6adff33cfdd-credential-socket\") pod \"70b897af-fd89-4e91-b0f4-e6adff33cfdd\" (UID: \"70b897af-fd89-4e91-b0f4-e6adff33cfdd\") "
Oct 30 01:16:36.088492 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:36.088405    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70b897af-fd89-4e91-b0f4-e6adff33cfdd-istio-data" pod "70b897af-fd89-4e91-b0f4-e6adff33cfdd" (UID: "70b897af-fd89-4e91-b0f4-e6adff33cfdd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:16:36.088492 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:36.088426    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70b897af-fd89-4e91-b0f4-e6adff33cfdd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70b897af-fd89-4e91-b0f4-e6adff33cfdd-workload-certs\") pod \"70b897af-fd89-4e91-b0f4-e6adff33cfdd\" (UID: \"70b897af-fd89-4e91-b0f4-e6adff33cfdd\") "
Oct 30 01:16:36.088492 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:36.088466    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70b897af-fd89-4e91-b0f4-e6adff33cfdd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70b897af-fd89-4e91-b0f4-e6adff33cfdd-workload-socket\") pod \"70b897af-fd89-4e91-b0f4-e6adff33cfdd\" (UID: \"70b897af-fd89-4e91-b0f4-e6adff33cfdd\") "
Oct 30 01:16:36.088772 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:36.088579    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70b897af-fd89-4e91-b0f4-e6adff33cfdd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70b897af-fd89-4e91-b0f4-e6adff33cfdd-istiod-ca-cert\") pod \"70b897af-fd89-4e91-b0f4-e6adff33cfdd\" (UID: \"70b897af-fd89-4e91-b0f4-e6adff33cfdd\") "
Oct 30 01:16:36.088772 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:36.088679    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70b897af-fd89-4e91-b0f4-e6adff33cfdd-kube-api-access-dkb6b\" (UniqueName: \"kubernetes.io/projected/70b897af-fd89-4e91-b0f4-e6adff33cfdd-kube-api-access-dkb6b\") pod \"70b897af-fd89-4e91-b0f4-e6adff33cfdd\" (UID: \"70b897af-fd89-4e91-b0f4-e6adff33cfdd\") "
Oct 30 01:16:36.088886 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:36.088811    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70b897af-fd89-4e91-b0f4-e6adff33cfdd-workload-certs" pod "70b897af-fd89-4e91-b0f4-e6adff33cfdd" (UID: "70b897af-fd89-4e91-b0f4-e6adff33cfdd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:16:36.088990 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:36.088971    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70b897af-fd89-4e91-b0f4-e6adff33cfdd-credential-socket" pod "70b897af-fd89-4e91-b0f4-e6adff33cfdd" (UID: "70b897af-fd89-4e91-b0f4-e6adff33cfdd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:16:36.089151 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:36.089133    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70b897af-fd89-4e91-b0f4-e6adff33cfdd-workload-socket" pod "70b897af-fd89-4e91-b0f4-e6adff33cfdd" (UID: "70b897af-fd89-4e91-b0f4-e6adff33cfdd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:16:36.089452 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:36.089422    5002 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70b897af-fd89-4e91-b0f4-e6adff33cfdd-istio-data\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:16:36.089452 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:36.089441    5002 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70b897af-fd89-4e91-b0f4-e6adff33cfdd-credential-socket\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:16:36.089452 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:36.089452    5002 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70b897af-fd89-4e91-b0f4-e6adff33cfdd-workload-certs\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:16:36.089721 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:36.089657    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70b897af-fd89-4e91-b0f4-e6adff33cfdd-istiod-ca-cert" pod "70b897af-fd89-4e91-b0f4-e6adff33cfdd" (UID: "70b897af-fd89-4e91-b0f4-e6adff33cfdd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:16:36.090233 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:36.089866    5002 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70b897af-fd89-4e91-b0f4-e6adff33cfdd-workload-socket\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:16:36.091065 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:36.091034    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70b897af-fd89-4e91-b0f4-e6adff33cfdd-istio-envoy" pod "70b897af-fd89-4e91-b0f4-e6adff33cfdd" (UID: "70b897af-fd89-4e91-b0f4-e6adff33cfdd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:16:36.091983 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:36.091953    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70b897af-fd89-4e91-b0f4-e6adff33cfdd-kube-api-access-dkb6b" pod "70b897af-fd89-4e91-b0f4-e6adff33cfdd" (UID: "70b897af-fd89-4e91-b0f4-e6adff33cfdd"). InnerVolumeSpecName "kube-api-access-dkb6b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:16:36.092269 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:36.092242    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/70b897af-fd89-4e91-b0f4-e6adff33cfdd-istio-podinfo" pod "70b897af-fd89-4e91-b0f4-e6adff33cfdd" (UID: "70b897af-fd89-4e91-b0f4-e6adff33cfdd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:16:36.092269 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:36.092243    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70b897af-fd89-4e91-b0f4-e6adff33cfdd-istio-token" pod "70b897af-fd89-4e91-b0f4-e6adff33cfdd" (UID: "70b897af-fd89-4e91-b0f4-e6adff33cfdd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:16:36.191799 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:36.191748    5002 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70b897af-fd89-4e91-b0f4-e6adff33cfdd-istio-envoy\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:16:36.191799 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:36.191787    5002 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70b897af-fd89-4e91-b0f4-e6adff33cfdd-istio-token\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:16:36.191799 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:36.191802    5002 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70b897af-fd89-4e91-b0f4-e6adff33cfdd-istio-podinfo\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:16:36.192073 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:36.191814    5002 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70b897af-fd89-4e91-b0f4-e6adff33cfdd-istiod-ca-cert\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:16:36.192073 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:36.191825    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dkb6b\" (UniqueName: \"kubernetes.io/projected/70b897af-fd89-4e91-b0f4-e6adff33cfdd-kube-api-access-dkb6b\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:16:36.399852 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:36.399803    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-8s7j6"]
Oct 30 01:16:36.404696 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:36.404667    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-8s7j6"]
Oct 30 01:16:36.497206 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:36.497115    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70b897af-fd89-4e91-b0f4-e6adff33cfdd" path="/var/lib/kubelet/pods/70b897af-fd89-4e91-b0f4-e6adff33cfdd/volumes"
Oct 30 01:16:36.497793 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:36.497767    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf6bfe10-0128-4cb2-93f6-1eb9eed7f8ad" path="/var/lib/kubelet/pods/bf6bfe10-0128-4cb2-93f6-1eb9eed7f8ad/volumes"
Oct 30 01:16:42.868418 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:16:42.868353    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbjdq" containerName="extractor"
Oct 30 01:16:42.879839 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:42.879806    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9cff2b63-94bdm"]
Oct 30 01:16:42.880424 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:42.880410    5002 cpu_manager.go:414] "containerMap: removing container" podUID="70b897af-fd89-4e91-b0f4-e6adff33cfdd" containerName="istio-proxy"
Oct 30 01:16:42.880474 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:42.880426    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70b897af-fd89-4e91-b0f4-e6adff33cfdd" containerName="istio-proxy"
Oct 30 01:16:42.880474 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:42.880471    5002 cpu_manager.go:414] "containerMap: removing container" podUID="3d22b0df-6481-4817-b64d-88466a69bceb" containerName="revoke-keys"
Oct 30 01:16:42.880542 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:42.880477    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d22b0df-6481-4817-b64d-88466a69bceb" containerName="revoke-keys"
Oct 30 01:16:42.880579 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:42.880569    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="70b897af-fd89-4e91-b0f4-e6adff33cfdd" containerName="istio-proxy"
Oct 30 01:16:42.880614 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:42.880586    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d22b0df-6481-4817-b64d-88466a69bceb" containerName="revoke-keys"
Oct 30 01:16:42.884086 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:42.884067    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cff2b63-94bdm"
Oct 30 01:16:42.904491 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:42.904459    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa210800-6sttw"]
Oct 30 01:16:42.908853 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:42.908827    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9cff2b63-94bdm"]
Oct 30 01:16:42.909013 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:42.908929    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa210800-6sttw"
Oct 30 01:16:42.920427 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:42.913270    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:16:42.920427 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:42.913276    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:16:42.920427 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:42.913338    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:16:42.920427 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:42.913540    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5phsh\""
Oct 30 01:16:42.929550 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:42.929516    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa210800-6sttw"]
Oct 30 01:16:42.937646 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:42.937610    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d8a608f-20c9-4568-9135-4d6e9ef23c97-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9cff2b63-94bdm\" (UID: \"0d8a608f-20c9-4568-9135-4d6e9ef23c97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cff2b63-94bdm"
Oct 30 01:16:42.937817 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:42.937735    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5p7p\" (UniqueName: \"kubernetes.io/projected/22a7542d-4dcf-4059-b4b6-84f6b33c4b6a-kube-api-access-b5p7p\") pod \"maas-api-revoke-sub-fa210800-6sttw\" (UID: \"22a7542d-4dcf-4059-b4b6-84f6b33c4b6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa210800-6sttw"
Oct 30 01:16:42.938071 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:42.937819    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22a7542d-4dcf-4059-b4b6-84f6b33c4b6a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fa210800-6sttw\" (UID: \"22a7542d-4dcf-4059-b4b6-84f6b33c4b6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa210800-6sttw"
Oct 30 01:16:42.938071 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:42.937989    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d8a608f-20c9-4568-9135-4d6e9ef23c97-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9cff2b63-94bdm\" (UID: \"0d8a608f-20c9-4568-9135-4d6e9ef23c97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cff2b63-94bdm"
Oct 30 01:16:42.938071 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:42.938047    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nskb9\" (UniqueName: \"kubernetes.io/projected/0d8a608f-20c9-4568-9135-4d6e9ef23c97-kube-api-access-nskb9\") pod \"maas-api-revoke-sub-9cff2b63-94bdm\" (UID: \"0d8a608f-20c9-4568-9135-4d6e9ef23c97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cff2b63-94bdm"
Oct 30 01:16:42.938305 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:42.938173    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22a7542d-4dcf-4059-b4b6-84f6b33c4b6a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fa210800-6sttw\" (UID: \"22a7542d-4dcf-4059-b4b6-84f6b33c4b6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa210800-6sttw"
Oct 30 01:16:43.134453 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:43.131152    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d8a608f-20c9-4568-9135-4d6e9ef23c97-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9cff2b63-94bdm\" (UID: \"0d8a608f-20c9-4568-9135-4d6e9ef23c97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cff2b63-94bdm"
Oct 30 01:16:43.134453 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:43.131229    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5p7p\" (UniqueName: \"kubernetes.io/projected/22a7542d-4dcf-4059-b4b6-84f6b33c4b6a-kube-api-access-b5p7p\") pod \"maas-api-revoke-sub-fa210800-6sttw\" (UID: \"22a7542d-4dcf-4059-b4b6-84f6b33c4b6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa210800-6sttw"
Oct 30 01:16:43.134453 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:43.131282    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22a7542d-4dcf-4059-b4b6-84f6b33c4b6a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fa210800-6sttw\" (UID: \"22a7542d-4dcf-4059-b4b6-84f6b33c4b6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa210800-6sttw"
Oct 30 01:16:43.134453 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:43.131369    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d8a608f-20c9-4568-9135-4d6e9ef23c97-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9cff2b63-94bdm\" (UID: \"0d8a608f-20c9-4568-9135-4d6e9ef23c97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cff2b63-94bdm"
Oct 30 01:16:43.134453 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:43.131442    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nskb9\" (UniqueName: \"kubernetes.io/projected/0d8a608f-20c9-4568-9135-4d6e9ef23c97-kube-api-access-nskb9\") pod \"maas-api-revoke-sub-9cff2b63-94bdm\" (UID: \"0d8a608f-20c9-4568-9135-4d6e9ef23c97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cff2b63-94bdm"
Oct 30 01:16:43.134453 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:43.131473    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22a7542d-4dcf-4059-b4b6-84f6b33c4b6a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fa210800-6sttw\" (UID: \"22a7542d-4dcf-4059-b4b6-84f6b33c4b6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa210800-6sttw"
Oct 30 01:16:43.157456 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:43.139261    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d8a608f-20c9-4568-9135-4d6e9ef23c97-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9cff2b63-94bdm\" (UID: \"0d8a608f-20c9-4568-9135-4d6e9ef23c97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cff2b63-94bdm"
Oct 30 01:16:43.193533 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:43.168636    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22a7542d-4dcf-4059-b4b6-84f6b33c4b6a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fa210800-6sttw\" (UID: \"22a7542d-4dcf-4059-b4b6-84f6b33c4b6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa210800-6sttw"
Oct 30 01:16:43.224013 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:43.200857    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d8a608f-20c9-4568-9135-4d6e9ef23c97-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9cff2b63-94bdm\" (UID: \"0d8a608f-20c9-4568-9135-4d6e9ef23c97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cff2b63-94bdm"
Oct 30 01:16:43.382406 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:43.372797    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nskb9\" (UniqueName: \"kubernetes.io/projected/0d8a608f-20c9-4568-9135-4d6e9ef23c97-kube-api-access-nskb9\") pod \"maas-api-revoke-sub-9cff2b63-94bdm\" (UID: \"0d8a608f-20c9-4568-9135-4d6e9ef23c97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cff2b63-94bdm"
Oct 30 01:16:43.417071 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:43.416994    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22a7542d-4dcf-4059-b4b6-84f6b33c4b6a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fa210800-6sttw\" (UID: \"22a7542d-4dcf-4059-b4b6-84f6b33c4b6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa210800-6sttw"
Oct 30 01:16:43.432689 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:43.431912    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5p7p\" (UniqueName: \"kubernetes.io/projected/22a7542d-4dcf-4059-b4b6-84f6b33c4b6a-kube-api-access-b5p7p\") pod \"maas-api-revoke-sub-fa210800-6sttw\" (UID: \"22a7542d-4dcf-4059-b4b6-84f6b33c4b6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa210800-6sttw"
Oct 30 01:16:43.499752 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:43.499710    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cff2b63-94bdm"
Oct 30 01:16:43.524636 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:43.524543    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w2-f64dc0/auth-test-model-f64dc0-kserve-7d44468c6f-xf5xp" podUID="54606b50-cba4-46d5-8bf4-12843a359919" containerName="main" containerID="cri-o://1f9cca611b5036823b7d809cfd1aca9bea737a048d756202a7774fbd149453ce" gracePeriod=45
Oct 30 01:16:43.524759 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:43.524679    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa210800-6sttw"
Oct 30 01:16:43.719299 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:43.718943    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w2-f64dc0/auth-test-model-f64dc0-kserve-7d44468c6f-xf5xp" podUID="54606b50-cba4-46d5-8bf4-12843a359919" containerName="main" probeResult="failure" output="Get \"https://10.132.0.211:8000/ready\": dial tcp 10.132.0.211:8000: connect: connection refused"
Oct 30 01:16:43.823859 ip-10-0-128-7 kubenswrapper[5002]: W1030 01:16:43.823816    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d8a608f_20c9_4568_9135_4d6e9ef23c97.slice/crio-10cdb212969cba65fef51ce336185d13fe8f1fcda77b89a5b93d04ddebed8b53 WatchSource:0}: Error finding container 10cdb212969cba65fef51ce336185d13fe8f1fcda77b89a5b93d04ddebed8b53: Status 404 returned error can't find the container with id 10cdb212969cba65fef51ce336185d13fe8f1fcda77b89a5b93d04ddebed8b53
Oct 30 01:16:43.826413 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:43.826370    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9cff2b63-94bdm"]
Oct 30 01:16:43.875960 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:43.875930    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-f64dc0/auth-test-model-f64dc0-kserve-7d44468c6f-xf5xp"
Oct 30 01:16:43.884977 ip-10-0-128-7 kubenswrapper[5002]: W1030 01:16:43.884928    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22a7542d_4dcf_4059_b4b6_84f6b33c4b6a.slice/crio-5273f3ed9e09bffb349d116a1a7d1bdb2919ded90e184da24e7833c55355678c WatchSource:0}: Error finding container 5273f3ed9e09bffb349d116a1a7d1bdb2919ded90e184da24e7833c55355678c: Status 404 returned error can't find the container with id 5273f3ed9e09bffb349d116a1a7d1bdb2919ded90e184da24e7833c55355678c
Oct 30 01:16:43.889923 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:43.889886    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa210800-6sttw"]
Oct 30 01:16:43.982345 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:43.982258    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54606b50-cba4-46d5-8bf4-12843a359919-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54606b50-cba4-46d5-8bf4-12843a359919-tmp-dir\") pod \"54606b50-cba4-46d5-8bf4-12843a359919\" (UID: \"54606b50-cba4-46d5-8bf4-12843a359919\") "
Oct 30 01:16:43.982588 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:43.982358    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54606b50-cba4-46d5-8bf4-12843a359919-kube-api-access-dgzxc\" (UniqueName: \"kubernetes.io/projected/54606b50-cba4-46d5-8bf4-12843a359919-kube-api-access-dgzxc\") pod \"54606b50-cba4-46d5-8bf4-12843a359919\" (UID: \"54606b50-cba4-46d5-8bf4-12843a359919\") "
Oct 30 01:16:43.982588 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:43.982469    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54606b50-cba4-46d5-8bf4-12843a359919-home\" (UniqueName: \"kubernetes.io/empty-dir/54606b50-cba4-46d5-8bf4-12843a359919-home\") pod \"54606b50-cba4-46d5-8bf4-12843a359919\" (UID: \"54606b50-cba4-46d5-8bf4-12843a359919\") "
Oct 30 01:16:43.982588 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:43.982540    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54606b50-cba4-46d5-8bf4-12843a359919-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54606b50-cba4-46d5-8bf4-12843a359919-model-cache\") pod \"54606b50-cba4-46d5-8bf4-12843a359919\" (UID: \"54606b50-cba4-46d5-8bf4-12843a359919\") "
Oct 30 01:16:43.982588 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:43.982561    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54606b50-cba4-46d5-8bf4-12843a359919-tmp-dir" pod "54606b50-cba4-46d5-8bf4-12843a359919" (UID: "54606b50-cba4-46d5-8bf4-12843a359919"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:16:43.982809 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:43.982709    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54606b50-cba4-46d5-8bf4-12843a359919-dshm\" (UniqueName: \"kubernetes.io/empty-dir/54606b50-cba4-46d5-8bf4-12843a359919-dshm\") pod \"54606b50-cba4-46d5-8bf4-12843a359919\" (UID: \"54606b50-cba4-46d5-8bf4-12843a359919\") "
Oct 30 01:16:43.982809 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:43.982745    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54606b50-cba4-46d5-8bf4-12843a359919-home" pod "54606b50-cba4-46d5-8bf4-12843a359919" (UID: "54606b50-cba4-46d5-8bf4-12843a359919"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:16:43.982809 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:43.982766    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54606b50-cba4-46d5-8bf4-12843a359919-model-cache" pod "54606b50-cba4-46d5-8bf4-12843a359919" (UID: "54606b50-cba4-46d5-8bf4-12843a359919"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:16:43.982938 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:43.982813    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/54606b50-cba4-46d5-8bf4-12843a359919-tls-certs\" (UniqueName: \"kubernetes.io/secret/54606b50-cba4-46d5-8bf4-12843a359919-tls-certs\") pod \"54606b50-cba4-46d5-8bf4-12843a359919\" (UID: \"54606b50-cba4-46d5-8bf4-12843a359919\") "
Oct 30 01:16:43.983339 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:43.983318    5002 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54606b50-cba4-46d5-8bf4-12843a359919-tmp-dir\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:16:43.983339 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:43.983338    5002 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54606b50-cba4-46d5-8bf4-12843a359919-home\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:16:43.983566 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:43.983356    5002 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54606b50-cba4-46d5-8bf4-12843a359919-model-cache\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:16:44.051465 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:44.051428    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54606b50-cba4-46d5-8bf4-12843a359919-kube-api-access-dgzxc" pod "54606b50-cba4-46d5-8bf4-12843a359919" (UID: "54606b50-cba4-46d5-8bf4-12843a359919"). InnerVolumeSpecName "kube-api-access-dgzxc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:16:44.051465 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:44.051452    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54606b50-cba4-46d5-8bf4-12843a359919-dshm" pod "54606b50-cba4-46d5-8bf4-12843a359919" (UID: "54606b50-cba4-46d5-8bf4-12843a359919"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:16:44.051671 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:44.051610    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54606b50-cba4-46d5-8bf4-12843a359919-tls-certs" pod "54606b50-cba4-46d5-8bf4-12843a359919" (UID: "54606b50-cba4-46d5-8bf4-12843a359919"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:16:44.084694 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:44.084657    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dgzxc\" (UniqueName: \"kubernetes.io/projected/54606b50-cba4-46d5-8bf4-12843a359919-kube-api-access-dgzxc\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:16:44.084694 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:44.084690    5002 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54606b50-cba4-46d5-8bf4-12843a359919-dshm\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:16:44.084892 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:44.084703    5002 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54606b50-cba4-46d5-8bf4-12843a359919-tls-certs\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:16:44.432999 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:44.432955    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cff2b63-94bdm" event={"ID":"0d8a608f-20c9-4568-9135-4d6e9ef23c97","Type":"ContainerStarted","Data":"bad96d37e14523ca508eef3afdc62587d6fd3fdd5cc51a9a50da1094d2903720"}
Oct 30 01:16:44.433280 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:44.433248    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cff2b63-94bdm" event={"ID":"0d8a608f-20c9-4568-9135-4d6e9ef23c97","Type":"ContainerStarted","Data":"10cdb212969cba65fef51ce336185d13fe8f1fcda77b89a5b93d04ddebed8b53"}
Oct 30 01:16:44.434738 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:44.434712    5002 generic.go:356] "Generic (PLEG): container finished" podID="54606b50-cba4-46d5-8bf4-12843a359919" containerID="1f9cca611b5036823b7d809cfd1aca9bea737a048d756202a7774fbd149453ce" exitCode=0
Oct 30 01:16:44.434855 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:44.434793    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-f64dc0/auth-test-model-f64dc0-kserve-7d44468c6f-xf5xp"
Oct 30 01:16:44.434938 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:44.434791    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-f64dc0/auth-test-model-f64dc0-kserve-7d44468c6f-xf5xp" event={"ID":"54606b50-cba4-46d5-8bf4-12843a359919","Type":"ContainerDied","Data":"1f9cca611b5036823b7d809cfd1aca9bea737a048d756202a7774fbd149453ce"}
Oct 30 01:16:44.434996 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:44.434936    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-f64dc0/auth-test-model-f64dc0-kserve-7d44468c6f-xf5xp" event={"ID":"54606b50-cba4-46d5-8bf4-12843a359919","Type":"ContainerDied","Data":"34caac5a9366015785342f3bc2965fd68e5cab8461a9e0742a63b3f04acab8e1"}
Oct 30 01:16:44.434996 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:44.434964    5002 scope.go:122] "RemoveContainer" containerID="1f9cca611b5036823b7d809cfd1aca9bea737a048d756202a7774fbd149453ce"
Oct 30 01:16:44.436662 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:44.436634    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa210800-6sttw" event={"ID":"22a7542d-4dcf-4059-b4b6-84f6b33c4b6a","Type":"ContainerStarted","Data":"385fc6acc6328c609ed4691b8e1bddc3de7155e3040e27f06aa79d9089e9bed8"}
Oct 30 01:16:44.436790 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:44.436682    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa210800-6sttw" event={"ID":"22a7542d-4dcf-4059-b4b6-84f6b33c4b6a","Type":"ContainerStarted","Data":"5273f3ed9e09bffb349d116a1a7d1bdb2919ded90e184da24e7833c55355678c"}
Oct 30 01:16:44.448612 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:44.448592    5002 scope.go:122] "RemoveContainer" containerID="1f9cca611b5036823b7d809cfd1aca9bea737a048d756202a7774fbd149453ce"
Oct 30 01:16:44.448906 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:16:44.448881    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f9cca611b5036823b7d809cfd1aca9bea737a048d756202a7774fbd149453ce\": container with ID starting with 1f9cca611b5036823b7d809cfd1aca9bea737a048d756202a7774fbd149453ce not found: ID does not exist" containerID="1f9cca611b5036823b7d809cfd1aca9bea737a048d756202a7774fbd149453ce"
Oct 30 01:16:44.448992 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:44.448917    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f9cca611b5036823b7d809cfd1aca9bea737a048d756202a7774fbd149453ce"} err="failed to get container status \"1f9cca611b5036823b7d809cfd1aca9bea737a048d756202a7774fbd149453ce\": rpc error: code = NotFound desc = could not find container \"1f9cca611b5036823b7d809cfd1aca9bea737a048d756202a7774fbd149453ce\": container with ID starting with 1f9cca611b5036823b7d809cfd1aca9bea737a048d756202a7774fbd149453ce not found: ID does not exist"
Oct 30 01:16:44.483480 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:44.483418    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cff2b63-94bdm" podStartSLOduration=2.483399699 podStartE2EDuration="2.483399699s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:16:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:16:44.478697977 +0000 UTC m=+2700.095130857" watchObservedRunningTime="2026-10-30 01:16:44.483399699 +0000 UTC m=+2700.099832580"
Oct 30 01:16:44.523804 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:44.523704    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-f64dc0/auth-test-model-f64dc0-kserve-7d44468c6f-xf5xp"]
Oct 30 01:16:44.529858 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:44.529823    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w2-f64dc0/auth-test-model-f64dc0-kserve-7d44468c6f-xf5xp"]
Oct 30 01:16:44.574882 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:44.574566    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa210800-6sttw" podStartSLOduration=2.574545924 podStartE2EDuration="2.574545924s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:16:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:16:44.568711543 +0000 UTC m=+2700.185144421" watchObservedRunningTime="2026-10-30 01:16:44.574545924 +0000 UTC m=+2700.190978804"
Oct 30 01:16:45.203973 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:45.203942    5002 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-g7c2w_5dc56e7e-f3c1-43eb-a21b-8be6bb4706cf/console-operator/1.log"
Oct 30 01:16:45.226285 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:45.226260    5002 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-g7c2w_5dc56e7e-f3c1-43eb-a21b-8be6bb4706cf/console-operator/1.log"
Oct 30 01:16:45.447228 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:45.447192    5002 generic.go:356] "Generic (PLEG): container finished" podID="22a7542d-4dcf-4059-b4b6-84f6b33c4b6a" containerID="385fc6acc6328c609ed4691b8e1bddc3de7155e3040e27f06aa79d9089e9bed8" exitCode=0
Oct 30 01:16:45.447464 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:45.447276    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa210800-6sttw" event={"ID":"22a7542d-4dcf-4059-b4b6-84f6b33c4b6a","Type":"ContainerDied","Data":"385fc6acc6328c609ed4691b8e1bddc3de7155e3040e27f06aa79d9089e9bed8"}
Oct 30 01:16:45.449420 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:45.449375    5002 generic.go:356] "Generic (PLEG): container finished" podID="0d8a608f-20c9-4568-9135-4d6e9ef23c97" containerID="bad96d37e14523ca508eef3afdc62587d6fd3fdd5cc51a9a50da1094d2903720" exitCode=0
Oct 30 01:16:45.449571 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:45.449432    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cff2b63-94bdm" event={"ID":"0d8a608f-20c9-4568-9135-4d6e9ef23c97","Type":"ContainerDied","Data":"bad96d37e14523ca508eef3afdc62587d6fd3fdd5cc51a9a50da1094d2903720"}
Oct 30 01:16:46.506183 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:46.506139    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54606b50-cba4-46d5-8bf4-12843a359919" path="/var/lib/kubelet/pods/54606b50-cba4-46d5-8bf4-12843a359919/volumes"
Oct 30 01:16:46.653595 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:46.653559    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa210800-6sttw"
Oct 30 01:16:46.714623 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:46.714557    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cff2b63-94bdm"
Oct 30 01:16:46.742600 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:46.742569    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d8a608f-20c9-4568-9135-4d6e9ef23c97-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d8a608f-20c9-4568-9135-4d6e9ef23c97-maas-api-openshift-ca\") pod \"0d8a608f-20c9-4568-9135-4d6e9ef23c97\" (UID: \"0d8a608f-20c9-4568-9135-4d6e9ef23c97\") "
Oct 30 01:16:46.742817 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:46.742681    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d8a608f-20c9-4568-9135-4d6e9ef23c97-kube-api-access-nskb9\" (UniqueName: \"kubernetes.io/projected/0d8a608f-20c9-4568-9135-4d6e9ef23c97-kube-api-access-nskb9\") pod \"0d8a608f-20c9-4568-9135-4d6e9ef23c97\" (UID: \"0d8a608f-20c9-4568-9135-4d6e9ef23c97\") "
Oct 30 01:16:46.742817 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:46.742735    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/22a7542d-4dcf-4059-b4b6-84f6b33c4b6a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22a7542d-4dcf-4059-b4b6-84f6b33c4b6a-maas-api-xks-ca\") pod \"22a7542d-4dcf-4059-b4b6-84f6b33c4b6a\" (UID: \"22a7542d-4dcf-4059-b4b6-84f6b33c4b6a\") "
Oct 30 01:16:46.742950 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:46.742871    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0d8a608f-20c9-4568-9135-4d6e9ef23c97-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d8a608f-20c9-4568-9135-4d6e9ef23c97-maas-api-xks-ca\") pod \"0d8a608f-20c9-4568-9135-4d6e9ef23c97\" (UID: \"0d8a608f-20c9-4568-9135-4d6e9ef23c97\") "
Oct 30 01:16:46.742950 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:46.742919    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22a7542d-4dcf-4059-b4b6-84f6b33c4b6a-kube-api-access-b5p7p\" (UniqueName: \"kubernetes.io/projected/22a7542d-4dcf-4059-b4b6-84f6b33c4b6a-kube-api-access-b5p7p\") pod \"22a7542d-4dcf-4059-b4b6-84f6b33c4b6a\" (UID: \"22a7542d-4dcf-4059-b4b6-84f6b33c4b6a\") "
Oct 30 01:16:46.743066 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:46.743003    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22a7542d-4dcf-4059-b4b6-84f6b33c4b6a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22a7542d-4dcf-4059-b4b6-84f6b33c4b6a-maas-api-openshift-ca\") pod \"22a7542d-4dcf-4059-b4b6-84f6b33c4b6a\" (UID: \"22a7542d-4dcf-4059-b4b6-84f6b33c4b6a\") "
Oct 30 01:16:46.743066 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:46.743019    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d8a608f-20c9-4568-9135-4d6e9ef23c97-maas-api-openshift-ca" pod "0d8a608f-20c9-4568-9135-4d6e9ef23c97" (UID: "0d8a608f-20c9-4568-9135-4d6e9ef23c97"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:16:46.743775 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:46.743749    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d8a608f-20c9-4568-9135-4d6e9ef23c97-maas-api-openshift-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:16:46.744072 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:46.744052    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22a7542d-4dcf-4059-b4b6-84f6b33c4b6a-maas-api-openshift-ca" pod "22a7542d-4dcf-4059-b4b6-84f6b33c4b6a" (UID: "22a7542d-4dcf-4059-b4b6-84f6b33c4b6a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:16:46.745516 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:46.745485    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22a7542d-4dcf-4059-b4b6-84f6b33c4b6a-maas-api-xks-ca" pod "22a7542d-4dcf-4059-b4b6-84f6b33c4b6a" (UID: "22a7542d-4dcf-4059-b4b6-84f6b33c4b6a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:16:46.745886 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:46.745862    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d8a608f-20c9-4568-9135-4d6e9ef23c97-kube-api-access-nskb9" pod "0d8a608f-20c9-4568-9135-4d6e9ef23c97" (UID: "0d8a608f-20c9-4568-9135-4d6e9ef23c97"). InnerVolumeSpecName "kube-api-access-nskb9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:16:46.746046 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:46.746022    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d8a608f-20c9-4568-9135-4d6e9ef23c97-maas-api-xks-ca" pod "0d8a608f-20c9-4568-9135-4d6e9ef23c97" (UID: "0d8a608f-20c9-4568-9135-4d6e9ef23c97"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:16:46.746118 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:46.746076    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22a7542d-4dcf-4059-b4b6-84f6b33c4b6a-kube-api-access-b5p7p" pod "22a7542d-4dcf-4059-b4b6-84f6b33c4b6a" (UID: "22a7542d-4dcf-4059-b4b6-84f6b33c4b6a"). InnerVolumeSpecName "kube-api-access-b5p7p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:16:46.845394 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:46.845346    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nskb9\" (UniqueName: \"kubernetes.io/projected/0d8a608f-20c9-4568-9135-4d6e9ef23c97-kube-api-access-nskb9\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:16:46.845586 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:46.845413    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22a7542d-4dcf-4059-b4b6-84f6b33c4b6a-maas-api-xks-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:16:46.845586 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:46.845433    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d8a608f-20c9-4568-9135-4d6e9ef23c97-maas-api-xks-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:16:46.845586 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:46.845446    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b5p7p\" (UniqueName: \"kubernetes.io/projected/22a7542d-4dcf-4059-b4b6-84f6b33c4b6a-kube-api-access-b5p7p\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:16:46.845586 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:46.845461    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22a7542d-4dcf-4059-b4b6-84f6b33c4b6a-maas-api-openshift-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:16:47.493986 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:47.493942    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cff2b63-94bdm"
Oct 30 01:16:47.494354 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:47.494323    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cff2b63-94bdm" event={"ID":"0d8a608f-20c9-4568-9135-4d6e9ef23c97","Type":"ContainerDied","Data":"10cdb212969cba65fef51ce336185d13fe8f1fcda77b89a5b93d04ddebed8b53"}
Oct 30 01:16:47.494481 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:47.494459    5002 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="10cdb212969cba65fef51ce336185d13fe8f1fcda77b89a5b93d04ddebed8b53"
Oct 30 01:16:47.495819 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:47.495780    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa210800-6sttw"
Oct 30 01:16:47.495924 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:47.495792    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa210800-6sttw" event={"ID":"22a7542d-4dcf-4059-b4b6-84f6b33c4b6a","Type":"ContainerDied","Data":"5273f3ed9e09bffb349d116a1a7d1bdb2919ded90e184da24e7833c55355678c"}
Oct 30 01:16:47.495979 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:47.495942    5002 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5273f3ed9e09bffb349d116a1a7d1bdb2919ded90e184da24e7833c55355678c"
Oct 30 01:16:48.193149 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:48.193116    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f6d5b8-865b864594-d5n25"]
Oct 30 01:16:48.213174 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:48.213143    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f6d5b8-55c9ff76c5-xh6w2"]
Oct 30 01:16:49.010405 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:49.009967    5002 scope.go:122] "RemoveContainer" containerID="e2488766af84367d08385f61563c800c92c0aa3c5179a21d052706a5b4d7ed76"
Oct 30 01:16:49.028762 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:49.028735    5002 scope.go:122] "RemoveContainer" containerID="849a7fedf6cc4a703eb8a93ffffbcc279e421b0a09a239d0e21dc998aecfa174"
Oct 30 01:16:49.040829 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:49.040805    5002 scope.go:122] "RemoveContainer" containerID="59ca3a29d72645f2b65acf6309c4ce820a7d20cf2ab8b5d51f0e80dfbd80127e"
Oct 30 01:16:49.052063 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:49.050444    5002 scope.go:122] "RemoveContainer" containerID="675513e0f17c304163c14f327e739f3aacda9c002e7c4bb6aa84c2e23046433d"
Oct 30 01:16:50.326724 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:50.326683    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-812ba2-7768d47f6d-bfbkj"]
Oct 30 01:16:50.327632 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:50.327605    5002 cpu_manager.go:414] "containerMap: removing container" podUID="54606b50-cba4-46d5-8bf4-12843a359919" containerName="main"
Oct 30 01:16:50.327826 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:50.327807    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54606b50-cba4-46d5-8bf4-12843a359919" containerName="main"
Oct 30 01:16:50.328009 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:50.327996    5002 cpu_manager.go:414] "containerMap: removing container" podUID="22a7542d-4dcf-4059-b4b6-84f6b33c4b6a" containerName="revoke-keys"
Oct 30 01:16:50.328092 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:50.328081    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22a7542d-4dcf-4059-b4b6-84f6b33c4b6a" containerName="revoke-keys"
Oct 30 01:16:50.328194 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:50.328166    5002 cpu_manager.go:414] "containerMap: removing container" podUID="0d8a608f-20c9-4568-9135-4d6e9ef23c97" containerName="revoke-keys"
Oct 30 01:16:50.328194 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:50.328193    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d8a608f-20c9-4568-9135-4d6e9ef23c97" containerName="revoke-keys"
Oct 30 01:16:50.328451 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:50.328433    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="22a7542d-4dcf-4059-b4b6-84f6b33c4b6a" containerName="revoke-keys"
Oct 30 01:16:50.328451 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:50.328450    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d8a608f-20c9-4568-9135-4d6e9ef23c97" containerName="revoke-keys"
Oct 30 01:16:50.328613 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:50.328466    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="54606b50-cba4-46d5-8bf4-12843a359919" containerName="main"
Oct 30 01:16:50.347406 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:50.343754    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-812ba2-7768d47f6d-bfbkj"
Oct 30 01:16:50.376403 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:50.373916    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-812ba2\""
Oct 30 01:16:50.403415 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:50.401634    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-812ba2-dockercfg-bjhm6\""
Oct 30 01:16:50.409403 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:50.406498    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-812ba2-7768d47f6d-bfbkj"]
Oct 30 01:16:50.440287 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:50.440240    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f8a6baf-d556-4d26-96a2-11ef06fac9ea-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-812ba2-7768d47f6d-bfbkj\" (UID: \"4f8a6baf-d556-4d26-96a2-11ef06fac9ea\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-812ba2-7768d47f6d-bfbkj"
Oct 30 01:16:50.440775 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:50.440746    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c58hv\" (UniqueName: \"kubernetes.io/projected/4f8a6baf-d556-4d26-96a2-11ef06fac9ea-kube-api-access-c58hv\") pod \"payload-processing-e2e-ipp-a-812ba2-7768d47f6d-bfbkj\" (UID: \"4f8a6baf-d556-4d26-96a2-11ef06fac9ea\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-812ba2-7768d47f6d-bfbkj"
Oct 30 01:16:50.542204 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:50.542065    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f8a6baf-d556-4d26-96a2-11ef06fac9ea-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-812ba2-7768d47f6d-bfbkj\" (UID: \"4f8a6baf-d556-4d26-96a2-11ef06fac9ea\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-812ba2-7768d47f6d-bfbkj"
Oct 30 01:16:50.545449 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:50.542680    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c58hv\" (UniqueName: \"kubernetes.io/projected/4f8a6baf-d556-4d26-96a2-11ef06fac9ea-kube-api-access-c58hv\") pod \"payload-processing-e2e-ipp-a-812ba2-7768d47f6d-bfbkj\" (UID: \"4f8a6baf-d556-4d26-96a2-11ef06fac9ea\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-812ba2-7768d47f6d-bfbkj"
Oct 30 01:16:50.545449 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:50.544333    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f8a6baf-d556-4d26-96a2-11ef06fac9ea-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-812ba2-7768d47f6d-bfbkj\" (UID: \"4f8a6baf-d556-4d26-96a2-11ef06fac9ea\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-812ba2-7768d47f6d-bfbkj"
Oct 30 01:16:50.605069 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:50.599662    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c58hv\" (UniqueName: \"kubernetes.io/projected/4f8a6baf-d556-4d26-96a2-11ef06fac9ea-kube-api-access-c58hv\") pod \"payload-processing-e2e-ipp-a-812ba2-7768d47f6d-bfbkj\" (UID: \"4f8a6baf-d556-4d26-96a2-11ef06fac9ea\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-812ba2-7768d47f6d-bfbkj"
Oct 30 01:16:50.707916 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:50.707882    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-812ba2-7768d47f6d-bfbkj"
Oct 30 01:16:51.019080 ip-10-0-128-7 kubenswrapper[5002]: W1030 01:16:51.019051    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f8a6baf_d556_4d26_96a2_11ef06fac9ea.slice/crio-da88e5b5a84ddaa489e53df58a77c451862342d00ef557c724bde6bea174b1c8 WatchSource:0}: Error finding container da88e5b5a84ddaa489e53df58a77c451862342d00ef557c724bde6bea174b1c8: Status 404 returned error can't find the container with id da88e5b5a84ddaa489e53df58a77c451862342d00ef557c724bde6bea174b1c8
Oct 30 01:16:51.021374 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:51.021342    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-812ba2-7768d47f6d-bfbkj"]
Oct 30 01:16:51.568103 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:51.568059    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-812ba2-7768d47f6d-bfbkj" event={"ID":"4f8a6baf-d556-4d26-96a2-11ef06fac9ea","Type":"ContainerStarted","Data":"22abfca05e4831f40a63f07f347cf676c1efe426b92941f255716638ea1520e1"}
Oct 30 01:16:51.568568 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:51.568133    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-812ba2-7768d47f6d-bfbkj" event={"ID":"4f8a6baf-d556-4d26-96a2-11ef06fac9ea","Type":"ContainerStarted","Data":"da88e5b5a84ddaa489e53df58a77c451862342d00ef557c724bde6bea174b1c8"}
Oct 30 01:16:51.568568 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:51.568278    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-812ba2-7768d47f6d-bfbkj"
Oct 30 01:16:51.613037 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:51.612984    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-812ba2-7768d47f6d-bfbkj" podStartSLOduration=1.6129698540000001 podStartE2EDuration="1.612969854s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:16:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:16:51.612412776 +0000 UTC m=+2707.228845654" watchObservedRunningTime="2026-10-30 01:16:51.612969854 +0000 UTC m=+2707.229402732"
Oct 30 01:16:51.896456 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:51.883249    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7d8z4"]
Oct 30 01:16:53.210401 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:53.194444    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-f6d5b8-865b864594-d5n25" podUID="9064e71e-4762-451a-8f62-bfb099e6c396" containerName="payload-processing" containerID="cri-o://9556aa13062f228be38f6103ba1f06cc992d9b4b41446d28a78583c44c9aa4d5" gracePeriod=25
Oct 30 01:16:53.220402 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:53.213910    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f6d5b8-55c9ff76c5-xh6w2" podUID="7dd42ca9-65d9-415d-9a0c-0151c50a72d6" containerName="payload-pre-processing" containerID="cri-o://1a03a5f8f3ce2ca39b8e836070e109f0aa870c55d64ae0d0416edabc862a9438" gracePeriod=25
Oct 30 01:16:53.220402 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:53.216321    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f47f1548-n9tw2"]
Oct 30 01:16:53.260629 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:53.258618    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f47f1548-n9tw2"
Oct 30 01:16:53.286928 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:53.276760    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f47f1548-n9tw2"]
Oct 30 01:16:53.286928 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:53.280184    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:16:53.286928 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:53.280235    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:16:53.286928 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:53.283269    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3b976fd-1e7c-4bb1-b514-abc54f0a188f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f47f1548-n9tw2\" (UID: \"b3b976fd-1e7c-4bb1-b514-abc54f0a188f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f47f1548-n9tw2"
Oct 30 01:16:53.286928 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:53.283439    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3b976fd-1e7c-4bb1-b514-abc54f0a188f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f47f1548-n9tw2\" (UID: \"b3b976fd-1e7c-4bb1-b514-abc54f0a188f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f47f1548-n9tw2"
Oct 30 01:16:53.286928 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:53.283531    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfgs4\" (UniqueName: \"kubernetes.io/projected/b3b976fd-1e7c-4bb1-b514-abc54f0a188f-kube-api-access-kfgs4\") pod \"maas-api-revoke-sub-f47f1548-n9tw2\" (UID: \"b3b976fd-1e7c-4bb1-b514-abc54f0a188f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f47f1548-n9tw2"
Oct 30 01:16:53.297101 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:53.297071    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:16:53.297519 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:53.297498    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5phsh\""
Oct 30 01:16:53.386067 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:53.384625    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3b976fd-1e7c-4bb1-b514-abc54f0a188f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f47f1548-n9tw2\" (UID: \"b3b976fd-1e7c-4bb1-b514-abc54f0a188f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f47f1548-n9tw2"
Oct 30 01:16:53.386067 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:53.384736    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfgs4\" (UniqueName: \"kubernetes.io/projected/b3b976fd-1e7c-4bb1-b514-abc54f0a188f-kube-api-access-kfgs4\") pod \"maas-api-revoke-sub-f47f1548-n9tw2\" (UID: \"b3b976fd-1e7c-4bb1-b514-abc54f0a188f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f47f1548-n9tw2"
Oct 30 01:16:53.386067 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:53.384930    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3b976fd-1e7c-4bb1-b514-abc54f0a188f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f47f1548-n9tw2\" (UID: \"b3b976fd-1e7c-4bb1-b514-abc54f0a188f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f47f1548-n9tw2"
Oct 30 01:16:53.386569 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:53.386532    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3b976fd-1e7c-4bb1-b514-abc54f0a188f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f47f1548-n9tw2\" (UID: \"b3b976fd-1e7c-4bb1-b514-abc54f0a188f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f47f1548-n9tw2"
Oct 30 01:16:53.392127 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:53.392097    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3b976fd-1e7c-4bb1-b514-abc54f0a188f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f47f1548-n9tw2\" (UID: \"b3b976fd-1e7c-4bb1-b514-abc54f0a188f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f47f1548-n9tw2"
Oct 30 01:16:53.420086 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:53.420023    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfgs4\" (UniqueName: \"kubernetes.io/projected/b3b976fd-1e7c-4bb1-b514-abc54f0a188f-kube-api-access-kfgs4\") pod \"maas-api-revoke-sub-f47f1548-n9tw2\" (UID: \"b3b976fd-1e7c-4bb1-b514-abc54f0a188f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f47f1548-n9tw2"
Oct 30 01:16:53.525440 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:53.525405    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f6d5b8-55c9ff76c5-xh6w2"
Oct 30 01:16:53.558853 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:53.558822    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f6d5b8-865b864594-d5n25"
Oct 30 01:16:53.587724 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:53.587686    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7dd42ca9-65d9-415d-9a0c-0151c50a72d6-kube-api-access-njv7k\" (UniqueName: \"kubernetes.io/projected/7dd42ca9-65d9-415d-9a0c-0151c50a72d6-kube-api-access-njv7k\") pod \"7dd42ca9-65d9-415d-9a0c-0151c50a72d6\" (UID: \"7dd42ca9-65d9-415d-9a0c-0151c50a72d6\") "
Oct 30 01:16:53.587906 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:53.587762    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7dd42ca9-65d9-415d-9a0c-0151c50a72d6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7dd42ca9-65d9-415d-9a0c-0151c50a72d6-plugins-config-volume\") pod \"7dd42ca9-65d9-415d-9a0c-0151c50a72d6\" (UID: \"7dd42ca9-65d9-415d-9a0c-0151c50a72d6\") "
Oct 30 01:16:53.597535 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:53.588012    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9064e71e-4762-451a-8f62-bfb099e6c396-kube-api-access-xm42x\" (UniqueName: \"kubernetes.io/projected/9064e71e-4762-451a-8f62-bfb099e6c396-kube-api-access-xm42x\") pod \"9064e71e-4762-451a-8f62-bfb099e6c396\" (UID: \"9064e71e-4762-451a-8f62-bfb099e6c396\") "
Oct 30 01:16:53.597535 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:53.588055    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9064e71e-4762-451a-8f62-bfb099e6c396-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9064e71e-4762-451a-8f62-bfb099e6c396-plugins-config-volume\") pod \"9064e71e-4762-451a-8f62-bfb099e6c396\" (UID: \"9064e71e-4762-451a-8f62-bfb099e6c396\") "
Oct 30 01:16:53.597535 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:53.588918    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9064e71e-4762-451a-8f62-bfb099e6c396-plugins-config-volume" pod "9064e71e-4762-451a-8f62-bfb099e6c396" (UID: "9064e71e-4762-451a-8f62-bfb099e6c396"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:16:53.597535 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:53.591651    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7dd42ca9-65d9-415d-9a0c-0151c50a72d6-kube-api-access-njv7k" pod "7dd42ca9-65d9-415d-9a0c-0151c50a72d6" (UID: "7dd42ca9-65d9-415d-9a0c-0151c50a72d6"). InnerVolumeSpecName "kube-api-access-njv7k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:16:53.597535 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:53.592007    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7dd42ca9-65d9-415d-9a0c-0151c50a72d6-plugins-config-volume" pod "7dd42ca9-65d9-415d-9a0c-0151c50a72d6" (UID: "7dd42ca9-65d9-415d-9a0c-0151c50a72d6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:16:53.601205 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:53.600636    5002 generic.go:356] "Generic (PLEG): container finished" podID="9064e71e-4762-451a-8f62-bfb099e6c396" containerID="9556aa13062f228be38f6103ba1f06cc992d9b4b41446d28a78583c44c9aa4d5" exitCode=0
Oct 30 01:16:53.601205 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:53.600819    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f6d5b8-865b864594-d5n25" event={"ID":"9064e71e-4762-451a-8f62-bfb099e6c396","Type":"ContainerDied","Data":"9556aa13062f228be38f6103ba1f06cc992d9b4b41446d28a78583c44c9aa4d5"}
Oct 30 01:16:53.601205 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:53.600871    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f6d5b8-865b864594-d5n25" event={"ID":"9064e71e-4762-451a-8f62-bfb099e6c396","Type":"ContainerDied","Data":"395ff2a2c136e24d715e48509183b19030d10dde2f3e5699c27762d69c7cf1da"}
Oct 30 01:16:53.601205 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:53.600902    5002 scope.go:122] "RemoveContainer" containerID="9556aa13062f228be38f6103ba1f06cc992d9b4b41446d28a78583c44c9aa4d5"
Oct 30 01:16:53.601205 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:53.601044    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f6d5b8-865b864594-d5n25"
Oct 30 01:16:53.612652 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:53.612586    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9064e71e-4762-451a-8f62-bfb099e6c396-kube-api-access-xm42x" pod "9064e71e-4762-451a-8f62-bfb099e6c396" (UID: "9064e71e-4762-451a-8f62-bfb099e6c396"). InnerVolumeSpecName "kube-api-access-xm42x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:16:53.614543 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:53.614004    5002 generic.go:356] "Generic (PLEG): container finished" podID="7dd42ca9-65d9-415d-9a0c-0151c50a72d6" containerID="1a03a5f8f3ce2ca39b8e836070e109f0aa870c55d64ae0d0416edabc862a9438" exitCode=0
Oct 30 01:16:53.614543 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:53.614063    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f6d5b8-55c9ff76c5-xh6w2" event={"ID":"7dd42ca9-65d9-415d-9a0c-0151c50a72d6","Type":"ContainerDied","Data":"1a03a5f8f3ce2ca39b8e836070e109f0aa870c55d64ae0d0416edabc862a9438"}
Oct 30 01:16:53.614543 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:53.614116    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f6d5b8-55c9ff76c5-xh6w2" event={"ID":"7dd42ca9-65d9-415d-9a0c-0151c50a72d6","Type":"ContainerDied","Data":"c06211688781955f873243b8e3ed1f42d2da4fe8ad2a122cd96729b10f499a0f"}
Oct 30 01:16:53.614543 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:53.614205    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f6d5b8-55c9ff76c5-xh6w2"
Oct 30 01:16:53.744447 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:53.713616    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f47f1548-n9tw2"
Oct 30 01:16:53.764462 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:53.745882    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xm42x\" (UniqueName: \"kubernetes.io/projected/9064e71e-4762-451a-8f62-bfb099e6c396-kube-api-access-xm42x\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:16:53.823883 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:53.823838    5002 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9064e71e-4762-451a-8f62-bfb099e6c396-plugins-config-volume\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:16:53.824169 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:53.824139    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-njv7k\" (UniqueName: \"kubernetes.io/projected/7dd42ca9-65d9-415d-9a0c-0151c50a72d6-kube-api-access-njv7k\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:16:53.824344 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:53.824327    5002 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7dd42ca9-65d9-415d-9a0c-0151c50a72d6-plugins-config-volume\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:16:54.015933 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:54.015835    5002 scope.go:122] "RemoveContainer" containerID="9556aa13062f228be38f6103ba1f06cc992d9b4b41446d28a78583c44c9aa4d5"
Oct 30 01:16:54.096578 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:16:54.067524    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9556aa13062f228be38f6103ba1f06cc992d9b4b41446d28a78583c44c9aa4d5\": container with ID starting with 9556aa13062f228be38f6103ba1f06cc992d9b4b41446d28a78583c44c9aa4d5 not found: ID does not exist" containerID="9556aa13062f228be38f6103ba1f06cc992d9b4b41446d28a78583c44c9aa4d5"
Oct 30 01:16:54.096578 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:54.067575    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9556aa13062f228be38f6103ba1f06cc992d9b4b41446d28a78583c44c9aa4d5"} err="failed to get container status \"9556aa13062f228be38f6103ba1f06cc992d9b4b41446d28a78583c44c9aa4d5\": rpc error: code = NotFound desc = could not find container \"9556aa13062f228be38f6103ba1f06cc992d9b4b41446d28a78583c44c9aa4d5\": container with ID starting with 9556aa13062f228be38f6103ba1f06cc992d9b4b41446d28a78583c44c9aa4d5 not found: ID does not exist"
Oct 30 01:16:54.096578 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:54.067606    5002 scope.go:122] "RemoveContainer" containerID="1a03a5f8f3ce2ca39b8e836070e109f0aa870c55d64ae0d0416edabc862a9438"
Oct 30 01:16:54.143611 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:54.142678    5002 scope.go:122] "RemoveContainer" containerID="1a03a5f8f3ce2ca39b8e836070e109f0aa870c55d64ae0d0416edabc862a9438"
Oct 30 01:16:54.143886 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:16:54.143812    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a03a5f8f3ce2ca39b8e836070e109f0aa870c55d64ae0d0416edabc862a9438\": container with ID starting with 1a03a5f8f3ce2ca39b8e836070e109f0aa870c55d64ae0d0416edabc862a9438 not found: ID does not exist" containerID="1a03a5f8f3ce2ca39b8e836070e109f0aa870c55d64ae0d0416edabc862a9438"
Oct 30 01:16:54.143886 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:54.143856    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a03a5f8f3ce2ca39b8e836070e109f0aa870c55d64ae0d0416edabc862a9438"} err="failed to get container status \"1a03a5f8f3ce2ca39b8e836070e109f0aa870c55d64ae0d0416edabc862a9438\": rpc error: code = NotFound desc = could not find container \"1a03a5f8f3ce2ca39b8e836070e109f0aa870c55d64ae0d0416edabc862a9438\": container with ID starting with 1a03a5f8f3ce2ca39b8e836070e109f0aa870c55d64ae0d0416edabc862a9438 not found: ID does not exist"
Oct 30 01:16:54.159429 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:54.159128    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f6d5b8-55c9ff76c5-xh6w2"]
Oct 30 01:16:54.172769 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:54.172728    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f6d5b8-55c9ff76c5-xh6w2"]
Oct 30 01:16:54.205297 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:54.205237    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f6d5b8-865b864594-d5n25"]
Oct 30 01:16:54.217661 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:54.217636    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f6d5b8-865b864594-d5n25"]
Oct 30 01:16:54.449283 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:54.449256    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f47f1548-n9tw2"]
Oct 30 01:16:54.449531 ip-10-0-128-7 kubenswrapper[5002]: W1030 01:16:54.449500    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3b976fd_1e7c_4bb1_b514_abc54f0a188f.slice/crio-dbefb06351caaca3511b240375ff458267d4a66fc75407b7d6e0b23cfa954bea WatchSource:0}: Error finding container dbefb06351caaca3511b240375ff458267d4a66fc75407b7d6e0b23cfa954bea: Status 404 returned error can't find the container with id dbefb06351caaca3511b240375ff458267d4a66fc75407b7d6e0b23cfa954bea
Oct 30 01:16:54.501519 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:54.501484    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7dd42ca9-65d9-415d-9a0c-0151c50a72d6" path="/var/lib/kubelet/pods/7dd42ca9-65d9-415d-9a0c-0151c50a72d6/volumes"
Oct 30 01:16:54.502126 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:54.502105    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9064e71e-4762-451a-8f62-bfb099e6c396" path="/var/lib/kubelet/pods/9064e71e-4762-451a-8f62-bfb099e6c396/volumes"
Oct 30 01:16:54.625717 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:54.625676    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f47f1548-n9tw2" event={"ID":"b3b976fd-1e7c-4bb1-b514-abc54f0a188f","Type":"ContainerStarted","Data":"dbefb06351caaca3511b240375ff458267d4a66fc75407b7d6e0b23cfa954bea"}
Oct 30 01:16:55.635962 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:55.635926    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f47f1548-n9tw2" event={"ID":"b3b976fd-1e7c-4bb1-b514-abc54f0a188f","Type":"ContainerStarted","Data":"3b26ec4ec28249a5da79daf51da509b69c91a086c81e539262b86f492209d527"}
Oct 30 01:16:55.711511 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:55.711464    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f47f1548-n9tw2" podStartSLOduration=2.711450466 podStartE2EDuration="2.711450466s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:16:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:16:55.710608517 +0000 UTC m=+2711.327041414" watchObservedRunningTime="2026-10-30 01:16:55.711450466 +0000 UTC m=+2711.327883345"
Oct 30 01:16:56.656406 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:56.649497    5002 generic.go:356] "Generic (PLEG): container finished" podID="b3b976fd-1e7c-4bb1-b514-abc54f0a188f" containerID="3b26ec4ec28249a5da79daf51da509b69c91a086c81e539262b86f492209d527" exitCode=0
Oct 30 01:16:56.656406 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:56.649632    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f47f1548-n9tw2" event={"ID":"b3b976fd-1e7c-4bb1-b514-abc54f0a188f","Type":"ContainerDied","Data":"3b26ec4ec28249a5da79daf51da509b69c91a086c81e539262b86f492209d527"}
Oct 30 01:16:57.940867 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:57.940142    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f47f1548-n9tw2"
Oct 30 01:16:58.005155 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:58.004923    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b3b976fd-1e7c-4bb1-b514-abc54f0a188f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3b976fd-1e7c-4bb1-b514-abc54f0a188f-maas-api-xks-ca\") pod \"b3b976fd-1e7c-4bb1-b514-abc54f0a188f\" (UID: \"b3b976fd-1e7c-4bb1-b514-abc54f0a188f\") "
Oct 30 01:16:58.005155 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:58.005030    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3b976fd-1e7c-4bb1-b514-abc54f0a188f-kube-api-access-kfgs4\" (UniqueName: \"kubernetes.io/projected/b3b976fd-1e7c-4bb1-b514-abc54f0a188f-kube-api-access-kfgs4\") pod \"b3b976fd-1e7c-4bb1-b514-abc54f0a188f\" (UID: \"b3b976fd-1e7c-4bb1-b514-abc54f0a188f\") "
Oct 30 01:16:58.005155 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:58.005121    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3b976fd-1e7c-4bb1-b514-abc54f0a188f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3b976fd-1e7c-4bb1-b514-abc54f0a188f-maas-api-openshift-ca\") pod \"b3b976fd-1e7c-4bb1-b514-abc54f0a188f\" (UID: \"b3b976fd-1e7c-4bb1-b514-abc54f0a188f\") "
Oct 30 01:16:58.006267 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:58.006235    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3b976fd-1e7c-4bb1-b514-abc54f0a188f-maas-api-openshift-ca" pod "b3b976fd-1e7c-4bb1-b514-abc54f0a188f" (UID: "b3b976fd-1e7c-4bb1-b514-abc54f0a188f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:16:58.012508 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:58.012480    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3b976fd-1e7c-4bb1-b514-abc54f0a188f-maas-api-xks-ca" pod "b3b976fd-1e7c-4bb1-b514-abc54f0a188f" (UID: "b3b976fd-1e7c-4bb1-b514-abc54f0a188f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:16:58.015544 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:58.015444    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3b976fd-1e7c-4bb1-b514-abc54f0a188f-kube-api-access-kfgs4" pod "b3b976fd-1e7c-4bb1-b514-abc54f0a188f" (UID: "b3b976fd-1e7c-4bb1-b514-abc54f0a188f"). InnerVolumeSpecName "kube-api-access-kfgs4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:16:58.107212 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:58.107170    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3b976fd-1e7c-4bb1-b514-abc54f0a188f-maas-api-xks-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:16:58.107212 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:58.107208    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kfgs4\" (UniqueName: \"kubernetes.io/projected/b3b976fd-1e7c-4bb1-b514-abc54f0a188f-kube-api-access-kfgs4\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:16:58.107212 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:58.107219    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3b976fd-1e7c-4bb1-b514-abc54f0a188f-maas-api-openshift-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:16:58.700454 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:58.700420    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f47f1548-n9tw2"
Oct 30 01:16:58.700627 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:58.700419    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f47f1548-n9tw2" event={"ID":"b3b976fd-1e7c-4bb1-b514-abc54f0a188f","Type":"ContainerDied","Data":"dbefb06351caaca3511b240375ff458267d4a66fc75407b7d6e0b23cfa954bea"}
Oct 30 01:16:58.700627 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:16:58.700552    5002 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dbefb06351caaca3511b240375ff458267d4a66fc75407b7d6e0b23cfa954bea"
Oct 30 01:17:02.578152 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:02.578111    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-a3619c-openshift-default-675f567c95-jsx2v"]
Oct 30 01:17:02.579055 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:02.579024    5002 cpu_manager.go:414] "containerMap: removing container" podUID="9064e71e-4762-451a-8f62-bfb099e6c396" containerName="payload-processing"
Oct 30 01:17:02.579055 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:02.579052    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9064e71e-4762-451a-8f62-bfb099e6c396" containerName="payload-processing"
Oct 30 01:17:02.579292 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:02.579107    5002 cpu_manager.go:414] "containerMap: removing container" podUID="7dd42ca9-65d9-415d-9a0c-0151c50a72d6" containerName="payload-pre-processing"
Oct 30 01:17:02.579292 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:02.579117    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7dd42ca9-65d9-415d-9a0c-0151c50a72d6" containerName="payload-pre-processing"
Oct 30 01:17:02.579292 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:02.579153    5002 cpu_manager.go:414] "containerMap: removing container" podUID="b3b976fd-1e7c-4bb1-b514-abc54f0a188f" containerName="revoke-keys"
Oct 30 01:17:02.579292 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:02.579161    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3b976fd-1e7c-4bb1-b514-abc54f0a188f" containerName="revoke-keys"
Oct 30 01:17:02.579549 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:02.579296    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="7dd42ca9-65d9-415d-9a0c-0151c50a72d6" containerName="payload-pre-processing"
Oct 30 01:17:02.579549 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:02.579315    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3b976fd-1e7c-4bb1-b514-abc54f0a188f" containerName="revoke-keys"
Oct 30 01:17:02.579549 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:02.579329    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="9064e71e-4762-451a-8f62-bfb099e6c396" containerName="payload-processing"
Oct 30 01:17:02.583654 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:02.583629    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-812ba2-7768d47f6d-bfbkj"
Oct 30 01:17:02.583801 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:02.583688    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-a3619c-openshift-default-675f567c95-jsx2v"
Oct 30 01:17:02.606873 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:02.606843    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-a3619c-openshift-default-dockercfg-4lvjq\""
Oct 30 01:17:02.620593 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:02.620548    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-a3619c-openshift-default-675f567c95-jsx2v"]
Oct 30 01:17:02.681297 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:02.681258    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9058f9e-a7e7-4f30-87be-e037e5378b3c-workload-certs\") pod \"e2e-shared-b-w2-a3619c-openshift-default-675f567c95-jsx2v\" (UID: \"b9058f9e-a7e7-4f30-87be-e037e5378b3c\") " pod="openshift-ingress/e2e-shared-b-w2-a3619c-openshift-default-675f567c95-jsx2v"
Oct 30 01:17:02.681495 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:02.681318    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9058f9e-a7e7-4f30-87be-e037e5378b3c-istio-envoy\") pod \"e2e-shared-b-w2-a3619c-openshift-default-675f567c95-jsx2v\" (UID: \"b9058f9e-a7e7-4f30-87be-e037e5378b3c\") " pod="openshift-ingress/e2e-shared-b-w2-a3619c-openshift-default-675f567c95-jsx2v"
Oct 30 01:17:02.681495 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:02.681391    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9058f9e-a7e7-4f30-87be-e037e5378b3c-istio-podinfo\") pod \"e2e-shared-b-w2-a3619c-openshift-default-675f567c95-jsx2v\" (UID: \"b9058f9e-a7e7-4f30-87be-e037e5378b3c\") " pod="openshift-ingress/e2e-shared-b-w2-a3619c-openshift-default-675f567c95-jsx2v"
Oct 30 01:17:02.681495 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:02.681421    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9058f9e-a7e7-4f30-87be-e037e5378b3c-istiod-ca-cert\") pod \"e2e-shared-b-w2-a3619c-openshift-default-675f567c95-jsx2v\" (UID: \"b9058f9e-a7e7-4f30-87be-e037e5378b3c\") " pod="openshift-ingress/e2e-shared-b-w2-a3619c-openshift-default-675f567c95-jsx2v"
Oct 30 01:17:02.681495 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:02.681473    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9058f9e-a7e7-4f30-87be-e037e5378b3c-credential-socket\") pod \"e2e-shared-b-w2-a3619c-openshift-default-675f567c95-jsx2v\" (UID: \"b9058f9e-a7e7-4f30-87be-e037e5378b3c\") " pod="openshift-ingress/e2e-shared-b-w2-a3619c-openshift-default-675f567c95-jsx2v"
Oct 30 01:17:02.681743 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:02.681512    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9058f9e-a7e7-4f30-87be-e037e5378b3c-workload-socket\") pod \"e2e-shared-b-w2-a3619c-openshift-default-675f567c95-jsx2v\" (UID: \"b9058f9e-a7e7-4f30-87be-e037e5378b3c\") " pod="openshift-ingress/e2e-shared-b-w2-a3619c-openshift-default-675f567c95-jsx2v"
Oct 30 01:17:02.681743 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:02.681554    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9058f9e-a7e7-4f30-87be-e037e5378b3c-istio-token\") pod \"e2e-shared-b-w2-a3619c-openshift-default-675f567c95-jsx2v\" (UID: \"b9058f9e-a7e7-4f30-87be-e037e5378b3c\") " pod="openshift-ingress/e2e-shared-b-w2-a3619c-openshift-default-675f567c95-jsx2v"
Oct 30 01:17:02.681743 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:02.681675    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xhnc\" (UniqueName: \"kubernetes.io/projected/b9058f9e-a7e7-4f30-87be-e037e5378b3c-kube-api-access-7xhnc\") pod \"e2e-shared-b-w2-a3619c-openshift-default-675f567c95-jsx2v\" (UID: \"b9058f9e-a7e7-4f30-87be-e037e5378b3c\") " pod="openshift-ingress/e2e-shared-b-w2-a3619c-openshift-default-675f567c95-jsx2v"
Oct 30 01:17:02.683662 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:02.681752    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9058f9e-a7e7-4f30-87be-e037e5378b3c-istio-data\") pod \"e2e-shared-b-w2-a3619c-openshift-default-675f567c95-jsx2v\" (UID: \"b9058f9e-a7e7-4f30-87be-e037e5378b3c\") " pod="openshift-ingress/e2e-shared-b-w2-a3619c-openshift-default-675f567c95-jsx2v"
Oct 30 01:17:02.783742 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:02.783693    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9058f9e-a7e7-4f30-87be-e037e5378b3c-workload-certs\") pod \"e2e-shared-b-w2-a3619c-openshift-default-675f567c95-jsx2v\" (UID: \"b9058f9e-a7e7-4f30-87be-e037e5378b3c\") " pod="openshift-ingress/e2e-shared-b-w2-a3619c-openshift-default-675f567c95-jsx2v"
Oct 30 01:17:02.783935 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:02.783761    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9058f9e-a7e7-4f30-87be-e037e5378b3c-istio-envoy\") pod \"e2e-shared-b-w2-a3619c-openshift-default-675f567c95-jsx2v\" (UID: \"b9058f9e-a7e7-4f30-87be-e037e5378b3c\") " pod="openshift-ingress/e2e-shared-b-w2-a3619c-openshift-default-675f567c95-jsx2v"
Oct 30 01:17:02.783935 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:02.783808    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9058f9e-a7e7-4f30-87be-e037e5378b3c-istio-podinfo\") pod \"e2e-shared-b-w2-a3619c-openshift-default-675f567c95-jsx2v\" (UID: \"b9058f9e-a7e7-4f30-87be-e037e5378b3c\") " pod="openshift-ingress/e2e-shared-b-w2-a3619c-openshift-default-675f567c95-jsx2v"
Oct 30 01:17:02.783935 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:02.783837    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9058f9e-a7e7-4f30-87be-e037e5378b3c-istiod-ca-cert\") pod \"e2e-shared-b-w2-a3619c-openshift-default-675f567c95-jsx2v\" (UID: \"b9058f9e-a7e7-4f30-87be-e037e5378b3c\") " pod="openshift-ingress/e2e-shared-b-w2-a3619c-openshift-default-675f567c95-jsx2v"
Oct 30 01:17:02.783935 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:02.783871    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9058f9e-a7e7-4f30-87be-e037e5378b3c-credential-socket\") pod \"e2e-shared-b-w2-a3619c-openshift-default-675f567c95-jsx2v\" (UID: \"b9058f9e-a7e7-4f30-87be-e037e5378b3c\") " pod="openshift-ingress/e2e-shared-b-w2-a3619c-openshift-default-675f567c95-jsx2v"
Oct 30 01:17:02.783935 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:02.783910    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9058f9e-a7e7-4f30-87be-e037e5378b3c-workload-socket\") pod \"e2e-shared-b-w2-a3619c-openshift-default-675f567c95-jsx2v\" (UID: \"b9058f9e-a7e7-4f30-87be-e037e5378b3c\") " pod="openshift-ingress/e2e-shared-b-w2-a3619c-openshift-default-675f567c95-jsx2v"
Oct 30 01:17:02.784324 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:02.783941    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9058f9e-a7e7-4f30-87be-e037e5378b3c-istio-token\") pod \"e2e-shared-b-w2-a3619c-openshift-default-675f567c95-jsx2v\" (UID: \"b9058f9e-a7e7-4f30-87be-e037e5378b3c\") " pod="openshift-ingress/e2e-shared-b-w2-a3619c-openshift-default-675f567c95-jsx2v"
Oct 30 01:17:02.784324 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:02.784064    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xhnc\" (UniqueName: \"kubernetes.io/projected/b9058f9e-a7e7-4f30-87be-e037e5378b3c-kube-api-access-7xhnc\") pod \"e2e-shared-b-w2-a3619c-openshift-default-675f567c95-jsx2v\" (UID: \"b9058f9e-a7e7-4f30-87be-e037e5378b3c\") " pod="openshift-ingress/e2e-shared-b-w2-a3619c-openshift-default-675f567c95-jsx2v"
Oct 30 01:17:02.784513 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:02.784434    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9058f9e-a7e7-4f30-87be-e037e5378b3c-istio-data\") pod \"e2e-shared-b-w2-a3619c-openshift-default-675f567c95-jsx2v\" (UID: \"b9058f9e-a7e7-4f30-87be-e037e5378b3c\") " pod="openshift-ingress/e2e-shared-b-w2-a3619c-openshift-default-675f567c95-jsx2v"
Oct 30 01:17:02.784942 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:02.784820    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9058f9e-a7e7-4f30-87be-e037e5378b3c-workload-socket\") pod \"e2e-shared-b-w2-a3619c-openshift-default-675f567c95-jsx2v\" (UID: \"b9058f9e-a7e7-4f30-87be-e037e5378b3c\") " pod="openshift-ingress/e2e-shared-b-w2-a3619c-openshift-default-675f567c95-jsx2v"
Oct 30 01:17:02.784942 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:02.784884    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9058f9e-a7e7-4f30-87be-e037e5378b3c-istio-data\") pod \"e2e-shared-b-w2-a3619c-openshift-default-675f567c95-jsx2v\" (UID: \"b9058f9e-a7e7-4f30-87be-e037e5378b3c\") " pod="openshift-ingress/e2e-shared-b-w2-a3619c-openshift-default-675f567c95-jsx2v"
Oct 30 01:17:02.785127 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:02.785065    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9058f9e-a7e7-4f30-87be-e037e5378b3c-istiod-ca-cert\") pod \"e2e-shared-b-w2-a3619c-openshift-default-675f567c95-jsx2v\" (UID: \"b9058f9e-a7e7-4f30-87be-e037e5378b3c\") " pod="openshift-ingress/e2e-shared-b-w2-a3619c-openshift-default-675f567c95-jsx2v"
Oct 30 01:17:02.785187 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:02.785128    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9058f9e-a7e7-4f30-87be-e037e5378b3c-credential-socket\") pod \"e2e-shared-b-w2-a3619c-openshift-default-675f567c95-jsx2v\" (UID: \"b9058f9e-a7e7-4f30-87be-e037e5378b3c\") " pod="openshift-ingress/e2e-shared-b-w2-a3619c-openshift-default-675f567c95-jsx2v"
Oct 30 01:17:02.785507 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:02.785422    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9058f9e-a7e7-4f30-87be-e037e5378b3c-workload-certs\") pod \"e2e-shared-b-w2-a3619c-openshift-default-675f567c95-jsx2v\" (UID: \"b9058f9e-a7e7-4f30-87be-e037e5378b3c\") " pod="openshift-ingress/e2e-shared-b-w2-a3619c-openshift-default-675f567c95-jsx2v"
Oct 30 01:17:02.787621 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:02.787600    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9058f9e-a7e7-4f30-87be-e037e5378b3c-istio-envoy\") pod \"e2e-shared-b-w2-a3619c-openshift-default-675f567c95-jsx2v\" (UID: \"b9058f9e-a7e7-4f30-87be-e037e5378b3c\") " pod="openshift-ingress/e2e-shared-b-w2-a3619c-openshift-default-675f567c95-jsx2v"
Oct 30 01:17:02.787839 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:02.787818    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9058f9e-a7e7-4f30-87be-e037e5378b3c-istio-podinfo\") pod \"e2e-shared-b-w2-a3619c-openshift-default-675f567c95-jsx2v\" (UID: \"b9058f9e-a7e7-4f30-87be-e037e5378b3c\") " pod="openshift-ingress/e2e-shared-b-w2-a3619c-openshift-default-675f567c95-jsx2v"
Oct 30 01:17:02.809607 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:02.809571    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9058f9e-a7e7-4f30-87be-e037e5378b3c-istio-token\") pod \"e2e-shared-b-w2-a3619c-openshift-default-675f567c95-jsx2v\" (UID: \"b9058f9e-a7e7-4f30-87be-e037e5378b3c\") " pod="openshift-ingress/e2e-shared-b-w2-a3619c-openshift-default-675f567c95-jsx2v"
Oct 30 01:17:02.809734 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:02.809706    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xhnc\" (UniqueName: \"kubernetes.io/projected/b9058f9e-a7e7-4f30-87be-e037e5378b3c-kube-api-access-7xhnc\") pod \"e2e-shared-b-w2-a3619c-openshift-default-675f567c95-jsx2v\" (UID: \"b9058f9e-a7e7-4f30-87be-e037e5378b3c\") " pod="openshift-ingress/e2e-shared-b-w2-a3619c-openshift-default-675f567c95-jsx2v"
Oct 30 01:17:02.907451 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:02.906729    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-a3619c-openshift-default-675f567c95-jsx2v"
Oct 30 01:17:03.248030 ip-10-0-128-7 kubenswrapper[5002]: W1030 01:17:03.247995    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9058f9e_a7e7_4f30_87be_e037e5378b3c.slice/crio-e62ecc26493257d537196a656760ba3f7bb18d69a2786687be6bee286711fd41 WatchSource:0}: Error finding container e62ecc26493257d537196a656760ba3f7bb18d69a2786687be6bee286711fd41: Status 404 returned error can't find the container with id e62ecc26493257d537196a656760ba3f7bb18d69a2786687be6bee286711fd41
Oct 30 01:17:03.248539 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:03.248511    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-a3619c-openshift-default-675f567c95-jsx2v"]
Oct 30 01:17:03.251681 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:03.251634    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:17:03.251786 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:03.251730    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:17:03.251786 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:03.251772    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:17:03.618908 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:03.618817    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-34f855e6-84xqh"]
Oct 30 01:17:03.623071 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:03.623050    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34f855e6-84xqh"
Oct 30 01:17:03.645989 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:03.645950    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-34f855e6-84xqh"]
Oct 30 01:17:03.663802 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:03.656666    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:17:03.663802 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:03.656763    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:17:03.663802 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:03.656879    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5phsh\""
Oct 30 01:17:03.663802 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:03.656951    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:17:03.706471 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:03.706431    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ccwsh\" (UniqueName: \"kubernetes.io/projected/8933c664-e833-4091-98c0-2e39fc21e321-kube-api-access-ccwsh\") pod \"maas-api-revoke-sub-34f855e6-84xqh\" (UID: \"8933c664-e833-4091-98c0-2e39fc21e321\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34f855e6-84xqh"
Oct 30 01:17:03.706651 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:03.706511    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8933c664-e833-4091-98c0-2e39fc21e321-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-34f855e6-84xqh\" (UID: \"8933c664-e833-4091-98c0-2e39fc21e321\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34f855e6-84xqh"
Oct 30 01:17:03.706710 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:03.706675    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8933c664-e833-4091-98c0-2e39fc21e321-maas-api-xks-ca\") pod \"maas-api-revoke-sub-34f855e6-84xqh\" (UID: \"8933c664-e833-4091-98c0-2e39fc21e321\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34f855e6-84xqh"
Oct 30 01:17:03.748330 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:03.748290    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-a3619c-openshift-default-675f567c95-jsx2v" event={"ID":"b9058f9e-a7e7-4f30-87be-e037e5378b3c","Type":"ContainerStarted","Data":"9e600f9dc55a970f153f67517d7cfcd8cc39a6ad43ac338a7c64cc453af40360"}
Oct 30 01:17:03.748565 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:03.748356    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-a3619c-openshift-default-675f567c95-jsx2v" event={"ID":"b9058f9e-a7e7-4f30-87be-e037e5378b3c","Type":"ContainerStarted","Data":"e62ecc26493257d537196a656760ba3f7bb18d69a2786687be6bee286711fd41"}
Oct 30 01:17:03.800925 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:03.800856    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-a3619c-openshift-default-675f567c95-jsx2v" podStartSLOduration=1.8008338099999999 podStartE2EDuration="1.80083381s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:17:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:17:03.797374955 +0000 UTC m=+2719.413807862" watchObservedRunningTime="2026-10-30 01:17:03.80083381 +0000 UTC m=+2719.417266695"
Oct 30 01:17:03.808471 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:03.808424    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccwsh\" (UniqueName: \"kubernetes.io/projected/8933c664-e833-4091-98c0-2e39fc21e321-kube-api-access-ccwsh\") pod \"maas-api-revoke-sub-34f855e6-84xqh\" (UID: \"8933c664-e833-4091-98c0-2e39fc21e321\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34f855e6-84xqh"
Oct 30 01:17:03.808660 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:03.808498    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8933c664-e833-4091-98c0-2e39fc21e321-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-34f855e6-84xqh\" (UID: \"8933c664-e833-4091-98c0-2e39fc21e321\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34f855e6-84xqh"
Oct 30 01:17:03.808739 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:03.808665    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8933c664-e833-4091-98c0-2e39fc21e321-maas-api-xks-ca\") pod \"maas-api-revoke-sub-34f855e6-84xqh\" (UID: \"8933c664-e833-4091-98c0-2e39fc21e321\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34f855e6-84xqh"
Oct 30 01:17:03.809239 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:03.809212    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8933c664-e833-4091-98c0-2e39fc21e321-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-34f855e6-84xqh\" (UID: \"8933c664-e833-4091-98c0-2e39fc21e321\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34f855e6-84xqh"
Oct 30 01:17:03.811656 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:03.811630    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8933c664-e833-4091-98c0-2e39fc21e321-maas-api-xks-ca\") pod \"maas-api-revoke-sub-34f855e6-84xqh\" (UID: \"8933c664-e833-4091-98c0-2e39fc21e321\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34f855e6-84xqh"
Oct 30 01:17:03.834458 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:03.834372    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ccwsh\" (UniqueName: \"kubernetes.io/projected/8933c664-e833-4091-98c0-2e39fc21e321-kube-api-access-ccwsh\") pod \"maas-api-revoke-sub-34f855e6-84xqh\" (UID: \"8933c664-e833-4091-98c0-2e39fc21e321\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34f855e6-84xqh"
Oct 30 01:17:03.907746 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:03.907657    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-a3619c-openshift-default-675f567c95-jsx2v"
Oct 30 01:17:03.941131 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:03.941093    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34f855e6-84xqh"
Oct 30 01:17:04.292630 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:04.292520    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-34f855e6-84xqh"]
Oct 30 01:17:04.297400 ip-10-0-128-7 kubenswrapper[5002]: W1030 01:17:04.297328    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8933c664_e833_4091_98c0_2e39fc21e321.slice/crio-30dedd0f507eb33288114086d69f141922636645d885a64dc2fbc8586df5f47c WatchSource:0}: Error finding container 30dedd0f507eb33288114086d69f141922636645d885a64dc2fbc8586df5f47c: Status 404 returned error can't find the container with id 30dedd0f507eb33288114086d69f141922636645d885a64dc2fbc8586df5f47c
Oct 30 01:17:04.789513 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:04.789468    5002 generic.go:356] "Generic (PLEG): container finished" podID="45d94cd6-d704-48c7-b1fb-e97c89e7d74b" containerID="3ffbaae21284766ab63e7484c47642a5f0e7df30c05526a2204493a9825c63c1" exitCode=0
Oct 30 01:17:04.790005 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:04.789541    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f6d5b8-openshift-default-68ddb87d77-n4kjt" event={"ID":"45d94cd6-d704-48c7-b1fb-e97c89e7d74b","Type":"ContainerDied","Data":"3ffbaae21284766ab63e7484c47642a5f0e7df30c05526a2204493a9825c63c1"}
Oct 30 01:17:04.790133 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:04.790087    5002 scope.go:122] "RemoveContainer" containerID="3ffbaae21284766ab63e7484c47642a5f0e7df30c05526a2204493a9825c63c1"
Oct 30 01:17:04.792237 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:04.792203    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:17:04.792406 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:04.792293    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:17:04.792406 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:04.792348    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:17:04.792542 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:04.792473    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34f855e6-84xqh" event={"ID":"8933c664-e833-4091-98c0-2e39fc21e321","Type":"ContainerStarted","Data":"7f91e196f6bf40bcafeb373523735a8df5da484e31b442f6abc950d7ba876573"}
Oct 30 01:17:04.792542 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:04.792522    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34f855e6-84xqh" event={"ID":"8933c664-e833-4091-98c0-2e39fc21e321","Type":"ContainerStarted","Data":"30dedd0f507eb33288114086d69f141922636645d885a64dc2fbc8586df5f47c"}
Oct 30 01:17:04.908465 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:04.905989    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34f855e6-84xqh" podStartSLOduration=1.905974157 podStartE2EDuration="1.905974157s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:17:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:17:04.905545778 +0000 UTC m=+2720.521978683" watchObservedRunningTime="2026-10-30 01:17:04.905974157 +0000 UTC m=+2720.522407036"
Oct 30 01:17:04.910677 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:04.910650    5002 patch_prober.go:29] interesting pod/e2e-shared-b-w2-a3619c-openshift-default-675f567c95-jsx2v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.255:15021/healthz/ready\": dial tcp 10.132.0.255:15021: connect: connection refused" start-of-body=
Oct 30 01:17:04.911724 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:04.911694    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-a3619c-openshift-default-675f567c95-jsx2v" podUID="b9058f9e-a7e7-4f30-87be-e037e5378b3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.255:15021/healthz/ready\": dial tcp 10.132.0.255:15021: connect: connection refused"
Oct 30 01:17:05.815789 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:05.815684    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f6d5b8-openshift-default-68ddb87d77-n4kjt" event={"ID":"45d94cd6-d704-48c7-b1fb-e97c89e7d74b","Type":"ContainerStarted","Data":"46a8d9dd43666738dcd4774c805c1e18802a96df9f50015c0add7defb66c9b6f"}
Oct 30 01:17:05.819473 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:05.819206    5002 generic.go:356] "Generic (PLEG): container finished" podID="8933c664-e833-4091-98c0-2e39fc21e321" containerID="7f91e196f6bf40bcafeb373523735a8df5da484e31b442f6abc950d7ba876573" exitCode=0
Oct 30 01:17:05.819473 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:05.819399    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34f855e6-84xqh" event={"ID":"8933c664-e833-4091-98c0-2e39fc21e321","Type":"ContainerDied","Data":"7f91e196f6bf40bcafeb373523735a8df5da484e31b442f6abc950d7ba876573"}
Oct 30 01:17:05.913819 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:05.913779    5002 patch_prober.go:29] interesting pod/e2e-shared-b-w2-a3619c-openshift-default-675f567c95-jsx2v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.255:15021/healthz/ready\": dial tcp 10.132.0.255:15021: connect: connection refused" start-of-body=
Oct 30 01:17:05.914100 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:05.914072    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-a3619c-openshift-default-675f567c95-jsx2v" podUID="b9058f9e-a7e7-4f30-87be-e037e5378b3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.255:15021/healthz/ready\": dial tcp 10.132.0.255:15021: connect: connection refused"
Oct 30 01:17:06.497404 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:06.496491    5002 patch_prober.go:29] interesting pod/e2e-ipp-a-f6d5b8-openshift-default-68ddb87d77-n4kjt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused" start-of-body=
Oct 30 01:17:06.497404 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:06.496569    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f6d5b8-openshift-default-68ddb87d77-n4kjt" podUID="45d94cd6-d704-48c7-b1fb-e97c89e7d74b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused"
Oct 30 01:17:06.518261 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:06.518228    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-f6d5b8-openshift-default-68ddb87d77-n4kjt"
Oct 30 01:17:06.909406 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:06.907851    5002 patch_prober.go:29] interesting pod/e2e-shared-b-w2-a3619c-openshift-default-675f567c95-jsx2v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.255:15021/healthz/ready\": dial tcp 10.132.0.255:15021: connect: connection refused" start-of-body=
Oct 30 01:17:06.909406 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:06.907913    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-a3619c-openshift-default-675f567c95-jsx2v" podUID="b9058f9e-a7e7-4f30-87be-e037e5378b3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.255:15021/healthz/ready\": dial tcp 10.132.0.255:15021: connect: connection refused"
Oct 30 01:17:07.228663 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:07.228629    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34f855e6-84xqh"
Oct 30 01:17:07.330409 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:07.329489    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8933c664-e833-4091-98c0-2e39fc21e321-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8933c664-e833-4091-98c0-2e39fc21e321-maas-api-openshift-ca\") pod \"8933c664-e833-4091-98c0-2e39fc21e321\" (UID: \"8933c664-e833-4091-98c0-2e39fc21e321\") "
Oct 30 01:17:07.330409 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:07.329565    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8933c664-e833-4091-98c0-2e39fc21e321-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8933c664-e833-4091-98c0-2e39fc21e321-maas-api-xks-ca\") pod \"8933c664-e833-4091-98c0-2e39fc21e321\" (UID: \"8933c664-e833-4091-98c0-2e39fc21e321\") "
Oct 30 01:17:07.330409 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:07.329810    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8933c664-e833-4091-98c0-2e39fc21e321-kube-api-access-ccwsh\" (UniqueName: \"kubernetes.io/projected/8933c664-e833-4091-98c0-2e39fc21e321-kube-api-access-ccwsh\") pod \"8933c664-e833-4091-98c0-2e39fc21e321\" (UID: \"8933c664-e833-4091-98c0-2e39fc21e321\") "
Oct 30 01:17:07.330409 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:07.330048    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8933c664-e833-4091-98c0-2e39fc21e321-maas-api-openshift-ca" pod "8933c664-e833-4091-98c0-2e39fc21e321" (UID: "8933c664-e833-4091-98c0-2e39fc21e321"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:17:07.330409 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:07.330412    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8933c664-e833-4091-98c0-2e39fc21e321-maas-api-openshift-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:17:07.337429 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:07.335323    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8933c664-e833-4091-98c0-2e39fc21e321-kube-api-access-ccwsh" pod "8933c664-e833-4091-98c0-2e39fc21e321" (UID: "8933c664-e833-4091-98c0-2e39fc21e321"). InnerVolumeSpecName "kube-api-access-ccwsh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:17:07.337598 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:07.337550    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8933c664-e833-4091-98c0-2e39fc21e321-maas-api-xks-ca" pod "8933c664-e833-4091-98c0-2e39fc21e321" (UID: "8933c664-e833-4091-98c0-2e39fc21e321"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:17:07.432370 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:07.432327    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8933c664-e833-4091-98c0-2e39fc21e321-maas-api-xks-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:17:07.432370 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:07.432358    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ccwsh\" (UniqueName: \"kubernetes.io/projected/8933c664-e833-4091-98c0-2e39fc21e321-kube-api-access-ccwsh\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:17:07.513406 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:07.512552    5002 patch_prober.go:29] interesting pod/e2e-ipp-a-f6d5b8-openshift-default-68ddb87d77-n4kjt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused" start-of-body=
Oct 30 01:17:07.513406 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:07.512602    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f6d5b8-openshift-default-68ddb87d77-n4kjt" podUID="45d94cd6-d704-48c7-b1fb-e97c89e7d74b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused"
Oct 30 01:17:07.840609 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:07.840491    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34f855e6-84xqh" event={"ID":"8933c664-e833-4091-98c0-2e39fc21e321","Type":"ContainerDied","Data":"30dedd0f507eb33288114086d69f141922636645d885a64dc2fbc8586df5f47c"}
Oct 30 01:17:07.840609 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:07.840546    5002 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="30dedd0f507eb33288114086d69f141922636645d885a64dc2fbc8586df5f47c"
Oct 30 01:17:07.840609 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:07.840507    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34f855e6-84xqh"
Oct 30 01:17:07.945404 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:07.924811    5002 patch_prober.go:29] interesting pod/e2e-shared-b-w2-a3619c-openshift-default-675f567c95-jsx2v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.255:15021/healthz/ready\": dial tcp 10.132.0.255:15021: connect: connection refused" start-of-body=
Oct 30 01:17:07.945404 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:07.924877    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-a3619c-openshift-default-675f567c95-jsx2v" podUID="b9058f9e-a7e7-4f30-87be-e037e5378b3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.255:15021/healthz/ready\": dial tcp 10.132.0.255:15021: connect: connection refused"
Oct 30 01:17:08.482402 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:08.481892    5002 patch_prober.go:29] interesting pod/e2e-ipp-a-f6d5b8-openshift-default-68ddb87d77-n4kjt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused" start-of-body=
Oct 30 01:17:08.482402 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:08.481971    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f6d5b8-openshift-default-68ddb87d77-n4kjt" podUID="45d94cd6-d704-48c7-b1fb-e97c89e7d74b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused"
Oct 30 01:17:08.908577 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:08.908483    5002 patch_prober.go:29] interesting pod/e2e-shared-b-w2-a3619c-openshift-default-675f567c95-jsx2v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.255:15021/healthz/ready\": dial tcp 10.132.0.255:15021: connect: connection refused" start-of-body=
Oct 30 01:17:08.908850 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:08.908818    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-a3619c-openshift-default-675f567c95-jsx2v" podUID="b9058f9e-a7e7-4f30-87be-e037e5378b3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.255:15021/healthz/ready\": dial tcp 10.132.0.255:15021: connect: connection refused"
Oct 30 01:17:09.482404 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:09.481791    5002 patch_prober.go:29] interesting pod/e2e-ipp-a-f6d5b8-openshift-default-68ddb87d77-n4kjt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused" start-of-body=
Oct 30 01:17:09.482404 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:09.481860    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f6d5b8-openshift-default-68ddb87d77-n4kjt" podUID="45d94cd6-d704-48c7-b1fb-e97c89e7d74b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused"
Oct 30 01:17:09.908092 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:09.908005    5002 patch_prober.go:29] interesting pod/e2e-shared-b-w2-a3619c-openshift-default-675f567c95-jsx2v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.255:15021/healthz/ready\": dial tcp 10.132.0.255:15021: connect: connection refused" start-of-body=
Oct 30 01:17:09.908092 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:09.908059    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-a3619c-openshift-default-675f567c95-jsx2v" podUID="b9058f9e-a7e7-4f30-87be-e037e5378b3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.255:15021/healthz/ready\": dial tcp 10.132.0.255:15021: connect: connection refused"
Oct 30 01:17:10.366540 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:10.366497    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-f6d5b8-openshift-default-68ddb87d77-n4kjt"
Oct 30 01:17:10.483850 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:10.483815    5002 patch_prober.go:29] interesting pod/e2e-ipp-a-f6d5b8-openshift-default-68ddb87d77-n4kjt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused" start-of-body=
Oct 30 01:17:10.484311 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:10.484281    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f6d5b8-openshift-default-68ddb87d77-n4kjt" podUID="45d94cd6-d704-48c7-b1fb-e97c89e7d74b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused"
Oct 30 01:17:10.912947 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:10.912908    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-a3619c-openshift-default-675f567c95-jsx2v"
Oct 30 01:17:10.913305 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:10.913284    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-a3619c-openshift-default-675f567c95-jsx2v"
Oct 30 01:17:10.913931 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:10.913910    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-a3619c-openshift-default-675f567c95-jsx2v"
Oct 30 01:17:11.506562 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:11.506521    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-f6d5b8-openshift-default-68ddb87d77-n4kjt"
Oct 30 01:17:11.508463 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:11.508437    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-f6d5b8-openshift-default-68ddb87d77-n4kjt"
Oct 30 01:17:12.862787 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:17:12.862743    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbjdq" containerName="extractor"
Oct 30 01:17:19.454104 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:19.454060    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w6-009e78/rate-test-model-009e78-kserve-65478f8b7c-4z5ns"]
Oct 30 01:17:21.683514 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:21.683477    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-x66jp"]
Oct 30 01:17:21.684010 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:21.683689    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-x66jp" podUID="37340929-9f26-4db9-ad7d-66d0bd14a7cd" containerName="istio-proxy" containerID="cri-o://4e3d340fe1ad2d5300d74d0c655c804d66f0fd0a4e3a8c9b07a631aa16d34830" gracePeriod=30
Oct 30 01:17:22.976960 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:22.976910    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-cce513-plbvz"]
Oct 30 01:17:22.996906 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:22.996862    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-cce513-plbvz"]
Oct 30 01:17:24.497859 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:24.497825    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d22b0df-6481-4817-b64d-88466a69bceb" path="/var/lib/kubelet/pods/3d22b0df-6481-4817-b64d-88466a69bceb/volumes"
Oct 30 01:17:26.348763 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:26.348679    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-52650f-x22xt"]
Oct 30 01:17:26.364429 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:26.364374    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-52650f-x22xt"]
Oct 30 01:17:26.496283 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:26.496240    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc526934-1cca-481c-bfe8-1b9bce696916" path="/var/lib/kubelet/pods/cc526934-1cca-481c-bfe8-1b9bce696916/volumes"
Oct 30 01:17:27.001969 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:27.001941    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-x66jp"
Oct 30 01:17:27.058281 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:27.058235    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37340929-9f26-4db9-ad7d-66d0bd14a7cd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37340929-9f26-4db9-ad7d-66d0bd14a7cd-istio-envoy\") pod \"37340929-9f26-4db9-ad7d-66d0bd14a7cd\" (UID: \"37340929-9f26-4db9-ad7d-66d0bd14a7cd\") "
Oct 30 01:17:27.058493 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:27.058349    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37340929-9f26-4db9-ad7d-66d0bd14a7cd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37340929-9f26-4db9-ad7d-66d0bd14a7cd-credential-socket\") pod \"37340929-9f26-4db9-ad7d-66d0bd14a7cd\" (UID: \"37340929-9f26-4db9-ad7d-66d0bd14a7cd\") "
Oct 30 01:17:27.058493 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:27.058413    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37340929-9f26-4db9-ad7d-66d0bd14a7cd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37340929-9f26-4db9-ad7d-66d0bd14a7cd-workload-socket\") pod \"37340929-9f26-4db9-ad7d-66d0bd14a7cd\" (UID: \"37340929-9f26-4db9-ad7d-66d0bd14a7cd\") "
Oct 30 01:17:27.058493 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:27.058463    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37340929-9f26-4db9-ad7d-66d0bd14a7cd-kube-api-access-lbx6c\" (UniqueName: \"kubernetes.io/projected/37340929-9f26-4db9-ad7d-66d0bd14a7cd-kube-api-access-lbx6c\") pod \"37340929-9f26-4db9-ad7d-66d0bd14a7cd\" (UID: \"37340929-9f26-4db9-ad7d-66d0bd14a7cd\") "
Oct 30 01:17:27.058694 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:27.058514    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37340929-9f26-4db9-ad7d-66d0bd14a7cd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37340929-9f26-4db9-ad7d-66d0bd14a7cd-istio-data\") pod \"37340929-9f26-4db9-ad7d-66d0bd14a7cd\" (UID: \"37340929-9f26-4db9-ad7d-66d0bd14a7cd\") "
Oct 30 01:17:27.058694 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:27.058634    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37340929-9f26-4db9-ad7d-66d0bd14a7cd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37340929-9f26-4db9-ad7d-66d0bd14a7cd-workload-certs\") pod \"37340929-9f26-4db9-ad7d-66d0bd14a7cd\" (UID: \"37340929-9f26-4db9-ad7d-66d0bd14a7cd\") "
Oct 30 01:17:27.058801 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:27.058724    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37340929-9f26-4db9-ad7d-66d0bd14a7cd-credential-socket" pod "37340929-9f26-4db9-ad7d-66d0bd14a7cd" (UID: "37340929-9f26-4db9-ad7d-66d0bd14a7cd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:17:27.058801 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:27.058743    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37340929-9f26-4db9-ad7d-66d0bd14a7cd-istio-token\" (UniqueName: \"kubernetes.io/projected/37340929-9f26-4db9-ad7d-66d0bd14a7cd-istio-token\") pod \"37340929-9f26-4db9-ad7d-66d0bd14a7cd\" (UID: \"37340929-9f26-4db9-ad7d-66d0bd14a7cd\") "
Oct 30 01:17:27.059161 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:27.058799    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37340929-9f26-4db9-ad7d-66d0bd14a7cd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37340929-9f26-4db9-ad7d-66d0bd14a7cd-istiod-ca-cert\") pod \"37340929-9f26-4db9-ad7d-66d0bd14a7cd\" (UID: \"37340929-9f26-4db9-ad7d-66d0bd14a7cd\") "
Oct 30 01:17:27.059161 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:27.058879    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/37340929-9f26-4db9-ad7d-66d0bd14a7cd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37340929-9f26-4db9-ad7d-66d0bd14a7cd-istio-podinfo\") pod \"37340929-9f26-4db9-ad7d-66d0bd14a7cd\" (UID: \"37340929-9f26-4db9-ad7d-66d0bd14a7cd\") "
Oct 30 01:17:27.059653 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:27.059633    5002 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37340929-9f26-4db9-ad7d-66d0bd14a7cd-credential-socket\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:17:27.059756 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:27.059627    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37340929-9f26-4db9-ad7d-66d0bd14a7cd-workload-socket" pod "37340929-9f26-4db9-ad7d-66d0bd14a7cd" (UID: "37340929-9f26-4db9-ad7d-66d0bd14a7cd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:17:27.059983 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:27.059962    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37340929-9f26-4db9-ad7d-66d0bd14a7cd-workload-certs" pod "37340929-9f26-4db9-ad7d-66d0bd14a7cd" (UID: "37340929-9f26-4db9-ad7d-66d0bd14a7cd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:17:27.060111 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:27.060009    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37340929-9f26-4db9-ad7d-66d0bd14a7cd-istiod-ca-cert" pod "37340929-9f26-4db9-ad7d-66d0bd14a7cd" (UID: "37340929-9f26-4db9-ad7d-66d0bd14a7cd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:17:27.060358 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:27.060333    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37340929-9f26-4db9-ad7d-66d0bd14a7cd-istio-data" pod "37340929-9f26-4db9-ad7d-66d0bd14a7cd" (UID: "37340929-9f26-4db9-ad7d-66d0bd14a7cd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:17:27.061020 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:27.060998    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37340929-9f26-4db9-ad7d-66d0bd14a7cd-istio-envoy" pod "37340929-9f26-4db9-ad7d-66d0bd14a7cd" (UID: "37340929-9f26-4db9-ad7d-66d0bd14a7cd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:17:27.062092 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:27.062070    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/37340929-9f26-4db9-ad7d-66d0bd14a7cd-istio-podinfo" pod "37340929-9f26-4db9-ad7d-66d0bd14a7cd" (UID: "37340929-9f26-4db9-ad7d-66d0bd14a7cd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:17:27.062467 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:27.062432    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37340929-9f26-4db9-ad7d-66d0bd14a7cd-istio-token" pod "37340929-9f26-4db9-ad7d-66d0bd14a7cd" (UID: "37340929-9f26-4db9-ad7d-66d0bd14a7cd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:17:27.062843 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:27.062817    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37340929-9f26-4db9-ad7d-66d0bd14a7cd-kube-api-access-lbx6c" pod "37340929-9f26-4db9-ad7d-66d0bd14a7cd" (UID: "37340929-9f26-4db9-ad7d-66d0bd14a7cd"). InnerVolumeSpecName "kube-api-access-lbx6c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:17:27.161592 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:27.161544    5002 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37340929-9f26-4db9-ad7d-66d0bd14a7cd-istio-data\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:17:27.161592 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:27.161584    5002 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37340929-9f26-4db9-ad7d-66d0bd14a7cd-workload-certs\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:17:27.161592 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:27.161598    5002 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37340929-9f26-4db9-ad7d-66d0bd14a7cd-istio-token\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:17:27.161907 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:27.161611    5002 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37340929-9f26-4db9-ad7d-66d0bd14a7cd-istiod-ca-cert\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:17:27.161907 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:27.161625    5002 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37340929-9f26-4db9-ad7d-66d0bd14a7cd-istio-podinfo\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:17:27.161907 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:27.161638    5002 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37340929-9f26-4db9-ad7d-66d0bd14a7cd-istio-envoy\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:17:27.161907 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:27.161650    5002 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37340929-9f26-4db9-ad7d-66d0bd14a7cd-workload-socket\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:17:27.161907 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:27.161664    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lbx6c\" (UniqueName: \"kubernetes.io/projected/37340929-9f26-4db9-ad7d-66d0bd14a7cd-kube-api-access-lbx6c\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:17:27.166784 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:27.166747    5002 generic.go:356] "Generic (PLEG): container finished" podID="37340929-9f26-4db9-ad7d-66d0bd14a7cd" containerID="4e3d340fe1ad2d5300d74d0c655c804d66f0fd0a4e3a8c9b07a631aa16d34830" exitCode=0
Oct 30 01:17:27.166971 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:27.166839    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-x66jp" event={"ID":"37340929-9f26-4db9-ad7d-66d0bd14a7cd","Type":"ContainerDied","Data":"4e3d340fe1ad2d5300d74d0c655c804d66f0fd0a4e3a8c9b07a631aa16d34830"}
Oct 30 01:17:27.166971 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:27.166884    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-x66jp" event={"ID":"37340929-9f26-4db9-ad7d-66d0bd14a7cd","Type":"ContainerDied","Data":"cd1c33552899d0f1ef5bf878847a2befa2d20871324353534a73c5417bfb1aa4"}
Oct 30 01:17:27.166971 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:27.166915    5002 scope.go:122] "RemoveContainer" containerID="4e3d340fe1ad2d5300d74d0c655c804d66f0fd0a4e3a8c9b07a631aa16d34830"
Oct 30 01:17:27.167126 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:27.167095    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-x66jp"
Oct 30 01:17:27.191956 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:27.191927    5002 scope.go:122] "RemoveContainer" containerID="4e3d340fe1ad2d5300d74d0c655c804d66f0fd0a4e3a8c9b07a631aa16d34830"
Oct 30 01:17:27.192374 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:17:27.192352    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e3d340fe1ad2d5300d74d0c655c804d66f0fd0a4e3a8c9b07a631aa16d34830\": container with ID starting with 4e3d340fe1ad2d5300d74d0c655c804d66f0fd0a4e3a8c9b07a631aa16d34830 not found: ID does not exist" containerID="4e3d340fe1ad2d5300d74d0c655c804d66f0fd0a4e3a8c9b07a631aa16d34830"
Oct 30 01:17:27.192374 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:27.192404    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e3d340fe1ad2d5300d74d0c655c804d66f0fd0a4e3a8c9b07a631aa16d34830"} err="failed to get container status \"4e3d340fe1ad2d5300d74d0c655c804d66f0fd0a4e3a8c9b07a631aa16d34830\": rpc error: code = NotFound desc = could not find container \"4e3d340fe1ad2d5300d74d0c655c804d66f0fd0a4e3a8c9b07a631aa16d34830\": container with ID starting with 4e3d340fe1ad2d5300d74d0c655c804d66f0fd0a4e3a8c9b07a631aa16d34830 not found: ID does not exist"
Oct 30 01:17:27.232925 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:27.232826    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-x66jp"]
Oct 30 01:17:27.237034 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:27.237004    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-x66jp"]
Oct 30 01:17:28.496887 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:28.496848    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37340929-9f26-4db9-ad7d-66d0bd14a7cd" path="/var/lib/kubelet/pods/37340929-9f26-4db9-ad7d-66d0bd14a7cd/volumes"
Oct 30 01:17:29.736610 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:29.736570    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-b73232-openshift-default-59c5464ccc-2kpc7"]
Oct 30 01:17:29.737498 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:29.737467    5002 cpu_manager.go:414] "containerMap: removing container" podUID="8933c664-e833-4091-98c0-2e39fc21e321" containerName="revoke-keys"
Oct 30 01:17:29.737498 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:29.737496    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8933c664-e833-4091-98c0-2e39fc21e321" containerName="revoke-keys"
Oct 30 01:17:29.737800 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:29.737521    5002 cpu_manager.go:414] "containerMap: removing container" podUID="37340929-9f26-4db9-ad7d-66d0bd14a7cd" containerName="istio-proxy"
Oct 30 01:17:29.737800 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:29.737531    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37340929-9f26-4db9-ad7d-66d0bd14a7cd" containerName="istio-proxy"
Oct 30 01:17:29.737800 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:29.737717    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="8933c664-e833-4091-98c0-2e39fc21e321" containerName="revoke-keys"
Oct 30 01:17:29.737800 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:29.737735    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="37340929-9f26-4db9-ad7d-66d0bd14a7cd" containerName="istio-proxy"
Oct 30 01:17:29.743942 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:29.743118    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-b73232-openshift-default-59c5464ccc-2kpc7"
Oct 30 01:17:29.760902 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:29.760861    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-b73232-openshift-default-dockercfg-xzswd\""
Oct 30 01:17:29.788000 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:29.787965    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-b73232-openshift-default-59c5464ccc-2kpc7"]
Oct 30 01:17:29.834154 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:29.834093    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c2a594b-3bbf-4c94-8a9f-7b79c663f135-istiod-ca-cert\") pod \"e2e-ipp-a-b73232-openshift-default-59c5464ccc-2kpc7\" (UID: \"8c2a594b-3bbf-4c94-8a9f-7b79c663f135\") " pod="openshift-ingress/e2e-ipp-a-b73232-openshift-default-59c5464ccc-2kpc7"
Oct 30 01:17:29.834363 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:29.834240    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c2a594b-3bbf-4c94-8a9f-7b79c663f135-istio-envoy\") pod \"e2e-ipp-a-b73232-openshift-default-59c5464ccc-2kpc7\" (UID: \"8c2a594b-3bbf-4c94-8a9f-7b79c663f135\") " pod="openshift-ingress/e2e-ipp-a-b73232-openshift-default-59c5464ccc-2kpc7"
Oct 30 01:17:29.834363 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:29.834279    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c2a594b-3bbf-4c94-8a9f-7b79c663f135-istio-token\") pod \"e2e-ipp-a-b73232-openshift-default-59c5464ccc-2kpc7\" (UID: \"8c2a594b-3bbf-4c94-8a9f-7b79c663f135\") " pod="openshift-ingress/e2e-ipp-a-b73232-openshift-default-59c5464ccc-2kpc7"
Oct 30 01:17:29.834363 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:29.834358    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c2a594b-3bbf-4c94-8a9f-7b79c663f135-istio-data\") pod \"e2e-ipp-a-b73232-openshift-default-59c5464ccc-2kpc7\" (UID: \"8c2a594b-3bbf-4c94-8a9f-7b79c663f135\") " pod="openshift-ingress/e2e-ipp-a-b73232-openshift-default-59c5464ccc-2kpc7"
Oct 30 01:17:29.834565 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:29.834426    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jj6kb\" (UniqueName: \"kubernetes.io/projected/8c2a594b-3bbf-4c94-8a9f-7b79c663f135-kube-api-access-jj6kb\") pod \"e2e-ipp-a-b73232-openshift-default-59c5464ccc-2kpc7\" (UID: \"8c2a594b-3bbf-4c94-8a9f-7b79c663f135\") " pod="openshift-ingress/e2e-ipp-a-b73232-openshift-default-59c5464ccc-2kpc7"
Oct 30 01:17:29.834565 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:29.834450    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c2a594b-3bbf-4c94-8a9f-7b79c663f135-credential-socket\") pod \"e2e-ipp-a-b73232-openshift-default-59c5464ccc-2kpc7\" (UID: \"8c2a594b-3bbf-4c94-8a9f-7b79c663f135\") " pod="openshift-ingress/e2e-ipp-a-b73232-openshift-default-59c5464ccc-2kpc7"
Oct 30 01:17:29.834565 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:29.834493    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c2a594b-3bbf-4c94-8a9f-7b79c663f135-workload-certs\") pod \"e2e-ipp-a-b73232-openshift-default-59c5464ccc-2kpc7\" (UID: \"8c2a594b-3bbf-4c94-8a9f-7b79c663f135\") " pod="openshift-ingress/e2e-ipp-a-b73232-openshift-default-59c5464ccc-2kpc7"
Oct 30 01:17:29.834565 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:29.834521    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c2a594b-3bbf-4c94-8a9f-7b79c663f135-istio-podinfo\") pod \"e2e-ipp-a-b73232-openshift-default-59c5464ccc-2kpc7\" (UID: \"8c2a594b-3bbf-4c94-8a9f-7b79c663f135\") " pod="openshift-ingress/e2e-ipp-a-b73232-openshift-default-59c5464ccc-2kpc7"
Oct 30 01:17:29.834707 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:29.834567    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c2a594b-3bbf-4c94-8a9f-7b79c663f135-workload-socket\") pod \"e2e-ipp-a-b73232-openshift-default-59c5464ccc-2kpc7\" (UID: \"8c2a594b-3bbf-4c94-8a9f-7b79c663f135\") " pod="openshift-ingress/e2e-ipp-a-b73232-openshift-default-59c5464ccc-2kpc7"
Oct 30 01:17:29.935437 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:29.935398    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c2a594b-3bbf-4c94-8a9f-7b79c663f135-istio-envoy\") pod \"e2e-ipp-a-b73232-openshift-default-59c5464ccc-2kpc7\" (UID: \"8c2a594b-3bbf-4c94-8a9f-7b79c663f135\") " pod="openshift-ingress/e2e-ipp-a-b73232-openshift-default-59c5464ccc-2kpc7"
Oct 30 01:17:29.935642 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:29.935458    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c2a594b-3bbf-4c94-8a9f-7b79c663f135-istio-token\") pod \"e2e-ipp-a-b73232-openshift-default-59c5464ccc-2kpc7\" (UID: \"8c2a594b-3bbf-4c94-8a9f-7b79c663f135\") " pod="openshift-ingress/e2e-ipp-a-b73232-openshift-default-59c5464ccc-2kpc7"
Oct 30 01:17:29.935642 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:29.935596    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c2a594b-3bbf-4c94-8a9f-7b79c663f135-istio-data\") pod \"e2e-ipp-a-b73232-openshift-default-59c5464ccc-2kpc7\" (UID: \"8c2a594b-3bbf-4c94-8a9f-7b79c663f135\") " pod="openshift-ingress/e2e-ipp-a-b73232-openshift-default-59c5464ccc-2kpc7"
Oct 30 01:17:29.935779 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:29.935675    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jj6kb\" (UniqueName: \"kubernetes.io/projected/8c2a594b-3bbf-4c94-8a9f-7b79c663f135-kube-api-access-jj6kb\") pod \"e2e-ipp-a-b73232-openshift-default-59c5464ccc-2kpc7\" (UID: \"8c2a594b-3bbf-4c94-8a9f-7b79c663f135\") " pod="openshift-ingress/e2e-ipp-a-b73232-openshift-default-59c5464ccc-2kpc7"
Oct 30 01:17:29.935779 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:29.935709    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c2a594b-3bbf-4c94-8a9f-7b79c663f135-credential-socket\") pod \"e2e-ipp-a-b73232-openshift-default-59c5464ccc-2kpc7\" (UID: \"8c2a594b-3bbf-4c94-8a9f-7b79c663f135\") " pod="openshift-ingress/e2e-ipp-a-b73232-openshift-default-59c5464ccc-2kpc7"
Oct 30 01:17:29.935887 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:29.935782    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c2a594b-3bbf-4c94-8a9f-7b79c663f135-workload-certs\") pod \"e2e-ipp-a-b73232-openshift-default-59c5464ccc-2kpc7\" (UID: \"8c2a594b-3bbf-4c94-8a9f-7b79c663f135\") " pod="openshift-ingress/e2e-ipp-a-b73232-openshift-default-59c5464ccc-2kpc7"
Oct 30 01:17:29.935887 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:29.935811    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c2a594b-3bbf-4c94-8a9f-7b79c663f135-istio-podinfo\") pod \"e2e-ipp-a-b73232-openshift-default-59c5464ccc-2kpc7\" (UID: \"8c2a594b-3bbf-4c94-8a9f-7b79c663f135\") " pod="openshift-ingress/e2e-ipp-a-b73232-openshift-default-59c5464ccc-2kpc7"
Oct 30 01:17:29.936011 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:29.935885    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c2a594b-3bbf-4c94-8a9f-7b79c663f135-workload-socket\") pod \"e2e-ipp-a-b73232-openshift-default-59c5464ccc-2kpc7\" (UID: \"8c2a594b-3bbf-4c94-8a9f-7b79c663f135\") " pod="openshift-ingress/e2e-ipp-a-b73232-openshift-default-59c5464ccc-2kpc7"
Oct 30 01:17:29.936076 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:29.936007    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c2a594b-3bbf-4c94-8a9f-7b79c663f135-istiod-ca-cert\") pod \"e2e-ipp-a-b73232-openshift-default-59c5464ccc-2kpc7\" (UID: \"8c2a594b-3bbf-4c94-8a9f-7b79c663f135\") " pod="openshift-ingress/e2e-ipp-a-b73232-openshift-default-59c5464ccc-2kpc7"
Oct 30 01:17:29.936505 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:29.936479    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c2a594b-3bbf-4c94-8a9f-7b79c663f135-credential-socket\") pod \"e2e-ipp-a-b73232-openshift-default-59c5464ccc-2kpc7\" (UID: \"8c2a594b-3bbf-4c94-8a9f-7b79c663f135\") " pod="openshift-ingress/e2e-ipp-a-b73232-openshift-default-59c5464ccc-2kpc7"
Oct 30 01:17:29.936621 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:29.936479    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c2a594b-3bbf-4c94-8a9f-7b79c663f135-workload-certs\") pod \"e2e-ipp-a-b73232-openshift-default-59c5464ccc-2kpc7\" (UID: \"8c2a594b-3bbf-4c94-8a9f-7b79c663f135\") " pod="openshift-ingress/e2e-ipp-a-b73232-openshift-default-59c5464ccc-2kpc7"
Oct 30 01:17:29.936621 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:29.936563    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c2a594b-3bbf-4c94-8a9f-7b79c663f135-istio-data\") pod \"e2e-ipp-a-b73232-openshift-default-59c5464ccc-2kpc7\" (UID: \"8c2a594b-3bbf-4c94-8a9f-7b79c663f135\") " pod="openshift-ingress/e2e-ipp-a-b73232-openshift-default-59c5464ccc-2kpc7"
Oct 30 01:17:29.936762 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:29.936719    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c2a594b-3bbf-4c94-8a9f-7b79c663f135-workload-socket\") pod \"e2e-ipp-a-b73232-openshift-default-59c5464ccc-2kpc7\" (UID: \"8c2a594b-3bbf-4c94-8a9f-7b79c663f135\") " pod="openshift-ingress/e2e-ipp-a-b73232-openshift-default-59c5464ccc-2kpc7"
Oct 30 01:17:29.937354 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:29.937330    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c2a594b-3bbf-4c94-8a9f-7b79c663f135-istiod-ca-cert\") pod \"e2e-ipp-a-b73232-openshift-default-59c5464ccc-2kpc7\" (UID: \"8c2a594b-3bbf-4c94-8a9f-7b79c663f135\") " pod="openshift-ingress/e2e-ipp-a-b73232-openshift-default-59c5464ccc-2kpc7"
Oct 30 01:17:29.938009 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:29.937988    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c2a594b-3bbf-4c94-8a9f-7b79c663f135-istio-envoy\") pod \"e2e-ipp-a-b73232-openshift-default-59c5464ccc-2kpc7\" (UID: \"8c2a594b-3bbf-4c94-8a9f-7b79c663f135\") " pod="openshift-ingress/e2e-ipp-a-b73232-openshift-default-59c5464ccc-2kpc7"
Oct 30 01:17:29.939254 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:29.939226    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c2a594b-3bbf-4c94-8a9f-7b79c663f135-istio-podinfo\") pod \"e2e-ipp-a-b73232-openshift-default-59c5464ccc-2kpc7\" (UID: \"8c2a594b-3bbf-4c94-8a9f-7b79c663f135\") " pod="openshift-ingress/e2e-ipp-a-b73232-openshift-default-59c5464ccc-2kpc7"
Oct 30 01:17:29.961710 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:29.961679    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c2a594b-3bbf-4c94-8a9f-7b79c663f135-istio-token\") pod \"e2e-ipp-a-b73232-openshift-default-59c5464ccc-2kpc7\" (UID: \"8c2a594b-3bbf-4c94-8a9f-7b79c663f135\") " pod="openshift-ingress/e2e-ipp-a-b73232-openshift-default-59c5464ccc-2kpc7"
Oct 30 01:17:29.961919 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:29.961865    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jj6kb\" (UniqueName: \"kubernetes.io/projected/8c2a594b-3bbf-4c94-8a9f-7b79c663f135-kube-api-access-jj6kb\") pod \"e2e-ipp-a-b73232-openshift-default-59c5464ccc-2kpc7\" (UID: \"8c2a594b-3bbf-4c94-8a9f-7b79c663f135\") " pod="openshift-ingress/e2e-ipp-a-b73232-openshift-default-59c5464ccc-2kpc7"
Oct 30 01:17:30.066302 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:30.065898    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-b73232-openshift-default-59c5464ccc-2kpc7"
Oct 30 01:17:30.724462 ip-10-0-128-7 kubenswrapper[5002]: W1030 01:17:30.724426    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c2a594b_3bbf_4c94_8a9f_7b79c663f135.slice/crio-d2f99feff5edcfe93cfb3b1b5f1d9dd7a5ed2a65f48529e130f59cf413bb7991 WatchSource:0}: Error finding container d2f99feff5edcfe93cfb3b1b5f1d9dd7a5ed2a65f48529e130f59cf413bb7991: Status 404 returned error can't find the container with id d2f99feff5edcfe93cfb3b1b5f1d9dd7a5ed2a65f48529e130f59cf413bb7991
Oct 30 01:17:30.724789 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:30.724769    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-b73232-openshift-default-59c5464ccc-2kpc7"]
Oct 30 01:17:30.727140 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:30.727101    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:17:30.727269 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:30.727181    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:17:30.727269 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:30.727231    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:17:31.268133 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:31.268089    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-b73232-openshift-default-59c5464ccc-2kpc7" event={"ID":"8c2a594b-3bbf-4c94-8a9f-7b79c663f135","Type":"ContainerStarted","Data":"bbec892f4b363921edf12e2be788aa12e7a2b4eda9645e63b5ecb9fdadb3d557"}
Oct 30 01:17:31.268806 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:31.268773    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-b73232-openshift-default-59c5464ccc-2kpc7" event={"ID":"8c2a594b-3bbf-4c94-8a9f-7b79c663f135","Type":"ContainerStarted","Data":"d2f99feff5edcfe93cfb3b1b5f1d9dd7a5ed2a65f48529e130f59cf413bb7991"}
Oct 30 01:17:31.322144 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:31.322071    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-b73232-openshift-default-59c5464ccc-2kpc7" podStartSLOduration=2.322051439 podStartE2EDuration="2.322051439s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:17:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:17:31.318928425 +0000 UTC m=+2746.935361304" watchObservedRunningTime="2026-10-30 01:17:31.322051439 +0000 UTC m=+2746.938484318"
Oct 30 01:17:31.504403 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:31.501537    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-29be73-rr7m4"]
Oct 30 01:17:31.522847 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:31.522771    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-29be73-rr7m4"
Oct 30 01:17:31.534298 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:31.529058    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-29be73-rr7m4"]
Oct 30 01:17:31.546081 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:31.544269    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:17:31.546081 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:31.544361    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:17:31.546081 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:31.545819    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5phsh\""
Oct 30 01:17:31.553445 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:31.550028    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:17:31.649096 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:31.649058    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2bg5\" (UniqueName: \"kubernetes.io/projected/3df7d750-133f-4d48-a34e-d425ea88031e-kube-api-access-p2bg5\") pod \"maas-api-revoke-keys-e2e-worker-w1-29be73-rr7m4\" (UID: \"3df7d750-133f-4d48-a34e-d425ea88031e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-29be73-rr7m4"
Oct 30 01:17:31.649305 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:31.649274    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3df7d750-133f-4d48-a34e-d425ea88031e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-29be73-rr7m4\" (UID: \"3df7d750-133f-4d48-a34e-d425ea88031e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-29be73-rr7m4"
Oct 30 01:17:31.649504 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:31.649486    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3df7d750-133f-4d48-a34e-d425ea88031e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-29be73-rr7m4\" (UID: \"3df7d750-133f-4d48-a34e-d425ea88031e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-29be73-rr7m4"
Oct 30 01:17:31.751635 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:31.751277    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3df7d750-133f-4d48-a34e-d425ea88031e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-29be73-rr7m4\" (UID: \"3df7d750-133f-4d48-a34e-d425ea88031e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-29be73-rr7m4"
Oct 30 01:17:31.751635 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:31.751371    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2bg5\" (UniqueName: \"kubernetes.io/projected/3df7d750-133f-4d48-a34e-d425ea88031e-kube-api-access-p2bg5\") pod \"maas-api-revoke-keys-e2e-worker-w1-29be73-rr7m4\" (UID: \"3df7d750-133f-4d48-a34e-d425ea88031e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-29be73-rr7m4"
Oct 30 01:17:31.751635 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:31.751608    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3df7d750-133f-4d48-a34e-d425ea88031e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-29be73-rr7m4\" (UID: \"3df7d750-133f-4d48-a34e-d425ea88031e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-29be73-rr7m4"
Oct 30 01:17:31.753413 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:31.752643    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3df7d750-133f-4d48-a34e-d425ea88031e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-29be73-rr7m4\" (UID: \"3df7d750-133f-4d48-a34e-d425ea88031e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-29be73-rr7m4"
Oct 30 01:17:31.756325 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:31.756294    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3df7d750-133f-4d48-a34e-d425ea88031e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-29be73-rr7m4\" (UID: \"3df7d750-133f-4d48-a34e-d425ea88031e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-29be73-rr7m4"
Oct 30 01:17:31.779304 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:31.779217    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2bg5\" (UniqueName: \"kubernetes.io/projected/3df7d750-133f-4d48-a34e-d425ea88031e-kube-api-access-p2bg5\") pod \"maas-api-revoke-keys-e2e-worker-w1-29be73-rr7m4\" (UID: \"3df7d750-133f-4d48-a34e-d425ea88031e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-29be73-rr7m4"
Oct 30 01:17:31.852791 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:31.852751    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-29be73-rr7m4"
Oct 30 01:17:32.069460 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:32.067113    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-b73232-openshift-default-59c5464ccc-2kpc7"
Oct 30 01:17:32.077416 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:32.075751    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-b73232-openshift-default-59c5464ccc-2kpc7"
Oct 30 01:17:32.170145 ip-10-0-128-7 kubenswrapper[5002]: W1030 01:17:32.170103    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3df7d750_133f_4d48_a34e_d425ea88031e.slice/crio-773f462507882898b5105b6c34af134b74974abcba48d6e66368d4a13389eb2a WatchSource:0}: Error finding container 773f462507882898b5105b6c34af134b74974abcba48d6e66368d4a13389eb2a: Status 404 returned error can't find the container with id 773f462507882898b5105b6c34af134b74974abcba48d6e66368d4a13389eb2a
Oct 30 01:17:32.170145 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:32.170134    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-29be73-rr7m4"]
Oct 30 01:17:32.355412 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:32.344444    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-29be73-rr7m4" event={"ID":"3df7d750-133f-4d48-a34e-d425ea88031e","Type":"ContainerStarted","Data":"773f462507882898b5105b6c34af134b74974abcba48d6e66368d4a13389eb2a"}
Oct 30 01:17:32.355412 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:32.353012    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-b73232-openshift-default-59c5464ccc-2kpc7"
Oct 30 01:17:32.373067 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:32.373012    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-b73232-openshift-default-59c5464ccc-2kpc7"
Oct 30 01:17:33.349329 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:33.349288    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-29be73-rr7m4" event={"ID":"3df7d750-133f-4d48-a34e-d425ea88031e","Type":"ContainerStarted","Data":"627ca91397436ec12df8ab9e25c36229328b229be58cff13224b8234796915c3"}
Oct 30 01:17:33.396291 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:33.396190    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-29be73-rr7m4" podStartSLOduration=2.3961698350000002 podStartE2EDuration="2.396169835s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:17:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:17:33.392612391 +0000 UTC m=+2749.009045272" watchObservedRunningTime="2026-10-30 01:17:33.396169835 +0000 UTC m=+2749.012602717"
Oct 30 01:17:33.580682 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:33.580641    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-b7vtt"]
Oct 30 01:17:33.581282 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:33.581226    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-b7vtt" podUID="43b2d112-3ccc-448f-8f03-e6e15fccfa4c" containerName="istio-proxy" containerID="cri-o://ce3946fa6c193d416d428d807339ac323f37e7f1d1aabaa78c3c38a8524ae8dd" gracePeriod=30
Oct 30 01:17:34.362429 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:34.362345    5002 generic.go:356] "Generic (PLEG): container finished" podID="3df7d750-133f-4d48-a34e-d425ea88031e" containerID="627ca91397436ec12df8ab9e25c36229328b229be58cff13224b8234796915c3" exitCode=0
Oct 30 01:17:34.362665 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:34.362443    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-29be73-rr7m4" event={"ID":"3df7d750-133f-4d48-a34e-d425ea88031e","Type":"ContainerDied","Data":"627ca91397436ec12df8ab9e25c36229328b229be58cff13224b8234796915c3"}
Oct 30 01:17:34.485352 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:34.485279    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w6-009e78/rate-test-model-009e78-kserve-65478f8b7c-4z5ns" podUID="c8fc10d8-e03d-4a46-bc04-410e1af5ffbf" containerName="main" containerID="cri-o://8a7443383c0442c5c03ca3e166c4ffa57deaaeeee009aa77bd4846118a1bd9bd" gracePeriod=45
Oct 30 01:17:34.658970 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:34.658870    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w6-009e78/rate-test-model-009e78-kserve-65478f8b7c-4z5ns" podUID="c8fc10d8-e03d-4a46-bc04-410e1af5ffbf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.210:8000/ready\": dial tcp 10.132.0.210:8000: connect: connection refused"
Oct 30 01:17:34.785931 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:34.785901    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w6-009e78/rate-test-model-009e78-kserve-65478f8b7c-4z5ns"
Oct 30 01:17:34.931331 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:34.931243    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8fc10d8-e03d-4a46-bc04-410e1af5ffbf-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c8fc10d8-e03d-4a46-bc04-410e1af5ffbf-tmp-dir\") pod \"c8fc10d8-e03d-4a46-bc04-410e1af5ffbf\" (UID: \"c8fc10d8-e03d-4a46-bc04-410e1af5ffbf\") "
Oct 30 01:17:34.931331 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:34.931291    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8fc10d8-e03d-4a46-bc04-410e1af5ffbf-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c8fc10d8-e03d-4a46-bc04-410e1af5ffbf-model-cache\") pod \"c8fc10d8-e03d-4a46-bc04-410e1af5ffbf\" (UID: \"c8fc10d8-e03d-4a46-bc04-410e1af5ffbf\") "
Oct 30 01:17:34.931591 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:34.931355    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8fc10d8-e03d-4a46-bc04-410e1af5ffbf-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c8fc10d8-e03d-4a46-bc04-410e1af5ffbf-dshm\") pod \"c8fc10d8-e03d-4a46-bc04-410e1af5ffbf\" (UID: \"c8fc10d8-e03d-4a46-bc04-410e1af5ffbf\") "
Oct 30 01:17:34.931591 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:34.931541    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8fc10d8-e03d-4a46-bc04-410e1af5ffbf-home\" (UniqueName: \"kubernetes.io/empty-dir/c8fc10d8-e03d-4a46-bc04-410e1af5ffbf-home\") pod \"c8fc10d8-e03d-4a46-bc04-410e1af5ffbf\" (UID: \"c8fc10d8-e03d-4a46-bc04-410e1af5ffbf\") "
Oct 30 01:17:34.931591 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:34.931562    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c8fc10d8-e03d-4a46-bc04-410e1af5ffbf-tls-certs\" (UniqueName: \"kubernetes.io/secret/c8fc10d8-e03d-4a46-bc04-410e1af5ffbf-tls-certs\") pod \"c8fc10d8-e03d-4a46-bc04-410e1af5ffbf\" (UID: \"c8fc10d8-e03d-4a46-bc04-410e1af5ffbf\") "
Oct 30 01:17:34.931766 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:34.931605    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8fc10d8-e03d-4a46-bc04-410e1af5ffbf-tmp-dir" pod "c8fc10d8-e03d-4a46-bc04-410e1af5ffbf" (UID: "c8fc10d8-e03d-4a46-bc04-410e1af5ffbf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:17:34.931766 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:34.931637    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8fc10d8-e03d-4a46-bc04-410e1af5ffbf-kube-api-access-gl55x\" (UniqueName: \"kubernetes.io/projected/c8fc10d8-e03d-4a46-bc04-410e1af5ffbf-kube-api-access-gl55x\") pod \"c8fc10d8-e03d-4a46-bc04-410e1af5ffbf\" (UID: \"c8fc10d8-e03d-4a46-bc04-410e1af5ffbf\") "
Oct 30 01:17:34.931939 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:34.931617    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8fc10d8-e03d-4a46-bc04-410e1af5ffbf-model-cache" pod "c8fc10d8-e03d-4a46-bc04-410e1af5ffbf" (UID: "c8fc10d8-e03d-4a46-bc04-410e1af5ffbf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:17:34.931939 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:34.931821    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8fc10d8-e03d-4a46-bc04-410e1af5ffbf-home" pod "c8fc10d8-e03d-4a46-bc04-410e1af5ffbf" (UID: "c8fc10d8-e03d-4a46-bc04-410e1af5ffbf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:17:34.932178 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:34.932143    5002 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c8fc10d8-e03d-4a46-bc04-410e1af5ffbf-tmp-dir\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:17:34.932178 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:34.932171    5002 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c8fc10d8-e03d-4a46-bc04-410e1af5ffbf-model-cache\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:17:34.932342 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:34.932185    5002 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c8fc10d8-e03d-4a46-bc04-410e1af5ffbf-home\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:17:34.933753 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:34.933730    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8fc10d8-e03d-4a46-bc04-410e1af5ffbf-dshm" pod "c8fc10d8-e03d-4a46-bc04-410e1af5ffbf" (UID: "c8fc10d8-e03d-4a46-bc04-410e1af5ffbf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:17:34.933753 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:34.933738    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8fc10d8-e03d-4a46-bc04-410e1af5ffbf-tls-certs" pod "c8fc10d8-e03d-4a46-bc04-410e1af5ffbf" (UID: "c8fc10d8-e03d-4a46-bc04-410e1af5ffbf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:17:34.933902 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:34.933844    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8fc10d8-e03d-4a46-bc04-410e1af5ffbf-kube-api-access-gl55x" pod "c8fc10d8-e03d-4a46-bc04-410e1af5ffbf" (UID: "c8fc10d8-e03d-4a46-bc04-410e1af5ffbf"). InnerVolumeSpecName "kube-api-access-gl55x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:17:35.039426 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:35.036262    5002 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c8fc10d8-e03d-4a46-bc04-410e1af5ffbf-dshm\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:17:35.039426 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:35.036331    5002 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8fc10d8-e03d-4a46-bc04-410e1af5ffbf-tls-certs\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:17:35.039426 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:35.036346    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gl55x\" (UniqueName: \"kubernetes.io/projected/c8fc10d8-e03d-4a46-bc04-410e1af5ffbf-kube-api-access-gl55x\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:17:35.374069 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:35.374027    5002 generic.go:356] "Generic (PLEG): container finished" podID="c8fc10d8-e03d-4a46-bc04-410e1af5ffbf" containerID="8a7443383c0442c5c03ca3e166c4ffa57deaaeeee009aa77bd4846118a1bd9bd" exitCode=0
Oct 30 01:17:35.374255 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:35.374097    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w6-009e78/rate-test-model-009e78-kserve-65478f8b7c-4z5ns"
Oct 30 01:17:35.374255 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:35.374110    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w6-009e78/rate-test-model-009e78-kserve-65478f8b7c-4z5ns" event={"ID":"c8fc10d8-e03d-4a46-bc04-410e1af5ffbf","Type":"ContainerDied","Data":"8a7443383c0442c5c03ca3e166c4ffa57deaaeeee009aa77bd4846118a1bd9bd"}
Oct 30 01:17:35.374255 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:35.374164    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w6-009e78/rate-test-model-009e78-kserve-65478f8b7c-4z5ns" event={"ID":"c8fc10d8-e03d-4a46-bc04-410e1af5ffbf","Type":"ContainerDied","Data":"54568528d138d94f43ca26a4e40a9cf2d07b3d5c80eb8d842371cbd211f3933c"}
Oct 30 01:17:35.374255 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:35.374188    5002 scope.go:122] "RemoveContainer" containerID="8a7443383c0442c5c03ca3e166c4ffa57deaaeeee009aa77bd4846118a1bd9bd"
Oct 30 01:17:35.390767 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:35.390730    5002 scope.go:122] "RemoveContainer" containerID="8a7443383c0442c5c03ca3e166c4ffa57deaaeeee009aa77bd4846118a1bd9bd"
Oct 30 01:17:35.391151 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:17:35.391120    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a7443383c0442c5c03ca3e166c4ffa57deaaeeee009aa77bd4846118a1bd9bd\": container with ID starting with 8a7443383c0442c5c03ca3e166c4ffa57deaaeeee009aa77bd4846118a1bd9bd not found: ID does not exist" containerID="8a7443383c0442c5c03ca3e166c4ffa57deaaeeee009aa77bd4846118a1bd9bd"
Oct 30 01:17:35.391271 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:35.391162    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a7443383c0442c5c03ca3e166c4ffa57deaaeeee009aa77bd4846118a1bd9bd"} err="failed to get container status \"8a7443383c0442c5c03ca3e166c4ffa57deaaeeee009aa77bd4846118a1bd9bd\": rpc error: code = NotFound desc = could not find container \"8a7443383c0442c5c03ca3e166c4ffa57deaaeeee009aa77bd4846118a1bd9bd\": container with ID starting with 8a7443383c0442c5c03ca3e166c4ffa57deaaeeee009aa77bd4846118a1bd9bd not found: ID does not exist"
Oct 30 01:17:35.442875 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:35.441542    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w6-009e78/rate-test-model-009e78-kserve-65478f8b7c-4z5ns"]
Oct 30 01:17:35.448537 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:35.448515    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w6-009e78/rate-test-model-009e78-kserve-65478f8b7c-4z5ns"]
Oct 30 01:17:35.594396 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:35.594355    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-29be73-rr7m4"
Oct 30 01:17:35.652298 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:35.652223    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-b73232-7f6ff6589f-nmvjx"]
Oct 30 01:17:35.652895 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:35.652877    5002 cpu_manager.go:414] "containerMap: removing container" podUID="c8fc10d8-e03d-4a46-bc04-410e1af5ffbf" containerName="main"
Oct 30 01:17:35.652937 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:35.652897    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8fc10d8-e03d-4a46-bc04-410e1af5ffbf" containerName="main"
Oct 30 01:17:35.652937 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:35.652921    5002 cpu_manager.go:414] "containerMap: removing container" podUID="3df7d750-133f-4d48-a34e-d425ea88031e" containerName="revoke-keys"
Oct 30 01:17:35.652937 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:35.652927    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3df7d750-133f-4d48-a34e-d425ea88031e" containerName="revoke-keys"
Oct 30 01:17:35.653053 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:35.653035    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8fc10d8-e03d-4a46-bc04-410e1af5ffbf" containerName="main"
Oct 30 01:17:35.653053 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:35.653046    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="3df7d750-133f-4d48-a34e-d425ea88031e" containerName="revoke-keys"
Oct 30 01:17:35.656610 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:35.656590    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b73232-7f6ff6589f-nmvjx"
Oct 30 01:17:35.678471 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:35.678440    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-b73232-dockercfg-4jx89\""
Oct 30 01:17:35.680626 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:35.680596    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-b73232-7f6ff6589f-nmvjx"]
Oct 30 01:17:35.724406 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:35.711725    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-b73232\""
Oct 30 01:17:35.752416 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:35.752194    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3df7d750-133f-4d48-a34e-d425ea88031e-kube-api-access-p2bg5\" (UniqueName: \"kubernetes.io/projected/3df7d750-133f-4d48-a34e-d425ea88031e-kube-api-access-p2bg5\") pod \"3df7d750-133f-4d48-a34e-d425ea88031e\" (UID: \"3df7d750-133f-4d48-a34e-d425ea88031e\") "
Oct 30 01:17:35.752416 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:35.752285    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3df7d750-133f-4d48-a34e-d425ea88031e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3df7d750-133f-4d48-a34e-d425ea88031e-maas-api-openshift-ca\") pod \"3df7d750-133f-4d48-a34e-d425ea88031e\" (UID: \"3df7d750-133f-4d48-a34e-d425ea88031e\") "
Oct 30 01:17:35.752416 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:35.752353    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3df7d750-133f-4d48-a34e-d425ea88031e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3df7d750-133f-4d48-a34e-d425ea88031e-maas-api-xks-ca\") pod \"3df7d750-133f-4d48-a34e-d425ea88031e\" (UID: \"3df7d750-133f-4d48-a34e-d425ea88031e\") "
Oct 30 01:17:35.782412 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:35.752768    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9359f1a2-ec9e-43de-a324-ab2aee55800d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-b73232-7f6ff6589f-nmvjx\" (UID: \"9359f1a2-ec9e-43de-a324-ab2aee55800d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b73232-7f6ff6589f-nmvjx"
Oct 30 01:17:35.782412 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:35.752984    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgvvj\" (UniqueName: \"kubernetes.io/projected/9359f1a2-ec9e-43de-a324-ab2aee55800d-kube-api-access-xgvvj\") pod \"payload-pre-processing-e2e-ipp-a-b73232-7f6ff6589f-nmvjx\" (UID: \"9359f1a2-ec9e-43de-a324-ab2aee55800d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b73232-7f6ff6589f-nmvjx"
Oct 30 01:17:35.782412 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:35.774686    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3df7d750-133f-4d48-a34e-d425ea88031e-maas-api-openshift-ca" pod "3df7d750-133f-4d48-a34e-d425ea88031e" (UID: "3df7d750-133f-4d48-a34e-d425ea88031e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:17:35.835410 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:35.830764    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3df7d750-133f-4d48-a34e-d425ea88031e-maas-api-xks-ca" pod "3df7d750-133f-4d48-a34e-d425ea88031e" (UID: "3df7d750-133f-4d48-a34e-d425ea88031e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:17:35.858819 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:35.856344    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9359f1a2-ec9e-43de-a324-ab2aee55800d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-b73232-7f6ff6589f-nmvjx\" (UID: \"9359f1a2-ec9e-43de-a324-ab2aee55800d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b73232-7f6ff6589f-nmvjx"
Oct 30 01:17:35.858819 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:35.856573    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgvvj\" (UniqueName: \"kubernetes.io/projected/9359f1a2-ec9e-43de-a324-ab2aee55800d-kube-api-access-xgvvj\") pod \"payload-pre-processing-e2e-ipp-a-b73232-7f6ff6589f-nmvjx\" (UID: \"9359f1a2-ec9e-43de-a324-ab2aee55800d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b73232-7f6ff6589f-nmvjx"
Oct 30 01:17:35.858819 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:35.856789    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3df7d750-133f-4d48-a34e-d425ea88031e-maas-api-xks-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:17:35.858819 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:35.856813    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3df7d750-133f-4d48-a34e-d425ea88031e-maas-api-openshift-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:17:35.858819 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:35.858054    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9359f1a2-ec9e-43de-a324-ab2aee55800d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-b73232-7f6ff6589f-nmvjx\" (UID: \"9359f1a2-ec9e-43de-a324-ab2aee55800d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b73232-7f6ff6589f-nmvjx"
Oct 30 01:17:35.873639 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:35.873586    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3df7d750-133f-4d48-a34e-d425ea88031e-kube-api-access-p2bg5" pod "3df7d750-133f-4d48-a34e-d425ea88031e" (UID: "3df7d750-133f-4d48-a34e-d425ea88031e"). InnerVolumeSpecName "kube-api-access-p2bg5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:17:35.895609 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:35.895542    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgvvj\" (UniqueName: \"kubernetes.io/projected/9359f1a2-ec9e-43de-a324-ab2aee55800d-kube-api-access-xgvvj\") pod \"payload-pre-processing-e2e-ipp-a-b73232-7f6ff6589f-nmvjx\" (UID: \"9359f1a2-ec9e-43de-a324-ab2aee55800d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b73232-7f6ff6589f-nmvjx"
Oct 30 01:17:35.960547 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:35.960493    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p2bg5\" (UniqueName: \"kubernetes.io/projected/3df7d750-133f-4d48-a34e-d425ea88031e-kube-api-access-p2bg5\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:17:35.979070 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:35.979026    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b73232-7f6ff6589f-nmvjx"
Oct 30 01:17:36.279635 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:36.279352    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-29be73/premium-w1-29be73-kserve-7b777cfc8c-7kgp5"]
Oct 30 01:17:36.288684 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:36.288650    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-29be73/simulator-w1-29be73-kserve-9958d76db-fks2t"]
Oct 30 01:17:36.293429 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:36.293042    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-b73232-7f6ff6589f-nmvjx"]
Oct 30 01:17:36.294729 ip-10-0-128-7 kubenswrapper[5002]: W1030 01:17:36.294705    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9359f1a2_ec9e_43de_a324_ab2aee55800d.slice/crio-0d22960afd66ae44c388b19100ea160a9f05ae04342a259c0c691b2f90dd098d WatchSource:0}: Error finding container 0d22960afd66ae44c388b19100ea160a9f05ae04342a259c0c691b2f90dd098d: Status 404 returned error can't find the container with id 0d22960afd66ae44c388b19100ea160a9f05ae04342a259c0c691b2f90dd098d
Oct 30 01:17:36.389258 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:36.389219    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-29be73-rr7m4" event={"ID":"3df7d750-133f-4d48-a34e-d425ea88031e","Type":"ContainerDied","Data":"773f462507882898b5105b6c34af134b74974abcba48d6e66368d4a13389eb2a"}
Oct 30 01:17:36.389503 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:36.389283    5002 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="773f462507882898b5105b6c34af134b74974abcba48d6e66368d4a13389eb2a"
Oct 30 01:17:36.389503 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:36.389395    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-29be73-rr7m4"
Oct 30 01:17:36.402014 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:36.401930    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b73232-7f6ff6589f-nmvjx" event={"ID":"9359f1a2-ec9e-43de-a324-ab2aee55800d","Type":"ContainerStarted","Data":"5c321c052697792f422ee1cce04fb55f3c0b622ceb953c3a3a6796fa66ba90c1"}
Oct 30 01:17:36.402014 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:36.401999    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b73232-7f6ff6589f-nmvjx" event={"ID":"9359f1a2-ec9e-43de-a324-ab2aee55800d","Type":"ContainerStarted","Data":"0d22960afd66ae44c388b19100ea160a9f05ae04342a259c0c691b2f90dd098d"}
Oct 30 01:17:36.451882 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:36.451827    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b73232-7f6ff6589f-nmvjx" podStartSLOduration=1.451811519 podStartE2EDuration="1.451811519s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:17:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:17:36.447698285 +0000 UTC m=+2752.064131185" watchObservedRunningTime="2026-10-30 01:17:36.451811519 +0000 UTC m=+2752.068244398"
Oct 30 01:17:36.511801 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:36.511765    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8fc10d8-e03d-4a46-bc04-410e1af5ffbf" path="/var/lib/kubelet/pods/c8fc10d8-e03d-4a46-bc04-410e1af5ffbf/volumes"
Oct 30 01:17:37.408971 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:37.408929    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b73232-7f6ff6589f-nmvjx"
Oct 30 01:17:38.911350 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:38.911267    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-b7vtt"
Oct 30 01:17:39.029405 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:39.028650    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43b2d112-3ccc-448f-8f03-e6e15fccfa4c-kube-api-access-vp2n4\" (UniqueName: \"kubernetes.io/projected/43b2d112-3ccc-448f-8f03-e6e15fccfa4c-kube-api-access-vp2n4\") pod \"43b2d112-3ccc-448f-8f03-e6e15fccfa4c\" (UID: \"43b2d112-3ccc-448f-8f03-e6e15fccfa4c\") "
Oct 30 01:17:39.029405 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:39.028760    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43b2d112-3ccc-448f-8f03-e6e15fccfa4c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43b2d112-3ccc-448f-8f03-e6e15fccfa4c-istiod-ca-cert\") pod \"43b2d112-3ccc-448f-8f03-e6e15fccfa4c\" (UID: \"43b2d112-3ccc-448f-8f03-e6e15fccfa4c\") "
Oct 30 01:17:39.029405 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:39.028793    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43b2d112-3ccc-448f-8f03-e6e15fccfa4c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43b2d112-3ccc-448f-8f03-e6e15fccfa4c-istio-data\") pod \"43b2d112-3ccc-448f-8f03-e6e15fccfa4c\" (UID: \"43b2d112-3ccc-448f-8f03-e6e15fccfa4c\") "
Oct 30 01:17:39.029405 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:39.028828    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/43b2d112-3ccc-448f-8f03-e6e15fccfa4c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43b2d112-3ccc-448f-8f03-e6e15fccfa4c-istio-podinfo\") pod \"43b2d112-3ccc-448f-8f03-e6e15fccfa4c\" (UID: \"43b2d112-3ccc-448f-8f03-e6e15fccfa4c\") "
Oct 30 01:17:39.029405 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:39.028915    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43b2d112-3ccc-448f-8f03-e6e15fccfa4c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43b2d112-3ccc-448f-8f03-e6e15fccfa4c-workload-socket\") pod \"43b2d112-3ccc-448f-8f03-e6e15fccfa4c\" (UID: \"43b2d112-3ccc-448f-8f03-e6e15fccfa4c\") "
Oct 30 01:17:39.029405 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:39.028962    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43b2d112-3ccc-448f-8f03-e6e15fccfa4c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43b2d112-3ccc-448f-8f03-e6e15fccfa4c-workload-certs\") pod \"43b2d112-3ccc-448f-8f03-e6e15fccfa4c\" (UID: \"43b2d112-3ccc-448f-8f03-e6e15fccfa4c\") "
Oct 30 01:17:39.029405 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:39.029150    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43b2d112-3ccc-448f-8f03-e6e15fccfa4c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43b2d112-3ccc-448f-8f03-e6e15fccfa4c-credential-socket\") pod \"43b2d112-3ccc-448f-8f03-e6e15fccfa4c\" (UID: \"43b2d112-3ccc-448f-8f03-e6e15fccfa4c\") "
Oct 30 01:17:39.029405 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:39.029238    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43b2d112-3ccc-448f-8f03-e6e15fccfa4c-istio-token\" (UniqueName: \"kubernetes.io/projected/43b2d112-3ccc-448f-8f03-e6e15fccfa4c-istio-token\") pod \"43b2d112-3ccc-448f-8f03-e6e15fccfa4c\" (UID: \"43b2d112-3ccc-448f-8f03-e6e15fccfa4c\") "
Oct 30 01:17:39.029405 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:39.029280    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43b2d112-3ccc-448f-8f03-e6e15fccfa4c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43b2d112-3ccc-448f-8f03-e6e15fccfa4c-istio-envoy\") pod \"43b2d112-3ccc-448f-8f03-e6e15fccfa4c\" (UID: \"43b2d112-3ccc-448f-8f03-e6e15fccfa4c\") "
Oct 30 01:17:39.032034 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:39.031029    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43b2d112-3ccc-448f-8f03-e6e15fccfa4c-workload-certs" pod "43b2d112-3ccc-448f-8f03-e6e15fccfa4c" (UID: "43b2d112-3ccc-448f-8f03-e6e15fccfa4c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:17:39.032034 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:39.032000    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43b2d112-3ccc-448f-8f03-e6e15fccfa4c-workload-socket" pod "43b2d112-3ccc-448f-8f03-e6e15fccfa4c" (UID: "43b2d112-3ccc-448f-8f03-e6e15fccfa4c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:17:39.035402 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:39.032347    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43b2d112-3ccc-448f-8f03-e6e15fccfa4c-istio-data" pod "43b2d112-3ccc-448f-8f03-e6e15fccfa4c" (UID: "43b2d112-3ccc-448f-8f03-e6e15fccfa4c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:17:39.035402 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:39.032588    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43b2d112-3ccc-448f-8f03-e6e15fccfa4c-credential-socket" pod "43b2d112-3ccc-448f-8f03-e6e15fccfa4c" (UID: "43b2d112-3ccc-448f-8f03-e6e15fccfa4c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:17:39.036694 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:39.036647    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43b2d112-3ccc-448f-8f03-e6e15fccfa4c-istiod-ca-cert" pod "43b2d112-3ccc-448f-8f03-e6e15fccfa4c" (UID: "43b2d112-3ccc-448f-8f03-e6e15fccfa4c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:17:39.102542 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:39.102501    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43b2d112-3ccc-448f-8f03-e6e15fccfa4c-kube-api-access-vp2n4" pod "43b2d112-3ccc-448f-8f03-e6e15fccfa4c" (UID: "43b2d112-3ccc-448f-8f03-e6e15fccfa4c"). InnerVolumeSpecName "kube-api-access-vp2n4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:17:39.103601 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:39.103533    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/43b2d112-3ccc-448f-8f03-e6e15fccfa4c-istio-podinfo" pod "43b2d112-3ccc-448f-8f03-e6e15fccfa4c" (UID: "43b2d112-3ccc-448f-8f03-e6e15fccfa4c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:17:39.103601 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:39.103573    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43b2d112-3ccc-448f-8f03-e6e15fccfa4c-istio-envoy" pod "43b2d112-3ccc-448f-8f03-e6e15fccfa4c" (UID: "43b2d112-3ccc-448f-8f03-e6e15fccfa4c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:17:39.104779 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:39.104731    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43b2d112-3ccc-448f-8f03-e6e15fccfa4c-istio-token" pod "43b2d112-3ccc-448f-8f03-e6e15fccfa4c" (UID: "43b2d112-3ccc-448f-8f03-e6e15fccfa4c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:17:39.131333 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:39.131239    5002 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43b2d112-3ccc-448f-8f03-e6e15fccfa4c-workload-certs\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:17:39.131333 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:39.131273    5002 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43b2d112-3ccc-448f-8f03-e6e15fccfa4c-credential-socket\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:17:39.131333 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:39.131287    5002 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43b2d112-3ccc-448f-8f03-e6e15fccfa4c-istio-token\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:17:39.131333 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:39.131299    5002 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43b2d112-3ccc-448f-8f03-e6e15fccfa4c-istio-envoy\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:17:39.131333 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:39.131310    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vp2n4\" (UniqueName: \"kubernetes.io/projected/43b2d112-3ccc-448f-8f03-e6e15fccfa4c-kube-api-access-vp2n4\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:17:39.131333 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:39.131323    5002 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43b2d112-3ccc-448f-8f03-e6e15fccfa4c-istiod-ca-cert\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:17:39.131333 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:39.131335    5002 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43b2d112-3ccc-448f-8f03-e6e15fccfa4c-istio-data\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:17:39.131909 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:39.131347    5002 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43b2d112-3ccc-448f-8f03-e6e15fccfa4c-istio-podinfo\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:17:39.131909 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:39.131359    5002 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43b2d112-3ccc-448f-8f03-e6e15fccfa4c-workload-socket\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:17:39.425009 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:39.424919    5002 generic.go:356] "Generic (PLEG): container finished" podID="43b2d112-3ccc-448f-8f03-e6e15fccfa4c" containerID="ce3946fa6c193d416d428d807339ac323f37e7f1d1aabaa78c3c38a8524ae8dd" exitCode=0
Oct 30 01:17:39.425009 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:39.424989    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-b7vtt"
Oct 30 01:17:39.425222 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:39.425014    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-b7vtt" event={"ID":"43b2d112-3ccc-448f-8f03-e6e15fccfa4c","Type":"ContainerDied","Data":"ce3946fa6c193d416d428d807339ac323f37e7f1d1aabaa78c3c38a8524ae8dd"}
Oct 30 01:17:39.425222 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:39.425084    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-b7vtt" event={"ID":"43b2d112-3ccc-448f-8f03-e6e15fccfa4c","Type":"ContainerDied","Data":"06a16911f2e8f75727e9eabe19613b5235935d82281aec19f9e10395718bb2b1"}
Oct 30 01:17:39.425222 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:39.425117    5002 scope.go:122] "RemoveContainer" containerID="ce3946fa6c193d416d428d807339ac323f37e7f1d1aabaa78c3c38a8524ae8dd"
Oct 30 01:17:39.436050 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:39.436032    5002 scope.go:122] "RemoveContainer" containerID="ce3946fa6c193d416d428d807339ac323f37e7f1d1aabaa78c3c38a8524ae8dd"
Oct 30 01:17:39.436330 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:17:39.436309    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce3946fa6c193d416d428d807339ac323f37e7f1d1aabaa78c3c38a8524ae8dd\": container with ID starting with ce3946fa6c193d416d428d807339ac323f37e7f1d1aabaa78c3c38a8524ae8dd not found: ID does not exist" containerID="ce3946fa6c193d416d428d807339ac323f37e7f1d1aabaa78c3c38a8524ae8dd"
Oct 30 01:17:39.436455 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:39.436338    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce3946fa6c193d416d428d807339ac323f37e7f1d1aabaa78c3c38a8524ae8dd"} err="failed to get container status \"ce3946fa6c193d416d428d807339ac323f37e7f1d1aabaa78c3c38a8524ae8dd\": rpc error: code = NotFound desc = could not find container \"ce3946fa6c193d416d428d807339ac323f37e7f1d1aabaa78c3c38a8524ae8dd\": container with ID starting with ce3946fa6c193d416d428d807339ac323f37e7f1d1aabaa78c3c38a8524ae8dd not found: ID does not exist"
Oct 30 01:17:39.483857 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:39.483819    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-b7vtt"]
Oct 30 01:17:39.487432 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:39.487402    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-b7vtt"]
Oct 30 01:17:40.502006 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:40.501968    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43b2d112-3ccc-448f-8f03-e6e15fccfa4c" path="/var/lib/kubelet/pods/43b2d112-3ccc-448f-8f03-e6e15fccfa4c/volumes"
Oct 30 01:17:41.621402 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:41.620405    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-16c30f-openshift-default-dffb9ddc9-6jkdj"]
Oct 30 01:17:41.621402 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:41.621351    5002 cpu_manager.go:414] "containerMap: removing container" podUID="43b2d112-3ccc-448f-8f03-e6e15fccfa4c" containerName="istio-proxy"
Oct 30 01:17:41.621934 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:41.621372    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43b2d112-3ccc-448f-8f03-e6e15fccfa4c" containerName="istio-proxy"
Oct 30 01:17:41.621934 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:41.621843    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="43b2d112-3ccc-448f-8f03-e6e15fccfa4c" containerName="istio-proxy"
Oct 30 01:17:41.631435 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:41.627758    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-16c30f-openshift-default-dffb9ddc9-6jkdj"
Oct 30 01:17:41.685503 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:41.668243    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-16c30f-openshift-default-dockercfg-8htnf\""
Oct 30 01:17:41.694825 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:41.694790    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-16c30f-openshift-default-dffb9ddc9-6jkdj"]
Oct 30 01:17:41.811410 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:41.808318    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1c17dcc-9b86-41c5-bea2-e284d0ec960c-istio-token\") pod \"e2e-ipp-b-16c30f-openshift-default-dffb9ddc9-6jkdj\" (UID: \"d1c17dcc-9b86-41c5-bea2-e284d0ec960c\") " pod="openshift-ingress/e2e-ipp-b-16c30f-openshift-default-dffb9ddc9-6jkdj"
Oct 30 01:17:41.811410 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:41.808465    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1c17dcc-9b86-41c5-bea2-e284d0ec960c-istio-podinfo\") pod \"e2e-ipp-b-16c30f-openshift-default-dffb9ddc9-6jkdj\" (UID: \"d1c17dcc-9b86-41c5-bea2-e284d0ec960c\") " pod="openshift-ingress/e2e-ipp-b-16c30f-openshift-default-dffb9ddc9-6jkdj"
Oct 30 01:17:41.811410 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:41.808501    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1c17dcc-9b86-41c5-bea2-e284d0ec960c-istiod-ca-cert\") pod \"e2e-ipp-b-16c30f-openshift-default-dffb9ddc9-6jkdj\" (UID: \"d1c17dcc-9b86-41c5-bea2-e284d0ec960c\") " pod="openshift-ingress/e2e-ipp-b-16c30f-openshift-default-dffb9ddc9-6jkdj"
Oct 30 01:17:41.811410 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:41.808592    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmp5m\" (UniqueName: \"kubernetes.io/projected/d1c17dcc-9b86-41c5-bea2-e284d0ec960c-kube-api-access-lmp5m\") pod \"e2e-ipp-b-16c30f-openshift-default-dffb9ddc9-6jkdj\" (UID: \"d1c17dcc-9b86-41c5-bea2-e284d0ec960c\") " pod="openshift-ingress/e2e-ipp-b-16c30f-openshift-default-dffb9ddc9-6jkdj"
Oct 30 01:17:41.811410 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:41.808643    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1c17dcc-9b86-41c5-bea2-e284d0ec960c-workload-socket\") pod \"e2e-ipp-b-16c30f-openshift-default-dffb9ddc9-6jkdj\" (UID: \"d1c17dcc-9b86-41c5-bea2-e284d0ec960c\") " pod="openshift-ingress/e2e-ipp-b-16c30f-openshift-default-dffb9ddc9-6jkdj"
Oct 30 01:17:41.811410 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:41.808695    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1c17dcc-9b86-41c5-bea2-e284d0ec960c-workload-certs\") pod \"e2e-ipp-b-16c30f-openshift-default-dffb9ddc9-6jkdj\" (UID: \"d1c17dcc-9b86-41c5-bea2-e284d0ec960c\") " pod="openshift-ingress/e2e-ipp-b-16c30f-openshift-default-dffb9ddc9-6jkdj"
Oct 30 01:17:41.811410 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:41.808776    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1c17dcc-9b86-41c5-bea2-e284d0ec960c-istio-data\") pod \"e2e-ipp-b-16c30f-openshift-default-dffb9ddc9-6jkdj\" (UID: \"d1c17dcc-9b86-41c5-bea2-e284d0ec960c\") " pod="openshift-ingress/e2e-ipp-b-16c30f-openshift-default-dffb9ddc9-6jkdj"
Oct 30 01:17:41.811410 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:41.808809    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1c17dcc-9b86-41c5-bea2-e284d0ec960c-istio-envoy\") pod \"e2e-ipp-b-16c30f-openshift-default-dffb9ddc9-6jkdj\" (UID: \"d1c17dcc-9b86-41c5-bea2-e284d0ec960c\") " pod="openshift-ingress/e2e-ipp-b-16c30f-openshift-default-dffb9ddc9-6jkdj"
Oct 30 01:17:41.811410 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:41.808872    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1c17dcc-9b86-41c5-bea2-e284d0ec960c-credential-socket\") pod \"e2e-ipp-b-16c30f-openshift-default-dffb9ddc9-6jkdj\" (UID: \"d1c17dcc-9b86-41c5-bea2-e284d0ec960c\") " pod="openshift-ingress/e2e-ipp-b-16c30f-openshift-default-dffb9ddc9-6jkdj"
Oct 30 01:17:41.916457 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:41.915325    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1c17dcc-9b86-41c5-bea2-e284d0ec960c-istio-data\") pod \"e2e-ipp-b-16c30f-openshift-default-dffb9ddc9-6jkdj\" (UID: \"d1c17dcc-9b86-41c5-bea2-e284d0ec960c\") " pod="openshift-ingress/e2e-ipp-b-16c30f-openshift-default-dffb9ddc9-6jkdj"
Oct 30 01:17:41.916457 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:41.915417    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1c17dcc-9b86-41c5-bea2-e284d0ec960c-istio-envoy\") pod \"e2e-ipp-b-16c30f-openshift-default-dffb9ddc9-6jkdj\" (UID: \"d1c17dcc-9b86-41c5-bea2-e284d0ec960c\") " pod="openshift-ingress/e2e-ipp-b-16c30f-openshift-default-dffb9ddc9-6jkdj"
Oct 30 01:17:41.916457 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:41.915523    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1c17dcc-9b86-41c5-bea2-e284d0ec960c-credential-socket\") pod \"e2e-ipp-b-16c30f-openshift-default-dffb9ddc9-6jkdj\" (UID: \"d1c17dcc-9b86-41c5-bea2-e284d0ec960c\") " pod="openshift-ingress/e2e-ipp-b-16c30f-openshift-default-dffb9ddc9-6jkdj"
Oct 30 01:17:41.916457 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:41.915618    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1c17dcc-9b86-41c5-bea2-e284d0ec960c-istio-token\") pod \"e2e-ipp-b-16c30f-openshift-default-dffb9ddc9-6jkdj\" (UID: \"d1c17dcc-9b86-41c5-bea2-e284d0ec960c\") " pod="openshift-ingress/e2e-ipp-b-16c30f-openshift-default-dffb9ddc9-6jkdj"
Oct 30 01:17:41.916457 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:41.915760    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1c17dcc-9b86-41c5-bea2-e284d0ec960c-istio-podinfo\") pod \"e2e-ipp-b-16c30f-openshift-default-dffb9ddc9-6jkdj\" (UID: \"d1c17dcc-9b86-41c5-bea2-e284d0ec960c\") " pod="openshift-ingress/e2e-ipp-b-16c30f-openshift-default-dffb9ddc9-6jkdj"
Oct 30 01:17:41.916457 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:41.915787    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1c17dcc-9b86-41c5-bea2-e284d0ec960c-istiod-ca-cert\") pod \"e2e-ipp-b-16c30f-openshift-default-dffb9ddc9-6jkdj\" (UID: \"d1c17dcc-9b86-41c5-bea2-e284d0ec960c\") " pod="openshift-ingress/e2e-ipp-b-16c30f-openshift-default-dffb9ddc9-6jkdj"
Oct 30 01:17:41.916457 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:41.915870    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmp5m\" (UniqueName: \"kubernetes.io/projected/d1c17dcc-9b86-41c5-bea2-e284d0ec960c-kube-api-access-lmp5m\") pod \"e2e-ipp-b-16c30f-openshift-default-dffb9ddc9-6jkdj\" (UID: \"d1c17dcc-9b86-41c5-bea2-e284d0ec960c\") " pod="openshift-ingress/e2e-ipp-b-16c30f-openshift-default-dffb9ddc9-6jkdj"
Oct 30 01:17:41.916457 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:41.915916    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1c17dcc-9b86-41c5-bea2-e284d0ec960c-workload-socket\") pod \"e2e-ipp-b-16c30f-openshift-default-dffb9ddc9-6jkdj\" (UID: \"d1c17dcc-9b86-41c5-bea2-e284d0ec960c\") " pod="openshift-ingress/e2e-ipp-b-16c30f-openshift-default-dffb9ddc9-6jkdj"
Oct 30 01:17:41.916457 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:41.915965    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1c17dcc-9b86-41c5-bea2-e284d0ec960c-workload-certs\") pod \"e2e-ipp-b-16c30f-openshift-default-dffb9ddc9-6jkdj\" (UID: \"d1c17dcc-9b86-41c5-bea2-e284d0ec960c\") " pod="openshift-ingress/e2e-ipp-b-16c30f-openshift-default-dffb9ddc9-6jkdj"
Oct 30 01:17:41.917018 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:41.916483    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1c17dcc-9b86-41c5-bea2-e284d0ec960c-workload-certs\") pod \"e2e-ipp-b-16c30f-openshift-default-dffb9ddc9-6jkdj\" (UID: \"d1c17dcc-9b86-41c5-bea2-e284d0ec960c\") " pod="openshift-ingress/e2e-ipp-b-16c30f-openshift-default-dffb9ddc9-6jkdj"
Oct 30 01:17:41.917018 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:41.916735    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1c17dcc-9b86-41c5-bea2-e284d0ec960c-istio-data\") pod \"e2e-ipp-b-16c30f-openshift-default-dffb9ddc9-6jkdj\" (UID: \"d1c17dcc-9b86-41c5-bea2-e284d0ec960c\") " pod="openshift-ingress/e2e-ipp-b-16c30f-openshift-default-dffb9ddc9-6jkdj"
Oct 30 01:17:41.934452 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:41.923565    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1c17dcc-9b86-41c5-bea2-e284d0ec960c-istiod-ca-cert\") pod \"e2e-ipp-b-16c30f-openshift-default-dffb9ddc9-6jkdj\" (UID: \"d1c17dcc-9b86-41c5-bea2-e284d0ec960c\") " pod="openshift-ingress/e2e-ipp-b-16c30f-openshift-default-dffb9ddc9-6jkdj"
Oct 30 01:17:41.934452 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:41.923845    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1c17dcc-9b86-41c5-bea2-e284d0ec960c-credential-socket\") pod \"e2e-ipp-b-16c30f-openshift-default-dffb9ddc9-6jkdj\" (UID: \"d1c17dcc-9b86-41c5-bea2-e284d0ec960c\") " pod="openshift-ingress/e2e-ipp-b-16c30f-openshift-default-dffb9ddc9-6jkdj"
Oct 30 01:17:41.934452 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:41.924089    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1c17dcc-9b86-41c5-bea2-e284d0ec960c-workload-socket\") pod \"e2e-ipp-b-16c30f-openshift-default-dffb9ddc9-6jkdj\" (UID: \"d1c17dcc-9b86-41c5-bea2-e284d0ec960c\") " pod="openshift-ingress/e2e-ipp-b-16c30f-openshift-default-dffb9ddc9-6jkdj"
Oct 30 01:17:41.951418 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:41.949011    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1c17dcc-9b86-41c5-bea2-e284d0ec960c-istio-podinfo\") pod \"e2e-ipp-b-16c30f-openshift-default-dffb9ddc9-6jkdj\" (UID: \"d1c17dcc-9b86-41c5-bea2-e284d0ec960c\") " pod="openshift-ingress/e2e-ipp-b-16c30f-openshift-default-dffb9ddc9-6jkdj"
Oct 30 01:17:41.967406 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:41.958131    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmp5m\" (UniqueName: \"kubernetes.io/projected/d1c17dcc-9b86-41c5-bea2-e284d0ec960c-kube-api-access-lmp5m\") pod \"e2e-ipp-b-16c30f-openshift-default-dffb9ddc9-6jkdj\" (UID: \"d1c17dcc-9b86-41c5-bea2-e284d0ec960c\") " pod="openshift-ingress/e2e-ipp-b-16c30f-openshift-default-dffb9ddc9-6jkdj"
Oct 30 01:17:41.967406 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:41.961973    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1c17dcc-9b86-41c5-bea2-e284d0ec960c-istio-token\") pod \"e2e-ipp-b-16c30f-openshift-default-dffb9ddc9-6jkdj\" (UID: \"d1c17dcc-9b86-41c5-bea2-e284d0ec960c\") " pod="openshift-ingress/e2e-ipp-b-16c30f-openshift-default-dffb9ddc9-6jkdj"
Oct 30 01:17:41.979410 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:41.977656    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1c17dcc-9b86-41c5-bea2-e284d0ec960c-istio-envoy\") pod \"e2e-ipp-b-16c30f-openshift-default-dffb9ddc9-6jkdj\" (UID: \"d1c17dcc-9b86-41c5-bea2-e284d0ec960c\") " pod="openshift-ingress/e2e-ipp-b-16c30f-openshift-default-dffb9ddc9-6jkdj"
Oct 30 01:17:42.247665 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:42.247578    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-16c30f-openshift-default-dffb9ddc9-6jkdj"
Oct 30 01:17:42.892588 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:42.884050    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:17:42.892588 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:42.884154    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:17:42.892588 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:42.884199    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:17:42.947217 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:42.947168    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-16c30f-openshift-default-dffb9ddc9-6jkdj"]
Oct 30 01:17:43.095884 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:17:43.095657    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbjdq" containerName="extractor"
Oct 30 01:17:43.503413 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:43.503278    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-16c30f-openshift-default-dffb9ddc9-6jkdj" event={"ID":"d1c17dcc-9b86-41c5-bea2-e284d0ec960c","Type":"ContainerStarted","Data":"69cd1b815a3940aefd4ac4b378500a4e582337393a76bd80c0d630e223325db5"}
Oct 30 01:17:43.503413 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:43.503358    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-16c30f-openshift-default-dffb9ddc9-6jkdj" event={"ID":"d1c17dcc-9b86-41c5-bea2-e284d0ec960c","Type":"ContainerStarted","Data":"d866a76fc1176643be4606907737a42d7b87e60b28263c628be534013376b6c8"}
Oct 30 01:17:44.248699 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:44.248662    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-16c30f-openshift-default-dffb9ddc9-6jkdj"
Oct 30 01:17:44.265016 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:44.264686    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-16c30f-openshift-default-dffb9ddc9-6jkdj"
Oct 30 01:17:44.332928 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:44.332870    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-16c30f-openshift-default-dffb9ddc9-6jkdj" podStartSLOduration=3.332855872 podStartE2EDuration="3.332855872s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:17:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:17:43.572734383 +0000 UTC m=+2759.189167288" watchObservedRunningTime="2026-10-30 01:17:44.332855872 +0000 UTC m=+2759.949288750"
Oct 30 01:17:44.528782 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:44.528705    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-16c30f-openshift-default-dffb9ddc9-6jkdj"
Oct 30 01:17:44.528782 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:44.528778    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-16c30f-openshift-default-dffb9ddc9-6jkdj"
Oct 30 01:17:48.419756 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:48.419722    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b73232-7f6ff6589f-nmvjx"
Oct 30 01:17:49.143955 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:49.143917    5002 scope.go:122] "RemoveContainer" containerID="458905a468f2867fa4f492d43c839631ef4cdb302bad5886770c56fb71efba89"
Oct 30 01:17:49.156359 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:49.156325    5002 scope.go:122] "RemoveContainer" containerID="a184ce67447af89684194ee1ccc178000b0677ce2c3d1a77772a0a08e111be34"
Oct 30 01:17:50.577628 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:50.577593    5002 generic.go:356] "Generic (PLEG): container finished" podID="423cd7a9-8a7c-4b26-b7c2-ba0e4053c042" containerID="f0c46b72fed307bda296524757f0270115e4ab84456c2250aa124fbb87458199" exitCode=0
Oct 30 01:17:50.578276 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:50.577622    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-9a8001-openshift-default-d885f8f7f-dt8r5" event={"ID":"423cd7a9-8a7c-4b26-b7c2-ba0e4053c042","Type":"ContainerDied","Data":"f0c46b72fed307bda296524757f0270115e4ab84456c2250aa124fbb87458199"}
Oct 30 01:17:50.578276 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:50.578091    5002 scope.go:122] "RemoveContainer" containerID="f0c46b72fed307bda296524757f0270115e4ab84456c2250aa124fbb87458199"
Oct 30 01:17:50.580465 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:50.580425    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:17:50.580591 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:50.580523    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:17:50.580591 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:50.580572    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:17:51.310188 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:51.310146    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-29be73/premium-w1-29be73-kserve-7b777cfc8c-7kgp5" podUID="e6fabbdd-86ca-4cbf-85be-ef250575a355" containerName="main" containerID="cri-o://c00c22ee1139f3eb9a72f3f8ed39348c061219c29a43527a488d35fd30f85075" gracePeriod=45
Oct 30 01:17:51.312458 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:51.312358    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-29be73/simulator-w1-29be73-kserve-9958d76db-fks2t" podUID="fbb3c806-8967-4d39-84c6-9dc022fc5cae" containerName="main" containerID="cri-o://7755ab725791bbc2c78a4725e93d034cf9dc02a71ae74c42d2456ac2b24758b7" gracePeriod=45
Oct 30 01:17:51.613964 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:51.612547    5002 generic.go:356] "Generic (PLEG): container finished" podID="e6fabbdd-86ca-4cbf-85be-ef250575a355" containerID="c00c22ee1139f3eb9a72f3f8ed39348c061219c29a43527a488d35fd30f85075" exitCode=0
Oct 30 01:17:51.613964 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:51.612657    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-29be73/premium-w1-29be73-kserve-7b777cfc8c-7kgp5" event={"ID":"e6fabbdd-86ca-4cbf-85be-ef250575a355","Type":"ContainerDied","Data":"c00c22ee1139f3eb9a72f3f8ed39348c061219c29a43527a488d35fd30f85075"}
Oct 30 01:17:51.615151 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:51.615075    5002 generic.go:356] "Generic (PLEG): container finished" podID="fbb3c806-8967-4d39-84c6-9dc022fc5cae" containerID="7755ab725791bbc2c78a4725e93d034cf9dc02a71ae74c42d2456ac2b24758b7" exitCode=0
Oct 30 01:17:51.615281 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:51.615192    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-29be73/simulator-w1-29be73-kserve-9958d76db-fks2t" event={"ID":"fbb3c806-8967-4d39-84c6-9dc022fc5cae","Type":"ContainerDied","Data":"7755ab725791bbc2c78a4725e93d034cf9dc02a71ae74c42d2456ac2b24758b7"}
Oct 30 01:17:51.623428 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:51.619863    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-9a8001-openshift-default-d885f8f7f-dt8r5" event={"ID":"423cd7a9-8a7c-4b26-b7c2-ba0e4053c042","Type":"ContainerStarted","Data":"8c0c4062cdc81472706ee2c877619232632302e6cbf645e3f4e36bf61b11b94a"}
Oct 30 01:17:51.700246 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:51.700199    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-29be73/simulator-w1-29be73-kserve-9958d76db-fks2t"
Oct 30 01:17:51.730636 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:51.727181    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-29be73/premium-w1-29be73-kserve-7b777cfc8c-7kgp5"
Oct 30 01:17:51.874476 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:51.859815    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-9a8001-openshift-default-d885f8f7f-dt8r5"
Oct 30 01:17:51.874476 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:51.859944    5002 patch_prober.go:29] interesting pod/e2e-ipp-b-9a8001-openshift-default-d885f8f7f-dt8r5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.249:15021/healthz/ready\": dial tcp 10.132.0.249:15021: connect: connection refused" start-of-body=
Oct 30 01:17:51.874476 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:51.859999    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-9a8001-openshift-default-d885f8f7f-dt8r5" podUID="423cd7a9-8a7c-4b26-b7c2-ba0e4053c042" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.249:15021/healthz/ready\": dial tcp 10.132.0.249:15021: connect: connection refused"
Oct 30 01:17:51.896797 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:51.896756    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbb3c806-8967-4d39-84c6-9dc022fc5cae-home\" (UniqueName: \"kubernetes.io/empty-dir/fbb3c806-8967-4d39-84c6-9dc022fc5cae-home\") pod \"fbb3c806-8967-4d39-84c6-9dc022fc5cae\" (UID: \"fbb3c806-8967-4d39-84c6-9dc022fc5cae\") "
Oct 30 01:17:51.897835 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:51.897809    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6fabbdd-86ca-4cbf-85be-ef250575a355-kube-api-access-vhw4w\" (UniqueName: \"kubernetes.io/projected/e6fabbdd-86ca-4cbf-85be-ef250575a355-kube-api-access-vhw4w\") pod \"e6fabbdd-86ca-4cbf-85be-ef250575a355\" (UID: \"e6fabbdd-86ca-4cbf-85be-ef250575a355\") "
Oct 30 01:17:51.898015 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:51.897998    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbb3c806-8967-4d39-84c6-9dc022fc5cae-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fbb3c806-8967-4d39-84c6-9dc022fc5cae-model-cache\") pod \"fbb3c806-8967-4d39-84c6-9dc022fc5cae\" (UID: \"fbb3c806-8967-4d39-84c6-9dc022fc5cae\") "
Oct 30 01:17:51.898131 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:51.898116    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e6fabbdd-86ca-4cbf-85be-ef250575a355-tls-certs\" (UniqueName: \"kubernetes.io/secret/e6fabbdd-86ca-4cbf-85be-ef250575a355-tls-certs\") pod \"e6fabbdd-86ca-4cbf-85be-ef250575a355\" (UID: \"e6fabbdd-86ca-4cbf-85be-ef250575a355\") "
Oct 30 01:17:51.898251 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:51.898237    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6fabbdd-86ca-4cbf-85be-ef250575a355-home\" (UniqueName: \"kubernetes.io/empty-dir/e6fabbdd-86ca-4cbf-85be-ef250575a355-home\") pod \"e6fabbdd-86ca-4cbf-85be-ef250575a355\" (UID: \"e6fabbdd-86ca-4cbf-85be-ef250575a355\") "
Oct 30 01:17:51.898424 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:51.898362    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6fabbdd-86ca-4cbf-85be-ef250575a355-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6fabbdd-86ca-4cbf-85be-ef250575a355-model-cache\") pod \"e6fabbdd-86ca-4cbf-85be-ef250575a355\" (UID: \"e6fabbdd-86ca-4cbf-85be-ef250575a355\") "
Oct 30 01:17:51.899542 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:51.899523    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbb3c806-8967-4d39-84c6-9dc022fc5cae-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fbb3c806-8967-4d39-84c6-9dc022fc5cae-tmp-dir\") pod \"fbb3c806-8967-4d39-84c6-9dc022fc5cae\" (UID: \"fbb3c806-8967-4d39-84c6-9dc022fc5cae\") "
Oct 30 01:17:51.899756 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:51.897279    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbb3c806-8967-4d39-84c6-9dc022fc5cae-home" pod "fbb3c806-8967-4d39-84c6-9dc022fc5cae" (UID: "fbb3c806-8967-4d39-84c6-9dc022fc5cae"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:17:51.899756 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:51.899727    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbb3c806-8967-4d39-84c6-9dc022fc5cae-tmp-dir" pod "fbb3c806-8967-4d39-84c6-9dc022fc5cae" (UID: "fbb3c806-8967-4d39-84c6-9dc022fc5cae"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:17:51.900213 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:51.899286    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6fabbdd-86ca-4cbf-85be-ef250575a355-home" pod "e6fabbdd-86ca-4cbf-85be-ef250575a355" (UID: "e6fabbdd-86ca-4cbf-85be-ef250575a355"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:17:51.900213 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:51.899300    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6fabbdd-86ca-4cbf-85be-ef250575a355-model-cache" pod "e6fabbdd-86ca-4cbf-85be-ef250575a355" (UID: "e6fabbdd-86ca-4cbf-85be-ef250575a355"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:17:51.900494 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:51.900463    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbb3c806-8967-4d39-84c6-9dc022fc5cae-model-cache" pod "fbb3c806-8967-4d39-84c6-9dc022fc5cae" (UID: "fbb3c806-8967-4d39-84c6-9dc022fc5cae"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:17:51.900644 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:51.900608    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fbb3c806-8967-4d39-84c6-9dc022fc5cae-tls-certs\" (UniqueName: \"kubernetes.io/secret/fbb3c806-8967-4d39-84c6-9dc022fc5cae-tls-certs\") pod \"fbb3c806-8967-4d39-84c6-9dc022fc5cae\" (UID: \"fbb3c806-8967-4d39-84c6-9dc022fc5cae\") "
Oct 30 01:17:51.900803 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:51.900789    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6fabbdd-86ca-4cbf-85be-ef250575a355-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6fabbdd-86ca-4cbf-85be-ef250575a355-dshm\") pod \"e6fabbdd-86ca-4cbf-85be-ef250575a355\" (UID: \"e6fabbdd-86ca-4cbf-85be-ef250575a355\") "
Oct 30 01:17:51.901223 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:51.901206    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbb3c806-8967-4d39-84c6-9dc022fc5cae-kube-api-access-m2xb8\" (UniqueName: \"kubernetes.io/projected/fbb3c806-8967-4d39-84c6-9dc022fc5cae-kube-api-access-m2xb8\") pod \"fbb3c806-8967-4d39-84c6-9dc022fc5cae\" (UID: \"fbb3c806-8967-4d39-84c6-9dc022fc5cae\") "
Oct 30 01:17:51.901672 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:51.901655    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbb3c806-8967-4d39-84c6-9dc022fc5cae-dshm\" (UniqueName: \"kubernetes.io/empty-dir/fbb3c806-8967-4d39-84c6-9dc022fc5cae-dshm\") pod \"fbb3c806-8967-4d39-84c6-9dc022fc5cae\" (UID: \"fbb3c806-8967-4d39-84c6-9dc022fc5cae\") "
Oct 30 01:17:51.902093 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:51.902076    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6fabbdd-86ca-4cbf-85be-ef250575a355-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6fabbdd-86ca-4cbf-85be-ef250575a355-tmp-dir\") pod \"e6fabbdd-86ca-4cbf-85be-ef250575a355\" (UID: \"e6fabbdd-86ca-4cbf-85be-ef250575a355\") "
Oct 30 01:17:51.902598 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:51.902394    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6fabbdd-86ca-4cbf-85be-ef250575a355-tmp-dir" pod "e6fabbdd-86ca-4cbf-85be-ef250575a355" (UID: "e6fabbdd-86ca-4cbf-85be-ef250575a355"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:17:51.903341 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:51.903320    5002 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fbb3c806-8967-4d39-84c6-9dc022fc5cae-model-cache\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:17:51.903526 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:51.903513    5002 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6fabbdd-86ca-4cbf-85be-ef250575a355-home\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:17:51.903627 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:51.903616    5002 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6fabbdd-86ca-4cbf-85be-ef250575a355-model-cache\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:17:51.903719 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:51.903708    5002 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fbb3c806-8967-4d39-84c6-9dc022fc5cae-tmp-dir\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:17:51.903803 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:51.903793    5002 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6fabbdd-86ca-4cbf-85be-ef250575a355-tmp-dir\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:17:51.903885 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:51.903875    5002 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fbb3c806-8967-4d39-84c6-9dc022fc5cae-home\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:17:51.905903 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:51.905868    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fbb3c806-8967-4d39-84c6-9dc022fc5cae-tls-certs" pod "fbb3c806-8967-4d39-84c6-9dc022fc5cae" (UID: "fbb3c806-8967-4d39-84c6-9dc022fc5cae"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:17:51.908167 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:51.908146    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6fabbdd-86ca-4cbf-85be-ef250575a355-tls-certs" pod "e6fabbdd-86ca-4cbf-85be-ef250575a355" (UID: "e6fabbdd-86ca-4cbf-85be-ef250575a355"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:17:51.908271 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:51.908146    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbb3c806-8967-4d39-84c6-9dc022fc5cae-kube-api-access-m2xb8" pod "fbb3c806-8967-4d39-84c6-9dc022fc5cae" (UID: "fbb3c806-8967-4d39-84c6-9dc022fc5cae"). InnerVolumeSpecName "kube-api-access-m2xb8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:17:51.908271 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:51.908212    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6fabbdd-86ca-4cbf-85be-ef250575a355-kube-api-access-vhw4w" pod "e6fabbdd-86ca-4cbf-85be-ef250575a355" (UID: "e6fabbdd-86ca-4cbf-85be-ef250575a355"). InnerVolumeSpecName "kube-api-access-vhw4w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:17:51.908579 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:51.908557    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6fabbdd-86ca-4cbf-85be-ef250575a355-dshm" pod "e6fabbdd-86ca-4cbf-85be-ef250575a355" (UID: "e6fabbdd-86ca-4cbf-85be-ef250575a355"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:17:51.909737 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:51.909706    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbb3c806-8967-4d39-84c6-9dc022fc5cae-dshm" pod "fbb3c806-8967-4d39-84c6-9dc022fc5cae" (UID: "fbb3c806-8967-4d39-84c6-9dc022fc5cae"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:17:52.005714 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:52.005673    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vhw4w\" (UniqueName: \"kubernetes.io/projected/e6fabbdd-86ca-4cbf-85be-ef250575a355-kube-api-access-vhw4w\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:17:52.005714 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:52.005710    5002 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6fabbdd-86ca-4cbf-85be-ef250575a355-tls-certs\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:17:52.005714 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:52.005721    5002 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fbb3c806-8967-4d39-84c6-9dc022fc5cae-tls-certs\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:17:52.005998 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:52.005729    5002 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6fabbdd-86ca-4cbf-85be-ef250575a355-dshm\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:17:52.005998 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:52.005738    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m2xb8\" (UniqueName: \"kubernetes.io/projected/fbb3c806-8967-4d39-84c6-9dc022fc5cae-kube-api-access-m2xb8\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:17:52.005998 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:52.005751    5002 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fbb3c806-8967-4d39-84c6-9dc022fc5cae-dshm\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:17:52.628661 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:52.628571    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-29be73/simulator-w1-29be73-kserve-9958d76db-fks2t"
Oct 30 01:17:52.628661 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:52.628591    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-29be73/simulator-w1-29be73-kserve-9958d76db-fks2t" event={"ID":"fbb3c806-8967-4d39-84c6-9dc022fc5cae","Type":"ContainerDied","Data":"a1976db29b972df66fbb9ebe249adb3d56df9c803db3df1d95169879749c360c"}
Oct 30 01:17:52.628661 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:52.628645    5002 scope.go:122] "RemoveContainer" containerID="7755ab725791bbc2c78a4725e93d034cf9dc02a71ae74c42d2456ac2b24758b7"
Oct 30 01:17:52.630555 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:52.630526    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-29be73/premium-w1-29be73-kserve-7b777cfc8c-7kgp5"
Oct 30 01:17:52.630715 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:52.630554    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-29be73/premium-w1-29be73-kserve-7b777cfc8c-7kgp5" event={"ID":"e6fabbdd-86ca-4cbf-85be-ef250575a355","Type":"ContainerDied","Data":"4287fa47a7aa917fca5fc19383794f5170a2eb3349930c8720ccccd5a4561e41"}
Oct 30 01:17:52.639274 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:52.639255    5002 scope.go:122] "RemoveContainer" containerID="c00c22ee1139f3eb9a72f3f8ed39348c061219c29a43527a488d35fd30f85075"
Oct 30 01:17:52.692771 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:52.692732    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-29be73/premium-w1-29be73-kserve-7b777cfc8c-7kgp5"]
Oct 30 01:17:52.706732 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:52.706700    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-29be73/premium-w1-29be73-kserve-7b777cfc8c-7kgp5"]
Oct 30 01:17:52.743735 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:52.743699    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-29be73/simulator-w1-29be73-kserve-9958d76db-fks2t"]
Oct 30 01:17:52.747162 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:52.747138    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-29be73/simulator-w1-29be73-kserve-9958d76db-fks2t"]
Oct 30 01:17:52.866409 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:52.859649    5002 patch_prober.go:29] interesting pod/e2e-ipp-b-9a8001-openshift-default-d885f8f7f-dt8r5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.249:15021/healthz/ready\": dial tcp 10.132.0.249:15021: connect: connection refused" start-of-body=
Oct 30 01:17:52.866409 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:52.859716    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-9a8001-openshift-default-d885f8f7f-dt8r5" podUID="423cd7a9-8a7c-4b26-b7c2-ba0e4053c042" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.249:15021/healthz/ready\": dial tcp 10.132.0.249:15021: connect: connection refused"
Oct 30 01:17:53.484524 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:53.484484    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5719d2a-tl4ff"]
Oct 30 01:17:53.485142 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:53.485123    5002 cpu_manager.go:414] "containerMap: removing container" podUID="e6fabbdd-86ca-4cbf-85be-ef250575a355" containerName="main"
Oct 30 01:17:53.485203 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:53.485144    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6fabbdd-86ca-4cbf-85be-ef250575a355" containerName="main"
Oct 30 01:17:53.485203 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:53.485170    5002 cpu_manager.go:414] "containerMap: removing container" podUID="fbb3c806-8967-4d39-84c6-9dc022fc5cae" containerName="main"
Oct 30 01:17:53.485203 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:53.485176    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbb3c806-8967-4d39-84c6-9dc022fc5cae" containerName="main"
Oct 30 01:17:53.485327 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:53.485299    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6fabbdd-86ca-4cbf-85be-ef250575a355" containerName="main"
Oct 30 01:17:53.485327 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:53.485316    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbb3c806-8967-4d39-84c6-9dc022fc5cae" containerName="main"
Oct 30 01:17:53.489105 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:53.489074    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5719d2a-tl4ff"
Oct 30 01:17:53.495399 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:53.493838    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b28e50b7-j7qkn"]
Oct 30 01:17:53.497696 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:53.497676    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b28e50b7-j7qkn"
Oct 30 01:17:53.499447 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:53.499352    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5719d2a-tl4ff"]
Oct 30 01:17:53.506015 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:53.505989    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:17:53.506187 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:53.506167    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:17:53.506351 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:53.506334    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:17:53.506463 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:53.506418    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5phsh\""
Oct 30 01:17:53.525346 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:53.525314    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b28e50b7-j7qkn"]
Oct 30 01:17:53.528610 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:53.528582    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ed8b340-cc1d-4a9a-a662-e5a01147829b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b5719d2a-tl4ff\" (UID: \"8ed8b340-cc1d-4a9a-a662-e5a01147829b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5719d2a-tl4ff"
Oct 30 01:17:53.528736 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:53.528635    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17eaa864-381e-4d91-b358-862f1f709619-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b28e50b7-j7qkn\" (UID: \"17eaa864-381e-4d91-b358-862f1f709619\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b28e50b7-j7qkn"
Oct 30 01:17:53.528786 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:53.528748    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7s9dz\" (UniqueName: \"kubernetes.io/projected/8ed8b340-cc1d-4a9a-a662-e5a01147829b-kube-api-access-7s9dz\") pod \"maas-api-revoke-sub-b5719d2a-tl4ff\" (UID: \"8ed8b340-cc1d-4a9a-a662-e5a01147829b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5719d2a-tl4ff"
Oct 30 01:17:53.528825 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:53.528787    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17eaa864-381e-4d91-b358-862f1f709619-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b28e50b7-j7qkn\" (UID: \"17eaa864-381e-4d91-b358-862f1f709619\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b28e50b7-j7qkn"
Oct 30 01:17:53.528901 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:53.528886    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ed8b340-cc1d-4a9a-a662-e5a01147829b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b5719d2a-tl4ff\" (UID: \"8ed8b340-cc1d-4a9a-a662-e5a01147829b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5719d2a-tl4ff"
Oct 30 01:17:53.528973 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:53.528954    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7rqp\" (UniqueName: \"kubernetes.io/projected/17eaa864-381e-4d91-b358-862f1f709619-kube-api-access-d7rqp\") pod \"maas-api-revoke-sub-b28e50b7-j7qkn\" (UID: \"17eaa864-381e-4d91-b358-862f1f709619\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b28e50b7-j7qkn"
Oct 30 01:17:53.630481 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:53.630438    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17eaa864-381e-4d91-b358-862f1f709619-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b28e50b7-j7qkn\" (UID: \"17eaa864-381e-4d91-b358-862f1f709619\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b28e50b7-j7qkn"
Oct 30 01:17:53.630982 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:53.630511    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7s9dz\" (UniqueName: \"kubernetes.io/projected/8ed8b340-cc1d-4a9a-a662-e5a01147829b-kube-api-access-7s9dz\") pod \"maas-api-revoke-sub-b5719d2a-tl4ff\" (UID: \"8ed8b340-cc1d-4a9a-a662-e5a01147829b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5719d2a-tl4ff"
Oct 30 01:17:53.630982 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:53.630553    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17eaa864-381e-4d91-b358-862f1f709619-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b28e50b7-j7qkn\" (UID: \"17eaa864-381e-4d91-b358-862f1f709619\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b28e50b7-j7qkn"
Oct 30 01:17:53.630982 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:53.630651    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ed8b340-cc1d-4a9a-a662-e5a01147829b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b5719d2a-tl4ff\" (UID: \"8ed8b340-cc1d-4a9a-a662-e5a01147829b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5719d2a-tl4ff"
Oct 30 01:17:53.630982 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:53.630790    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7rqp\" (UniqueName: \"kubernetes.io/projected/17eaa864-381e-4d91-b358-862f1f709619-kube-api-access-d7rqp\") pod \"maas-api-revoke-sub-b28e50b7-j7qkn\" (UID: \"17eaa864-381e-4d91-b358-862f1f709619\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b28e50b7-j7qkn"
Oct 30 01:17:53.630982 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:53.630977    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ed8b340-cc1d-4a9a-a662-e5a01147829b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b5719d2a-tl4ff\" (UID: \"8ed8b340-cc1d-4a9a-a662-e5a01147829b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5719d2a-tl4ff"
Oct 30 01:17:53.631505 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:53.631477    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17eaa864-381e-4d91-b358-862f1f709619-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b28e50b7-j7qkn\" (UID: \"17eaa864-381e-4d91-b358-862f1f709619\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b28e50b7-j7qkn"
Oct 30 01:17:53.631820 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:53.631793    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ed8b340-cc1d-4a9a-a662-e5a01147829b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b5719d2a-tl4ff\" (UID: \"8ed8b340-cc1d-4a9a-a662-e5a01147829b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5719d2a-tl4ff"
Oct 30 01:17:53.633321 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:53.633298    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ed8b340-cc1d-4a9a-a662-e5a01147829b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b5719d2a-tl4ff\" (UID: \"8ed8b340-cc1d-4a9a-a662-e5a01147829b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5719d2a-tl4ff"
Oct 30 01:17:53.633449 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:53.633366    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17eaa864-381e-4d91-b358-862f1f709619-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b28e50b7-j7qkn\" (UID: \"17eaa864-381e-4d91-b358-862f1f709619\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b28e50b7-j7qkn"
Oct 30 01:17:53.679485 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:53.668433    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7s9dz\" (UniqueName: \"kubernetes.io/projected/8ed8b340-cc1d-4a9a-a662-e5a01147829b-kube-api-access-7s9dz\") pod \"maas-api-revoke-sub-b5719d2a-tl4ff\" (UID: \"8ed8b340-cc1d-4a9a-a662-e5a01147829b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5719d2a-tl4ff"
Oct 30 01:17:53.686178 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:53.686139    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7rqp\" (UniqueName: \"kubernetes.io/projected/17eaa864-381e-4d91-b358-862f1f709619-kube-api-access-d7rqp\") pod \"maas-api-revoke-sub-b28e50b7-j7qkn\" (UID: \"17eaa864-381e-4d91-b358-862f1f709619\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b28e50b7-j7qkn"
Oct 30 01:17:53.811433 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:53.811301    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5719d2a-tl4ff"
Oct 30 01:17:53.827480 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:53.827205    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b28e50b7-j7qkn"
Oct 30 01:17:53.880408 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:53.870820    5002 patch_prober.go:29] interesting pod/e2e-ipp-b-9a8001-openshift-default-d885f8f7f-dt8r5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.249:15021/healthz/ready\": dial tcp 10.132.0.249:15021: connect: connection refused" start-of-body=
Oct 30 01:17:53.880408 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:53.870882    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-9a8001-openshift-default-d885f8f7f-dt8r5" podUID="423cd7a9-8a7c-4b26-b7c2-ba0e4053c042" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.249:15021/healthz/ready\": dial tcp 10.132.0.249:15021: connect: connection refused"
Oct 30 01:17:54.206852 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:54.206799    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5719d2a-tl4ff"]
Oct 30 01:17:54.213596 ip-10-0-128-7 kubenswrapper[5002]: W1030 01:17:54.213507    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ed8b340_cc1d_4a9a_a662_e5a01147829b.slice/crio-18d586f12f68a312f9e20573f62c2cd259d1d6d348d128e906306d9f4f018490 WatchSource:0}: Error finding container 18d586f12f68a312f9e20573f62c2cd259d1d6d348d128e906306d9f4f018490: Status 404 returned error can't find the container with id 18d586f12f68a312f9e20573f62c2cd259d1d6d348d128e906306d9f4f018490
Oct 30 01:17:54.403602 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:54.402791    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b28e50b7-j7qkn"]
Oct 30 01:17:54.511065 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:54.511025    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6fabbdd-86ca-4cbf-85be-ef250575a355" path="/var/lib/kubelet/pods/e6fabbdd-86ca-4cbf-85be-ef250575a355/volumes"
Oct 30 01:17:54.511567 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:54.511549    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fbb3c806-8967-4d39-84c6-9dc022fc5cae" path="/var/lib/kubelet/pods/fbb3c806-8967-4d39-84c6-9dc022fc5cae/volumes"
Oct 30 01:17:54.649072 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:54.649023    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b28e50b7-j7qkn" event={"ID":"17eaa864-381e-4d91-b358-862f1f709619","Type":"ContainerStarted","Data":"3d00c840ca68c7085997f0cabd9bc6a5364ef1e945dcc51f695be0ed0a11dd2a"}
Oct 30 01:17:54.687751 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:54.687707    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5719d2a-tl4ff" event={"ID":"8ed8b340-cc1d-4a9a-a662-e5a01147829b","Type":"ContainerStarted","Data":"18d586f12f68a312f9e20573f62c2cd259d1d6d348d128e906306d9f4f018490"}
Oct 30 01:17:54.880464 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:54.865760    5002 patch_prober.go:29] interesting pod/e2e-ipp-b-9a8001-openshift-default-d885f8f7f-dt8r5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.249:15021/healthz/ready\": dial tcp 10.132.0.249:15021: connect: connection refused" start-of-body=
Oct 30 01:17:54.880464 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:54.865820    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-9a8001-openshift-default-d885f8f7f-dt8r5" podUID="423cd7a9-8a7c-4b26-b7c2-ba0e4053c042" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.249:15021/healthz/ready\": dial tcp 10.132.0.249:15021: connect: connection refused"
Oct 30 01:17:55.697156 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:55.697121    5002 generic.go:356] "Generic (PLEG): container finished" podID="17eaa864-381e-4d91-b358-862f1f709619" containerID="e83e8ac28d10eabb02c0dd7a4a6b9613ab7c0901e5f1eeee42305112d8d87fe5" exitCode=0
Oct 30 01:17:55.697637 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:55.697201    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b28e50b7-j7qkn" event={"ID":"17eaa864-381e-4d91-b358-862f1f709619","Type":"ContainerDied","Data":"e83e8ac28d10eabb02c0dd7a4a6b9613ab7c0901e5f1eeee42305112d8d87fe5"}
Oct 30 01:17:55.698962 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:55.698928    5002 generic.go:356] "Generic (PLEG): container finished" podID="8ed8b340-cc1d-4a9a-a662-e5a01147829b" containerID="54c4b08cad4414856b2f1e1a87036e6d7d9e9eeec497095fdb218bea687be400" exitCode=0
Oct 30 01:17:55.699110 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:55.699008    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5719d2a-tl4ff" event={"ID":"8ed8b340-cc1d-4a9a-a662-e5a01147829b","Type":"ContainerDied","Data":"54c4b08cad4414856b2f1e1a87036e6d7d9e9eeec497095fdb218bea687be400"}
Oct 30 01:17:55.871472 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:55.868789    5002 patch_prober.go:29] interesting pod/e2e-ipp-b-9a8001-openshift-default-d885f8f7f-dt8r5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.249:15021/healthz/ready\": dial tcp 10.132.0.249:15021: connect: connection refused" start-of-body=
Oct 30 01:17:55.871472 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:55.868859    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-9a8001-openshift-default-d885f8f7f-dt8r5" podUID="423cd7a9-8a7c-4b26-b7c2-ba0e4053c042" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.249:15021/healthz/ready\": dial tcp 10.132.0.249:15021: connect: connection refused"
Oct 30 01:17:56.873404 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:56.872826    5002 patch_prober.go:29] interesting pod/e2e-ipp-b-9a8001-openshift-default-d885f8f7f-dt8r5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.249:15021/healthz/ready\": dial tcp 10.132.0.249:15021: connect: connection refused" start-of-body=
Oct 30 01:17:56.873404 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:56.872888    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-9a8001-openshift-default-d885f8f7f-dt8r5" podUID="423cd7a9-8a7c-4b26-b7c2-ba0e4053c042" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.249:15021/healthz/ready\": dial tcp 10.132.0.249:15021: connect: connection refused"
Oct 30 01:17:56.925138 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:56.925105    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5719d2a-tl4ff"
Oct 30 01:17:56.988534 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:56.988499    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b28e50b7-j7qkn"
Oct 30 01:17:57.013613 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:57.013580    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8ed8b340-cc1d-4a9a-a662-e5a01147829b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ed8b340-cc1d-4a9a-a662-e5a01147829b-maas-api-xks-ca\") pod \"8ed8b340-cc1d-4a9a-a662-e5a01147829b\" (UID: \"8ed8b340-cc1d-4a9a-a662-e5a01147829b\") "
Oct 30 01:17:57.013829 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:57.013669    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ed8b340-cc1d-4a9a-a662-e5a01147829b-kube-api-access-7s9dz\" (UniqueName: \"kubernetes.io/projected/8ed8b340-cc1d-4a9a-a662-e5a01147829b-kube-api-access-7s9dz\") pod \"8ed8b340-cc1d-4a9a-a662-e5a01147829b\" (UID: \"8ed8b340-cc1d-4a9a-a662-e5a01147829b\") "
Oct 30 01:17:57.013829 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:57.013744    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ed8b340-cc1d-4a9a-a662-e5a01147829b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ed8b340-cc1d-4a9a-a662-e5a01147829b-maas-api-openshift-ca\") pod \"8ed8b340-cc1d-4a9a-a662-e5a01147829b\" (UID: \"8ed8b340-cc1d-4a9a-a662-e5a01147829b\") "
Oct 30 01:17:57.014612 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:57.014578    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ed8b340-cc1d-4a9a-a662-e5a01147829b-maas-api-openshift-ca" pod "8ed8b340-cc1d-4a9a-a662-e5a01147829b" (UID: "8ed8b340-cc1d-4a9a-a662-e5a01147829b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:17:57.016153 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:57.016126    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ed8b340-cc1d-4a9a-a662-e5a01147829b-kube-api-access-7s9dz" pod "8ed8b340-cc1d-4a9a-a662-e5a01147829b" (UID: "8ed8b340-cc1d-4a9a-a662-e5a01147829b"). InnerVolumeSpecName "kube-api-access-7s9dz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:17:57.016415 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:57.016373    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ed8b340-cc1d-4a9a-a662-e5a01147829b-maas-api-xks-ca" pod "8ed8b340-cc1d-4a9a-a662-e5a01147829b" (UID: "8ed8b340-cc1d-4a9a-a662-e5a01147829b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:17:57.115223 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:57.115189    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17eaa864-381e-4d91-b358-862f1f709619-kube-api-access-d7rqp\" (UniqueName: \"kubernetes.io/projected/17eaa864-381e-4d91-b358-862f1f709619-kube-api-access-d7rqp\") pod \"17eaa864-381e-4d91-b358-862f1f709619\" (UID: \"17eaa864-381e-4d91-b358-862f1f709619\") "
Oct 30 01:17:57.115489 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:57.115468    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/17eaa864-381e-4d91-b358-862f1f709619-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17eaa864-381e-4d91-b358-862f1f709619-maas-api-xks-ca\") pod \"17eaa864-381e-4d91-b358-862f1f709619\" (UID: \"17eaa864-381e-4d91-b358-862f1f709619\") "
Oct 30 01:17:57.115567 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:57.115528    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17eaa864-381e-4d91-b358-862f1f709619-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17eaa864-381e-4d91-b358-862f1f709619-maas-api-openshift-ca\") pod \"17eaa864-381e-4d91-b358-862f1f709619\" (UID: \"17eaa864-381e-4d91-b358-862f1f709619\") "
Oct 30 01:17:57.115917 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:57.115889    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17eaa864-381e-4d91-b358-862f1f709619-maas-api-openshift-ca" pod "17eaa864-381e-4d91-b358-862f1f709619" (UID: "17eaa864-381e-4d91-b358-862f1f709619"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:17:57.116112 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:57.116090    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17eaa864-381e-4d91-b358-862f1f709619-maas-api-openshift-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:17:57.116205 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:57.116114    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ed8b340-cc1d-4a9a-a662-e5a01147829b-maas-api-openshift-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:17:57.116205 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:57.116124    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ed8b340-cc1d-4a9a-a662-e5a01147829b-maas-api-xks-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:17:57.116205 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:57.116134    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7s9dz\" (UniqueName: \"kubernetes.io/projected/8ed8b340-cc1d-4a9a-a662-e5a01147829b-kube-api-access-7s9dz\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:17:57.117541 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:57.117504    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17eaa864-381e-4d91-b358-862f1f709619-kube-api-access-d7rqp" pod "17eaa864-381e-4d91-b358-862f1f709619" (UID: "17eaa864-381e-4d91-b358-862f1f709619"). InnerVolumeSpecName "kube-api-access-d7rqp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:17:57.117653 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:57.117599    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17eaa864-381e-4d91-b358-862f1f709619-maas-api-xks-ca" pod "17eaa864-381e-4d91-b358-862f1f709619" (UID: "17eaa864-381e-4d91-b358-862f1f709619"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:17:57.227473 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:57.219530    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17eaa864-381e-4d91-b358-862f1f709619-maas-api-xks-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:17:57.227473 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:57.219574    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d7rqp\" (UniqueName: \"kubernetes.io/projected/17eaa864-381e-4d91-b358-862f1f709619-kube-api-access-d7rqp\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:17:57.716521 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:57.716488    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b28e50b7-j7qkn"
Oct 30 01:17:57.716726 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:57.716528    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b28e50b7-j7qkn" event={"ID":"17eaa864-381e-4d91-b358-862f1f709619","Type":"ContainerDied","Data":"3d00c840ca68c7085997f0cabd9bc6a5364ef1e945dcc51f695be0ed0a11dd2a"}
Oct 30 01:17:57.716726 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:57.716579    5002 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3d00c840ca68c7085997f0cabd9bc6a5364ef1e945dcc51f695be0ed0a11dd2a"
Oct 30 01:17:57.718078 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:57.718057    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5719d2a-tl4ff"
Oct 30 01:17:57.718246 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:57.718080    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5719d2a-tl4ff" event={"ID":"8ed8b340-cc1d-4a9a-a662-e5a01147829b","Type":"ContainerDied","Data":"18d586f12f68a312f9e20573f62c2cd259d1d6d348d128e906306d9f4f018490"}
Oct 30 01:17:57.718246 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:57.718116    5002 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="18d586f12f68a312f9e20573f62c2cd259d1d6d348d128e906306d9f4f018490"
Oct 30 01:17:57.861710 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:57.861675    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-9a8001-openshift-default-d885f8f7f-dt8r5"
Oct 30 01:17:57.862150 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:57.862122    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-9a8001-openshift-default-d885f8f7f-dt8r5"
Oct 30 01:17:57.862984 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:17:57.862964    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-9a8001-openshift-default-d885f8f7f-dt8r5"
Oct 30 01:18:00.048333 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:00.048283    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e31412e-t2bcw"]
Oct 30 01:18:00.054531 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:00.054497    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e31412e-t2bcw"]
Oct 30 01:18:00.499449 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:00.499412    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25f70018-d36a-4d1a-933a-1b7db5d8614d" path="/var/lib/kubelet/pods/25f70018-d36a-4d1a-933a-1b7db5d8614d/volumes"
Oct 30 01:18:05.810435 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:05.810396    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-a7ca1f-openshift-default-77b7bf58df-l8qlx"]
Oct 30 01:18:05.811027 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:05.811013    5002 cpu_manager.go:414] "containerMap: removing container" podUID="17eaa864-381e-4d91-b358-862f1f709619" containerName="revoke-keys"
Oct 30 01:18:05.811085 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:05.811028    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17eaa864-381e-4d91-b358-862f1f709619" containerName="revoke-keys"
Oct 30 01:18:05.811085 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:05.811062    5002 cpu_manager.go:414] "containerMap: removing container" podUID="8ed8b340-cc1d-4a9a-a662-e5a01147829b" containerName="revoke-keys"
Oct 30 01:18:05.811085 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:05.811067    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ed8b340-cc1d-4a9a-a662-e5a01147829b" containerName="revoke-keys"
Oct 30 01:18:05.811201 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:05.811149    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ed8b340-cc1d-4a9a-a662-e5a01147829b" containerName="revoke-keys"
Oct 30 01:18:05.811201 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:05.811163    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="17eaa864-381e-4d91-b358-862f1f709619" containerName="revoke-keys"
Oct 30 01:18:05.816053 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:05.816033    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-a7ca1f-openshift-default-77b7bf58df-l8qlx"
Oct 30 01:18:05.832616 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:05.832589    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-a7ca1f-openshift-default-dockercfg-774lk\""
Oct 30 01:18:05.855053 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:05.855005    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-a7ca1f-openshift-default-77b7bf58df-l8qlx"]
Oct 30 01:18:05.876896 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:05.876851    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lt4jg\" (UniqueName: \"kubernetes.io/projected/6c75a860-08fd-4293-8c2d-9aa04e4348fa-kube-api-access-lt4jg\") pod \"e2e-ipp-a-a7ca1f-openshift-default-77b7bf58df-l8qlx\" (UID: \"6c75a860-08fd-4293-8c2d-9aa04e4348fa\") " pod="openshift-ingress/e2e-ipp-a-a7ca1f-openshift-default-77b7bf58df-l8qlx"
Oct 30 01:18:05.877096 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:05.876908    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c75a860-08fd-4293-8c2d-9aa04e4348fa-istio-data\") pod \"e2e-ipp-a-a7ca1f-openshift-default-77b7bf58df-l8qlx\" (UID: \"6c75a860-08fd-4293-8c2d-9aa04e4348fa\") " pod="openshift-ingress/e2e-ipp-a-a7ca1f-openshift-default-77b7bf58df-l8qlx"
Oct 30 01:18:05.877096 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:05.877020    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c75a860-08fd-4293-8c2d-9aa04e4348fa-istio-token\") pod \"e2e-ipp-a-a7ca1f-openshift-default-77b7bf58df-l8qlx\" (UID: \"6c75a860-08fd-4293-8c2d-9aa04e4348fa\") " pod="openshift-ingress/e2e-ipp-a-a7ca1f-openshift-default-77b7bf58df-l8qlx"
Oct 30 01:18:05.877096 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:05.877055    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c75a860-08fd-4293-8c2d-9aa04e4348fa-workload-certs\") pod \"e2e-ipp-a-a7ca1f-openshift-default-77b7bf58df-l8qlx\" (UID: \"6c75a860-08fd-4293-8c2d-9aa04e4348fa\") " pod="openshift-ingress/e2e-ipp-a-a7ca1f-openshift-default-77b7bf58df-l8qlx"
Oct 30 01:18:05.877096 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:05.877085    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c75a860-08fd-4293-8c2d-9aa04e4348fa-workload-socket\") pod \"e2e-ipp-a-a7ca1f-openshift-default-77b7bf58df-l8qlx\" (UID: \"6c75a860-08fd-4293-8c2d-9aa04e4348fa\") " pod="openshift-ingress/e2e-ipp-a-a7ca1f-openshift-default-77b7bf58df-l8qlx"
Oct 30 01:18:05.877334 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:05.877121    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c75a860-08fd-4293-8c2d-9aa04e4348fa-istio-envoy\") pod \"e2e-ipp-a-a7ca1f-openshift-default-77b7bf58df-l8qlx\" (UID: \"6c75a860-08fd-4293-8c2d-9aa04e4348fa\") " pod="openshift-ingress/e2e-ipp-a-a7ca1f-openshift-default-77b7bf58df-l8qlx"
Oct 30 01:18:05.877334 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:05.877274    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c75a860-08fd-4293-8c2d-9aa04e4348fa-istiod-ca-cert\") pod \"e2e-ipp-a-a7ca1f-openshift-default-77b7bf58df-l8qlx\" (UID: \"6c75a860-08fd-4293-8c2d-9aa04e4348fa\") " pod="openshift-ingress/e2e-ipp-a-a7ca1f-openshift-default-77b7bf58df-l8qlx"
Oct 30 01:18:05.877334 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:05.877310    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c75a860-08fd-4293-8c2d-9aa04e4348fa-credential-socket\") pod \"e2e-ipp-a-a7ca1f-openshift-default-77b7bf58df-l8qlx\" (UID: \"6c75a860-08fd-4293-8c2d-9aa04e4348fa\") " pod="openshift-ingress/e2e-ipp-a-a7ca1f-openshift-default-77b7bf58df-l8qlx"
Oct 30 01:18:05.877524 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:05.877349    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c75a860-08fd-4293-8c2d-9aa04e4348fa-istio-podinfo\") pod \"e2e-ipp-a-a7ca1f-openshift-default-77b7bf58df-l8qlx\" (UID: \"6c75a860-08fd-4293-8c2d-9aa04e4348fa\") " pod="openshift-ingress/e2e-ipp-a-a7ca1f-openshift-default-77b7bf58df-l8qlx"
Oct 30 01:18:05.978760 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:05.978724    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c75a860-08fd-4293-8c2d-9aa04e4348fa-istiod-ca-cert\") pod \"e2e-ipp-a-a7ca1f-openshift-default-77b7bf58df-l8qlx\" (UID: \"6c75a860-08fd-4293-8c2d-9aa04e4348fa\") " pod="openshift-ingress/e2e-ipp-a-a7ca1f-openshift-default-77b7bf58df-l8qlx"
Oct 30 01:18:05.978760 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:05.978772    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c75a860-08fd-4293-8c2d-9aa04e4348fa-credential-socket\") pod \"e2e-ipp-a-a7ca1f-openshift-default-77b7bf58df-l8qlx\" (UID: \"6c75a860-08fd-4293-8c2d-9aa04e4348fa\") " pod="openshift-ingress/e2e-ipp-a-a7ca1f-openshift-default-77b7bf58df-l8qlx"
Oct 30 01:18:05.979035 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:05.978801    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c75a860-08fd-4293-8c2d-9aa04e4348fa-istio-podinfo\") pod \"e2e-ipp-a-a7ca1f-openshift-default-77b7bf58df-l8qlx\" (UID: \"6c75a860-08fd-4293-8c2d-9aa04e4348fa\") " pod="openshift-ingress/e2e-ipp-a-a7ca1f-openshift-default-77b7bf58df-l8qlx"
Oct 30 01:18:05.979035 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:05.978879    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lt4jg\" (UniqueName: \"kubernetes.io/projected/6c75a860-08fd-4293-8c2d-9aa04e4348fa-kube-api-access-lt4jg\") pod \"e2e-ipp-a-a7ca1f-openshift-default-77b7bf58df-l8qlx\" (UID: \"6c75a860-08fd-4293-8c2d-9aa04e4348fa\") " pod="openshift-ingress/e2e-ipp-a-a7ca1f-openshift-default-77b7bf58df-l8qlx"
Oct 30 01:18:05.979035 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:05.978907    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c75a860-08fd-4293-8c2d-9aa04e4348fa-istio-data\") pod \"e2e-ipp-a-a7ca1f-openshift-default-77b7bf58df-l8qlx\" (UID: \"6c75a860-08fd-4293-8c2d-9aa04e4348fa\") " pod="openshift-ingress/e2e-ipp-a-a7ca1f-openshift-default-77b7bf58df-l8qlx"
Oct 30 01:18:05.979035 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:05.978965    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c75a860-08fd-4293-8c2d-9aa04e4348fa-istio-token\") pod \"e2e-ipp-a-a7ca1f-openshift-default-77b7bf58df-l8qlx\" (UID: \"6c75a860-08fd-4293-8c2d-9aa04e4348fa\") " pod="openshift-ingress/e2e-ipp-a-a7ca1f-openshift-default-77b7bf58df-l8qlx"
Oct 30 01:18:05.979035 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:05.979001    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c75a860-08fd-4293-8c2d-9aa04e4348fa-workload-certs\") pod \"e2e-ipp-a-a7ca1f-openshift-default-77b7bf58df-l8qlx\" (UID: \"6c75a860-08fd-4293-8c2d-9aa04e4348fa\") " pod="openshift-ingress/e2e-ipp-a-a7ca1f-openshift-default-77b7bf58df-l8qlx"
Oct 30 01:18:05.979035 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:05.979030    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c75a860-08fd-4293-8c2d-9aa04e4348fa-workload-socket\") pod \"e2e-ipp-a-a7ca1f-openshift-default-77b7bf58df-l8qlx\" (UID: \"6c75a860-08fd-4293-8c2d-9aa04e4348fa\") " pod="openshift-ingress/e2e-ipp-a-a7ca1f-openshift-default-77b7bf58df-l8qlx"
Oct 30 01:18:05.979394 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:05.979071    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c75a860-08fd-4293-8c2d-9aa04e4348fa-istio-envoy\") pod \"e2e-ipp-a-a7ca1f-openshift-default-77b7bf58df-l8qlx\" (UID: \"6c75a860-08fd-4293-8c2d-9aa04e4348fa\") " pod="openshift-ingress/e2e-ipp-a-a7ca1f-openshift-default-77b7bf58df-l8qlx"
Oct 30 01:18:05.979394 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:05.979236    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c75a860-08fd-4293-8c2d-9aa04e4348fa-credential-socket\") pod \"e2e-ipp-a-a7ca1f-openshift-default-77b7bf58df-l8qlx\" (UID: \"6c75a860-08fd-4293-8c2d-9aa04e4348fa\") " pod="openshift-ingress/e2e-ipp-a-a7ca1f-openshift-default-77b7bf58df-l8qlx"
Oct 30 01:18:05.979604 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:05.979573    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c75a860-08fd-4293-8c2d-9aa04e4348fa-istio-data\") pod \"e2e-ipp-a-a7ca1f-openshift-default-77b7bf58df-l8qlx\" (UID: \"6c75a860-08fd-4293-8c2d-9aa04e4348fa\") " pod="openshift-ingress/e2e-ipp-a-a7ca1f-openshift-default-77b7bf58df-l8qlx"
Oct 30 01:18:05.979750 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:05.979619    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c75a860-08fd-4293-8c2d-9aa04e4348fa-istiod-ca-cert\") pod \"e2e-ipp-a-a7ca1f-openshift-default-77b7bf58df-l8qlx\" (UID: \"6c75a860-08fd-4293-8c2d-9aa04e4348fa\") " pod="openshift-ingress/e2e-ipp-a-a7ca1f-openshift-default-77b7bf58df-l8qlx"
Oct 30 01:18:05.979827 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:05.979778    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c75a860-08fd-4293-8c2d-9aa04e4348fa-workload-certs\") pod \"e2e-ipp-a-a7ca1f-openshift-default-77b7bf58df-l8qlx\" (UID: \"6c75a860-08fd-4293-8c2d-9aa04e4348fa\") " pod="openshift-ingress/e2e-ipp-a-a7ca1f-openshift-default-77b7bf58df-l8qlx"
Oct 30 01:18:05.979914 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:05.979892    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c75a860-08fd-4293-8c2d-9aa04e4348fa-workload-socket\") pod \"e2e-ipp-a-a7ca1f-openshift-default-77b7bf58df-l8qlx\" (UID: \"6c75a860-08fd-4293-8c2d-9aa04e4348fa\") " pod="openshift-ingress/e2e-ipp-a-a7ca1f-openshift-default-77b7bf58df-l8qlx"
Oct 30 01:18:05.981426 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:05.981402    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c75a860-08fd-4293-8c2d-9aa04e4348fa-istio-envoy\") pod \"e2e-ipp-a-a7ca1f-openshift-default-77b7bf58df-l8qlx\" (UID: \"6c75a860-08fd-4293-8c2d-9aa04e4348fa\") " pod="openshift-ingress/e2e-ipp-a-a7ca1f-openshift-default-77b7bf58df-l8qlx"
Oct 30 01:18:05.981678 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:05.981655    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c75a860-08fd-4293-8c2d-9aa04e4348fa-istio-podinfo\") pod \"e2e-ipp-a-a7ca1f-openshift-default-77b7bf58df-l8qlx\" (UID: \"6c75a860-08fd-4293-8c2d-9aa04e4348fa\") " pod="openshift-ingress/e2e-ipp-a-a7ca1f-openshift-default-77b7bf58df-l8qlx"
Oct 30 01:18:06.004324 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:06.004292    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c75a860-08fd-4293-8c2d-9aa04e4348fa-istio-token\") pod \"e2e-ipp-a-a7ca1f-openshift-default-77b7bf58df-l8qlx\" (UID: \"6c75a860-08fd-4293-8c2d-9aa04e4348fa\") " pod="openshift-ingress/e2e-ipp-a-a7ca1f-openshift-default-77b7bf58df-l8qlx"
Oct 30 01:18:06.004566 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:06.004541    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lt4jg\" (UniqueName: \"kubernetes.io/projected/6c75a860-08fd-4293-8c2d-9aa04e4348fa-kube-api-access-lt4jg\") pod \"e2e-ipp-a-a7ca1f-openshift-default-77b7bf58df-l8qlx\" (UID: \"6c75a860-08fd-4293-8c2d-9aa04e4348fa\") " pod="openshift-ingress/e2e-ipp-a-a7ca1f-openshift-default-77b7bf58df-l8qlx"
Oct 30 01:18:06.129884 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:06.129793    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-a7ca1f-openshift-default-77b7bf58df-l8qlx"
Oct 30 01:18:06.455989 ip-10-0-128-7 kubenswrapper[5002]: W1030 01:18:06.455954    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c75a860_08fd_4293_8c2d_9aa04e4348fa.slice/crio-c24b9f1466f945b4fc655d99f4e2ae77ee4d6c8cb51c2e31a96a410d030fa1da WatchSource:0}: Error finding container c24b9f1466f945b4fc655d99f4e2ae77ee4d6c8cb51c2e31a96a410d030fa1da: Status 404 returned error can't find the container with id c24b9f1466f945b4fc655d99f4e2ae77ee4d6c8cb51c2e31a96a410d030fa1da
Oct 30 01:18:06.457772 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:06.457745    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-a7ca1f-openshift-default-77b7bf58df-l8qlx"]
Oct 30 01:18:06.459588 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:06.459551    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:18:06.459701 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:06.459620    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:18:06.459701 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:06.459655    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:18:06.790879 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:06.790757    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a7ca1f-openshift-default-77b7bf58df-l8qlx" event={"ID":"6c75a860-08fd-4293-8c2d-9aa04e4348fa","Type":"ContainerStarted","Data":"7e12842879ac1d33d95b87234c9f4a05a54894efc661d754b0f43ad0562306a8"}
Oct 30 01:18:06.791046 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:06.790874    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a7ca1f-openshift-default-77b7bf58df-l8qlx" event={"ID":"6c75a860-08fd-4293-8c2d-9aa04e4348fa","Type":"ContainerStarted","Data":"c24b9f1466f945b4fc655d99f4e2ae77ee4d6c8cb51c2e31a96a410d030fa1da"}
Oct 30 01:18:06.854893 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:06.854831    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-a7ca1f-openshift-default-77b7bf58df-l8qlx" podStartSLOduration=1.854816992 podStartE2EDuration="1.854816992s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:18:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:18:06.85215081 +0000 UTC m=+2782.468583692" watchObservedRunningTime="2026-10-30 01:18:06.854816992 +0000 UTC m=+2782.471249868"
Oct 30 01:18:07.137481 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:07.131272    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-a7ca1f-openshift-default-77b7bf58df-l8qlx"
Oct 30 01:18:07.137481 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:07.134820    5002 patch_prober.go:29] interesting pod/e2e-ipp-a-a7ca1f-openshift-default-77b7bf58df-l8qlx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.7:15021/healthz/ready\": dial tcp 10.132.1.7:15021: connect: connection refused" start-of-body=
Oct 30 01:18:07.137481 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:07.134893    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a7ca1f-openshift-default-77b7bf58df-l8qlx" podUID="6c75a860-08fd-4293-8c2d-9aa04e4348fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.7:15021/healthz/ready\": dial tcp 10.132.1.7:15021: connect: connection refused"
Oct 30 01:18:08.130819 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:08.130779    5002 patch_prober.go:29] interesting pod/e2e-ipp-a-a7ca1f-openshift-default-77b7bf58df-l8qlx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.7:15021/healthz/ready\": dial tcp 10.132.1.7:15021: connect: connection refused" start-of-body=
Oct 30 01:18:08.131269 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:08.130845    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a7ca1f-openshift-default-77b7bf58df-l8qlx" podUID="6c75a860-08fd-4293-8c2d-9aa04e4348fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.7:15021/healthz/ready\": dial tcp 10.132.1.7:15021: connect: connection refused"
Oct 30 01:18:09.130185 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:09.130148    5002 patch_prober.go:29] interesting pod/e2e-ipp-a-a7ca1f-openshift-default-77b7bf58df-l8qlx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.7:15021/healthz/ready\": dial tcp 10.132.1.7:15021: connect: connection refused" start-of-body=
Oct 30 01:18:09.130440 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:09.130209    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a7ca1f-openshift-default-77b7bf58df-l8qlx" podUID="6c75a860-08fd-4293-8c2d-9aa04e4348fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.7:15021/healthz/ready\": dial tcp 10.132.1.7:15021: connect: connection refused"
Oct 30 01:18:10.030699 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:10.030662    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e7e87d1-db4wf"]
Oct 30 01:18:10.044551 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:10.044515    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e7e87d1-db4wf"]
Oct 30 01:18:10.130844 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:10.130805    5002 patch_prober.go:29] interesting pod/e2e-ipp-a-a7ca1f-openshift-default-77b7bf58df-l8qlx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.7:15021/healthz/ready\": dial tcp 10.132.1.7:15021: connect: connection refused" start-of-body=
Oct 30 01:18:10.131017 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:10.130876    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a7ca1f-openshift-default-77b7bf58df-l8qlx" podUID="6c75a860-08fd-4293-8c2d-9aa04e4348fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.7:15021/healthz/ready\": dial tcp 10.132.1.7:15021: connect: connection refused"
Oct 30 01:18:10.495682 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:10.495643    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d1761e3-ec85-4fbe-9b17-323cd7b220d3" path="/var/lib/kubelet/pods/6d1761e3-ec85-4fbe-9b17-323cd7b220d3/volumes"
Oct 30 01:18:11.130341 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:11.130306    5002 patch_prober.go:29] interesting pod/e2e-ipp-a-a7ca1f-openshift-default-77b7bf58df-l8qlx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.7:15021/healthz/ready\": dial tcp 10.132.1.7:15021: connect: connection refused" start-of-body=
Oct 30 01:18:11.130853 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:11.130367    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a7ca1f-openshift-default-77b7bf58df-l8qlx" podUID="6c75a860-08fd-4293-8c2d-9aa04e4348fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.7:15021/healthz/ready\": dial tcp 10.132.1.7:15021: connect: connection refused"
Oct 30 01:18:12.130311 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:12.130267    5002 patch_prober.go:29] interesting pod/e2e-ipp-a-a7ca1f-openshift-default-77b7bf58df-l8qlx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.7:15021/healthz/ready\": dial tcp 10.132.1.7:15021: connect: connection refused" start-of-body=
Oct 30 01:18:12.130560 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:12.130333    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a7ca1f-openshift-default-77b7bf58df-l8qlx" podUID="6c75a860-08fd-4293-8c2d-9aa04e4348fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.7:15021/healthz/ready\": dial tcp 10.132.1.7:15021: connect: connection refused"
Oct 30 01:18:12.879850 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:18:12.879740    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbjdq" containerName="extractor"
Oct 30 01:18:13.423349 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:13.423296    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-a7ca1f-openshift-default-77b7bf58df-l8qlx"
Oct 30 01:18:13.443235 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:13.441776    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-a7ca1f-openshift-default-77b7bf58df-l8qlx"
Oct 30 01:18:13.448062 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:13.443652    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-a7ca1f-openshift-default-77b7bf58df-l8qlx"
Oct 30 01:18:19.855787 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:19.855742    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7d8z4"]
Oct 30 01:18:20.332662 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:20.332609    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-812ba2-7768d47f6d-bfbkj"]
Oct 30 01:18:25.333705 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:25.333646    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-812ba2-7768d47f6d-bfbkj" podUID="4f8a6baf-d556-4d26-96a2-11ef06fac9ea" containerName="payload-processing" containerID="cri-o://22abfca05e4831f40a63f07f347cf676c1efe426b92941f255716638ea1520e1" gracePeriod=25
Oct 30 01:18:25.661887 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:25.661852    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-812ba2-7768d47f6d-bfbkj"
Oct 30 01:18:25.811534 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:25.811495    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f8a6baf-d556-4d26-96a2-11ef06fac9ea-kube-api-access-c58hv\" (UniqueName: \"kubernetes.io/projected/4f8a6baf-d556-4d26-96a2-11ef06fac9ea-kube-api-access-c58hv\") pod \"4f8a6baf-d556-4d26-96a2-11ef06fac9ea\" (UID: \"4f8a6baf-d556-4d26-96a2-11ef06fac9ea\") "
Oct 30 01:18:25.811731 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:25.811588    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f8a6baf-d556-4d26-96a2-11ef06fac9ea-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f8a6baf-d556-4d26-96a2-11ef06fac9ea-plugins-config-volume\") pod \"4f8a6baf-d556-4d26-96a2-11ef06fac9ea\" (UID: \"4f8a6baf-d556-4d26-96a2-11ef06fac9ea\") "
Oct 30 01:18:25.811963 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:25.811939    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f8a6baf-d556-4d26-96a2-11ef06fac9ea-plugins-config-volume" pod "4f8a6baf-d556-4d26-96a2-11ef06fac9ea" (UID: "4f8a6baf-d556-4d26-96a2-11ef06fac9ea"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:18:25.812280 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:25.812260    5002 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f8a6baf-d556-4d26-96a2-11ef06fac9ea-plugins-config-volume\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:18:25.813805 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:25.813778    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f8a6baf-d556-4d26-96a2-11ef06fac9ea-kube-api-access-c58hv" pod "4f8a6baf-d556-4d26-96a2-11ef06fac9ea" (UID: "4f8a6baf-d556-4d26-96a2-11ef06fac9ea"). InnerVolumeSpecName "kube-api-access-c58hv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:18:25.913905 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:25.913812    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c58hv\" (UniqueName: \"kubernetes.io/projected/4f8a6baf-d556-4d26-96a2-11ef06fac9ea-kube-api-access-c58hv\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:18:26.008848 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:26.008811    5002 generic.go:356] "Generic (PLEG): container finished" podID="4f8a6baf-d556-4d26-96a2-11ef06fac9ea" containerID="22abfca05e4831f40a63f07f347cf676c1efe426b92941f255716638ea1520e1" exitCode=0
Oct 30 01:18:26.009050 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:26.008867    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-812ba2-7768d47f6d-bfbkj"
Oct 30 01:18:26.009050 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:26.008893    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-812ba2-7768d47f6d-bfbkj" event={"ID":"4f8a6baf-d556-4d26-96a2-11ef06fac9ea","Type":"ContainerDied","Data":"22abfca05e4831f40a63f07f347cf676c1efe426b92941f255716638ea1520e1"}
Oct 30 01:18:26.009050 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:26.008946    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-812ba2-7768d47f6d-bfbkj" event={"ID":"4f8a6baf-d556-4d26-96a2-11ef06fac9ea","Type":"ContainerDied","Data":"da88e5b5a84ddaa489e53df58a77c451862342d00ef557c724bde6bea174b1c8"}
Oct 30 01:18:26.009050 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:26.008957    5002 scope.go:122] "RemoveContainer" containerID="22abfca05e4831f40a63f07f347cf676c1efe426b92941f255716638ea1520e1"
Oct 30 01:18:26.020160 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:26.020142    5002 scope.go:122] "RemoveContainer" containerID="22abfca05e4831f40a63f07f347cf676c1efe426b92941f255716638ea1520e1"
Oct 30 01:18:26.020455 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:18:26.020436    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22abfca05e4831f40a63f07f347cf676c1efe426b92941f255716638ea1520e1\": container with ID starting with 22abfca05e4831f40a63f07f347cf676c1efe426b92941f255716638ea1520e1 not found: ID does not exist" containerID="22abfca05e4831f40a63f07f347cf676c1efe426b92941f255716638ea1520e1"
Oct 30 01:18:26.020520 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:26.020467    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22abfca05e4831f40a63f07f347cf676c1efe426b92941f255716638ea1520e1"} err="failed to get container status \"22abfca05e4831f40a63f07f347cf676c1efe426b92941f255716638ea1520e1\": rpc error: code = NotFound desc = could not find container \"22abfca05e4831f40a63f07f347cf676c1efe426b92941f255716638ea1520e1\": container with ID starting with 22abfca05e4831f40a63f07f347cf676c1efe426b92941f255716638ea1520e1 not found: ID does not exist"
Oct 30 01:18:26.067609 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:26.067570    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-812ba2-7768d47f6d-bfbkj"]
Oct 30 01:18:26.082763 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:26.082731    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-812ba2-7768d47f6d-bfbkj"]
Oct 30 01:18:26.497141 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:26.497099    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f8a6baf-d556-4d26-96a2-11ef06fac9ea" path="/var/lib/kubelet/pods/4f8a6baf-d556-4d26-96a2-11ef06fac9ea/volumes"
Oct 30 01:18:28.752039 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:28.752002    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a7ca1f-kvl7v"]
Oct 30 01:18:28.752774 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:28.752753    5002 cpu_manager.go:414] "containerMap: removing container" podUID="4f8a6baf-d556-4d26-96a2-11ef06fac9ea" containerName="payload-processing"
Oct 30 01:18:28.752774 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:28.752775    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f8a6baf-d556-4d26-96a2-11ef06fac9ea" containerName="payload-processing"
Oct 30 01:18:28.752889 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:28.752873    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f8a6baf-d556-4d26-96a2-11ef06fac9ea" containerName="payload-processing"
Oct 30 01:18:28.758034 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:28.758008    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a7ca1f-kvl7v"
Oct 30 01:18:28.779757 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:28.779724    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a7ca1f-kvl7v"]
Oct 30 01:18:28.788765 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:28.788736    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:18:28.789002 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:28.788483    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:18:28.789940 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:28.789891    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5phsh\""
Oct 30 01:18:28.790198 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:28.790180    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:18:28.865183 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:28.865149    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/613b69b3-822c-4468-b9bb-18b34cca8ba5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-a7ca1f-kvl7v\" (UID: \"613b69b3-822c-4468-b9bb-18b34cca8ba5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a7ca1f-kvl7v"
Oct 30 01:18:28.865397 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:28.865219    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/613b69b3-822c-4468-b9bb-18b34cca8ba5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-a7ca1f-kvl7v\" (UID: \"613b69b3-822c-4468-b9bb-18b34cca8ba5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a7ca1f-kvl7v"
Oct 30 01:18:28.865397 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:28.865316    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvq79\" (UniqueName: \"kubernetes.io/projected/613b69b3-822c-4468-b9bb-18b34cca8ba5-kube-api-access-jvq79\") pod \"maas-api-revoke-keys-e2e-ipp-a-a7ca1f-kvl7v\" (UID: \"613b69b3-822c-4468-b9bb-18b34cca8ba5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a7ca1f-kvl7v"
Oct 30 01:18:28.966937 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:28.966894    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvq79\" (UniqueName: \"kubernetes.io/projected/613b69b3-822c-4468-b9bb-18b34cca8ba5-kube-api-access-jvq79\") pod \"maas-api-revoke-keys-e2e-ipp-a-a7ca1f-kvl7v\" (UID: \"613b69b3-822c-4468-b9bb-18b34cca8ba5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a7ca1f-kvl7v"
Oct 30 01:18:28.967187 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:28.967153    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/613b69b3-822c-4468-b9bb-18b34cca8ba5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-a7ca1f-kvl7v\" (UID: \"613b69b3-822c-4468-b9bb-18b34cca8ba5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a7ca1f-kvl7v"
Oct 30 01:18:28.967272 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:28.967235    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/613b69b3-822c-4468-b9bb-18b34cca8ba5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-a7ca1f-kvl7v\" (UID: \"613b69b3-822c-4468-b9bb-18b34cca8ba5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a7ca1f-kvl7v"
Oct 30 01:18:28.968246 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:28.968219    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/613b69b3-822c-4468-b9bb-18b34cca8ba5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-a7ca1f-kvl7v\" (UID: \"613b69b3-822c-4468-b9bb-18b34cca8ba5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a7ca1f-kvl7v"
Oct 30 01:18:28.970055 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:28.970032    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/613b69b3-822c-4468-b9bb-18b34cca8ba5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-a7ca1f-kvl7v\" (UID: \"613b69b3-822c-4468-b9bb-18b34cca8ba5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a7ca1f-kvl7v"
Oct 30 01:18:28.992082 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:28.992052    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvq79\" (UniqueName: \"kubernetes.io/projected/613b69b3-822c-4468-b9bb-18b34cca8ba5-kube-api-access-jvq79\") pod \"maas-api-revoke-keys-e2e-ipp-a-a7ca1f-kvl7v\" (UID: \"613b69b3-822c-4468-b9bb-18b34cca8ba5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a7ca1f-kvl7v"
Oct 30 01:18:29.076907 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:29.076820    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a7ca1f-kvl7v"
Oct 30 01:18:29.385826 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:29.385788    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a7ca1f-kvl7v"]
Oct 30 01:18:30.063457 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:30.063406    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a7ca1f-kvl7v" event={"ID":"613b69b3-822c-4468-b9bb-18b34cca8ba5","Type":"ContainerStarted","Data":"01f558c05d63c0d1920299adfb232c13fece79e41d37df9ec69c162cb77d21a1"}
Oct 30 01:18:30.064081 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:30.064045    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a7ca1f-kvl7v" event={"ID":"613b69b3-822c-4468-b9bb-18b34cca8ba5","Type":"ContainerStarted","Data":"57321e851824a37a5b4c91567eba4e8001425becda32b8deace7f91c87c7f7f5"}
Oct 30 01:18:30.109879 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:30.109815    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a7ca1f-kvl7v" podStartSLOduration=2.109800104 podStartE2EDuration="2.109800104s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:18:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:18:30.108629036 +0000 UTC m=+2805.725061916" watchObservedRunningTime="2026-10-30 01:18:30.109800104 +0000 UTC m=+2805.726232983"
Oct 30 01:18:31.076526 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:31.076487    5002 generic.go:356] "Generic (PLEG): container finished" podID="613b69b3-822c-4468-b9bb-18b34cca8ba5" containerID="01f558c05d63c0d1920299adfb232c13fece79e41d37df9ec69c162cb77d21a1" exitCode=0
Oct 30 01:18:31.077001 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:31.076575    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a7ca1f-kvl7v" event={"ID":"613b69b3-822c-4468-b9bb-18b34cca8ba5","Type":"ContainerDied","Data":"01f558c05d63c0d1920299adfb232c13fece79e41d37df9ec69c162cb77d21a1"}
Oct 30 01:18:32.315058 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:32.315028    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a7ca1f-kvl7v"
Oct 30 01:18:32.343669 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:32.343637    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/613b69b3-822c-4468-b9bb-18b34cca8ba5-kube-api-access-jvq79\" (UniqueName: \"kubernetes.io/projected/613b69b3-822c-4468-b9bb-18b34cca8ba5-kube-api-access-jvq79\") pod \"613b69b3-822c-4468-b9bb-18b34cca8ba5\" (UID: \"613b69b3-822c-4468-b9bb-18b34cca8ba5\") "
Oct 30 01:18:32.343846 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:32.343734    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/613b69b3-822c-4468-b9bb-18b34cca8ba5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/613b69b3-822c-4468-b9bb-18b34cca8ba5-maas-api-xks-ca\") pod \"613b69b3-822c-4468-b9bb-18b34cca8ba5\" (UID: \"613b69b3-822c-4468-b9bb-18b34cca8ba5\") "
Oct 30 01:18:32.343846 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:32.343765    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/613b69b3-822c-4468-b9bb-18b34cca8ba5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/613b69b3-822c-4468-b9bb-18b34cca8ba5-maas-api-openshift-ca\") pod \"613b69b3-822c-4468-b9bb-18b34cca8ba5\" (UID: \"613b69b3-822c-4468-b9bb-18b34cca8ba5\") "
Oct 30 01:18:32.344310 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:32.344247    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/613b69b3-822c-4468-b9bb-18b34cca8ba5-maas-api-openshift-ca" pod "613b69b3-822c-4468-b9bb-18b34cca8ba5" (UID: "613b69b3-822c-4468-b9bb-18b34cca8ba5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:18:32.346282 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:32.346250    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/613b69b3-822c-4468-b9bb-18b34cca8ba5-maas-api-xks-ca" pod "613b69b3-822c-4468-b9bb-18b34cca8ba5" (UID: "613b69b3-822c-4468-b9bb-18b34cca8ba5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:18:32.346282 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:32.346266    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/613b69b3-822c-4468-b9bb-18b34cca8ba5-kube-api-access-jvq79" pod "613b69b3-822c-4468-b9bb-18b34cca8ba5" (UID: "613b69b3-822c-4468-b9bb-18b34cca8ba5"). InnerVolumeSpecName "kube-api-access-jvq79". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:18:32.445205 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:32.445166    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvq79\" (UniqueName: \"kubernetes.io/projected/613b69b3-822c-4468-b9bb-18b34cca8ba5-kube-api-access-jvq79\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:18:32.445205 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:32.445208    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/613b69b3-822c-4468-b9bb-18b34cca8ba5-maas-api-xks-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:18:32.445456 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:32.445223    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/613b69b3-822c-4468-b9bb-18b34cca8ba5-maas-api-openshift-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:18:33.150072 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:33.150044    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a7ca1f-kvl7v"
Oct 30 01:18:33.150072 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:33.150058    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a7ca1f-kvl7v" event={"ID":"613b69b3-822c-4468-b9bb-18b34cca8ba5","Type":"ContainerDied","Data":"57321e851824a37a5b4c91567eba4e8001425becda32b8deace7f91c87c7f7f5"}
Oct 30 01:18:33.150294 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:33.150103    5002 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="57321e851824a37a5b4c91567eba4e8001425becda32b8deace7f91c87c7f7f5"
Oct 30 01:18:39.331007 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:39.330962    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-29be73-rr7m4"]
Oct 30 01:18:39.344688 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:39.344651    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-29be73-rr7m4"]
Oct 30 01:18:40.495402 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:40.495349    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3df7d750-133f-4d48-a34e-d425ea88031e" path="/var/lib/kubelet/pods/3df7d750-133f-4d48-a34e-d425ea88031e/volumes"
Oct 30 01:18:43.000204 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:18:43.000158    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbjdq" containerName="extractor"
Oct 30 01:18:46.241881 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:46.241831    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-5tcsp"]
Oct 30 01:18:46.243397 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:46.243141    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-5tcsp" podUID="62422ac3-d47e-40a0-8997-90a3fba2a424" containerName="istio-proxy" containerID="cri-o://8fc8dd904bcddbd6ed9a21b6d6a5543ab1cd58452f15e943d350738b78f81f16" gracePeriod=30
Oct 30 01:18:47.326395 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:47.326354    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed4e8e7a-2xvx7"]
Oct 30 01:18:47.326978 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:47.326963    5002 cpu_manager.go:414] "containerMap: removing container" podUID="613b69b3-822c-4468-b9bb-18b34cca8ba5" containerName="revoke-keys"
Oct 30 01:18:47.327023 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:47.326978    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="613b69b3-822c-4468-b9bb-18b34cca8ba5" containerName="revoke-keys"
Oct 30 01:18:47.327105 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:47.327094    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="613b69b3-822c-4468-b9bb-18b34cca8ba5" containerName="revoke-keys"
Oct 30 01:18:47.330825 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:47.330802    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed4e8e7a-2xvx7"
Oct 30 01:18:47.351174 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:47.351138    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed4e8e7a-2xvx7"]
Oct 30 01:18:47.359491 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:47.359447    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:18:47.359683 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:47.359453    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5phsh\""
Oct 30 01:18:47.359799 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:47.359583    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:18:47.362431 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:47.362341    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:18:47.388527 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:47.388490    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05abbb95-d56a-4abb-a43f-c043b2266219-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ed4e8e7a-2xvx7\" (UID: \"05abbb95-d56a-4abb-a43f-c043b2266219\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed4e8e7a-2xvx7"
Oct 30 01:18:47.388854 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:47.388828    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2z59j\" (UniqueName: \"kubernetes.io/projected/05abbb95-d56a-4abb-a43f-c043b2266219-kube-api-access-2z59j\") pod \"maas-api-revoke-sub-ed4e8e7a-2xvx7\" (UID: \"05abbb95-d56a-4abb-a43f-c043b2266219\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed4e8e7a-2xvx7"
Oct 30 01:18:47.389116 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:47.389092    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05abbb95-d56a-4abb-a43f-c043b2266219-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ed4e8e7a-2xvx7\" (UID: \"05abbb95-d56a-4abb-a43f-c043b2266219\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed4e8e7a-2xvx7"
Oct 30 01:18:47.446462 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:47.446424    5002 patch_prober.go:29] interesting pod/e2e-worker-w1-29be73-openshift-default-66467f5dcf-5tcsp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 01:18:47.446631 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:47.446487    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-5tcsp" podUID="62422ac3-d47e-40a0-8997-90a3fba2a424" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 01:18:47.490439 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:47.490367    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05abbb95-d56a-4abb-a43f-c043b2266219-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ed4e8e7a-2xvx7\" (UID: \"05abbb95-d56a-4abb-a43f-c043b2266219\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed4e8e7a-2xvx7"
Oct 30 01:18:47.490644 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:47.490491    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2z59j\" (UniqueName: \"kubernetes.io/projected/05abbb95-d56a-4abb-a43f-c043b2266219-kube-api-access-2z59j\") pod \"maas-api-revoke-sub-ed4e8e7a-2xvx7\" (UID: \"05abbb95-d56a-4abb-a43f-c043b2266219\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed4e8e7a-2xvx7"
Oct 30 01:18:47.490644 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:47.490606    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05abbb95-d56a-4abb-a43f-c043b2266219-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ed4e8e7a-2xvx7\" (UID: \"05abbb95-d56a-4abb-a43f-c043b2266219\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed4e8e7a-2xvx7"
Oct 30 01:18:47.491256 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:47.491233    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05abbb95-d56a-4abb-a43f-c043b2266219-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ed4e8e7a-2xvx7\" (UID: \"05abbb95-d56a-4abb-a43f-c043b2266219\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed4e8e7a-2xvx7"
Oct 30 01:18:47.493026 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:47.492995    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05abbb95-d56a-4abb-a43f-c043b2266219-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ed4e8e7a-2xvx7\" (UID: \"05abbb95-d56a-4abb-a43f-c043b2266219\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed4e8e7a-2xvx7"
Oct 30 01:18:47.515678 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:47.515650    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2z59j\" (UniqueName: \"kubernetes.io/projected/05abbb95-d56a-4abb-a43f-c043b2266219-kube-api-access-2z59j\") pod \"maas-api-revoke-sub-ed4e8e7a-2xvx7\" (UID: \"05abbb95-d56a-4abb-a43f-c043b2266219\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed4e8e7a-2xvx7"
Oct 30 01:18:47.645361 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:47.645258    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed4e8e7a-2xvx7"
Oct 30 01:18:48.023694 ip-10-0-128-7 kubenswrapper[5002]: W1030 01:18:48.023664    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05abbb95_d56a_4abb_a43f_c043b2266219.slice/crio-a3734f982dda07a6c922c2909a0bd2d0232447b0df4f7248a2335a7581bcff05 WatchSource:0}: Error finding container a3734f982dda07a6c922c2909a0bd2d0232447b0df4f7248a2335a7581bcff05: Status 404 returned error can't find the container with id a3734f982dda07a6c922c2909a0bd2d0232447b0df4f7248a2335a7581bcff05
Oct 30 01:18:48.024393 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:48.024346    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed4e8e7a-2xvx7"]
Oct 30 01:18:48.405596 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:48.405555    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed4e8e7a-2xvx7" event={"ID":"05abbb95-d56a-4abb-a43f-c043b2266219","Type":"ContainerStarted","Data":"2c800e74243482d0afbd6cf7d107f8938f43eaa6c42b74c53266e03caf681283"}
Oct 30 01:18:48.406036 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:48.405618    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed4e8e7a-2xvx7" event={"ID":"05abbb95-d56a-4abb-a43f-c043b2266219","Type":"ContainerStarted","Data":"a3734f982dda07a6c922c2909a0bd2d0232447b0df4f7248a2335a7581bcff05"}
Oct 30 01:18:48.452368 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:48.452302    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed4e8e7a-2xvx7" podStartSLOduration=1.452282266 podStartE2EDuration="1.452282266s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:18:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:18:48.448440403 +0000 UTC m=+2824.064873298" watchObservedRunningTime="2026-10-30 01:18:48.452282266 +0000 UTC m=+2824.068715145"
Oct 30 01:18:49.439214 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:49.439171    5002 generic.go:356] "Generic (PLEG): container finished" podID="05abbb95-d56a-4abb-a43f-c043b2266219" containerID="2c800e74243482d0afbd6cf7d107f8938f43eaa6c42b74c53266e03caf681283" exitCode=0
Oct 30 01:18:49.439765 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:49.439337    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed4e8e7a-2xvx7" event={"ID":"05abbb95-d56a-4abb-a43f-c043b2266219","Type":"ContainerDied","Data":"2c800e74243482d0afbd6cf7d107f8938f43eaa6c42b74c53266e03caf681283"}
Oct 30 01:18:50.630493 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:50.630464    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed4e8e7a-2xvx7"
Oct 30 01:18:50.653473 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:50.653434    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05abbb95-d56a-4abb-a43f-c043b2266219-kube-api-access-2z59j\" (UniqueName: \"kubernetes.io/projected/05abbb95-d56a-4abb-a43f-c043b2266219-kube-api-access-2z59j\") pod \"05abbb95-d56a-4abb-a43f-c043b2266219\" (UID: \"05abbb95-d56a-4abb-a43f-c043b2266219\") "
Oct 30 01:18:50.653642 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:50.653487    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05abbb95-d56a-4abb-a43f-c043b2266219-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05abbb95-d56a-4abb-a43f-c043b2266219-maas-api-openshift-ca\") pod \"05abbb95-d56a-4abb-a43f-c043b2266219\" (UID: \"05abbb95-d56a-4abb-a43f-c043b2266219\") "
Oct 30 01:18:50.653642 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:50.653598    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/05abbb95-d56a-4abb-a43f-c043b2266219-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05abbb95-d56a-4abb-a43f-c043b2266219-maas-api-xks-ca\") pod \"05abbb95-d56a-4abb-a43f-c043b2266219\" (UID: \"05abbb95-d56a-4abb-a43f-c043b2266219\") "
Oct 30 01:18:50.653839 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:50.653818    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05abbb95-d56a-4abb-a43f-c043b2266219-maas-api-openshift-ca" pod "05abbb95-d56a-4abb-a43f-c043b2266219" (UID: "05abbb95-d56a-4abb-a43f-c043b2266219"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:18:50.654294 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:50.654261    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05abbb95-d56a-4abb-a43f-c043b2266219-maas-api-openshift-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:18:50.655804 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:50.655772    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05abbb95-d56a-4abb-a43f-c043b2266219-kube-api-access-2z59j" pod "05abbb95-d56a-4abb-a43f-c043b2266219" (UID: "05abbb95-d56a-4abb-a43f-c043b2266219"). InnerVolumeSpecName "kube-api-access-2z59j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:18:50.655927 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:50.655772    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05abbb95-d56a-4abb-a43f-c043b2266219-maas-api-xks-ca" pod "05abbb95-d56a-4abb-a43f-c043b2266219" (UID: "05abbb95-d56a-4abb-a43f-c043b2266219"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:18:50.756028 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:50.755942    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2z59j\" (UniqueName: \"kubernetes.io/projected/05abbb95-d56a-4abb-a43f-c043b2266219-kube-api-access-2z59j\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:18:50.756028 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:50.755978    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05abbb95-d56a-4abb-a43f-c043b2266219-maas-api-xks-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:18:51.466092 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:51.466050    5002 generic.go:356] "Generic (PLEG): container finished" podID="62422ac3-d47e-40a0-8997-90a3fba2a424" containerID="8fc8dd904bcddbd6ed9a21b6d6a5543ab1cd58452f15e943d350738b78f81f16" exitCode=0
Oct 30 01:18:51.466340 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:51.466148    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-5tcsp" event={"ID":"62422ac3-d47e-40a0-8997-90a3fba2a424","Type":"ContainerDied","Data":"8fc8dd904bcddbd6ed9a21b6d6a5543ab1cd58452f15e943d350738b78f81f16"}
Oct 30 01:18:51.469215 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:51.469186    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed4e8e7a-2xvx7" event={"ID":"05abbb95-d56a-4abb-a43f-c043b2266219","Type":"ContainerDied","Data":"a3734f982dda07a6c922c2909a0bd2d0232447b0df4f7248a2335a7581bcff05"}
Oct 30 01:18:51.469321 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:51.469253    5002 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a3734f982dda07a6c922c2909a0bd2d0232447b0df4f7248a2335a7581bcff05"
Oct 30 01:18:51.469440 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:51.469339    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed4e8e7a-2xvx7"
Oct 30 01:18:51.515112 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:51.515083    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-5tcsp"
Oct 30 01:18:51.599024 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:51.598940    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62422ac3-d47e-40a0-8997-90a3fba2a424-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62422ac3-d47e-40a0-8997-90a3fba2a424-istio-data\") pod \"62422ac3-d47e-40a0-8997-90a3fba2a424\" (UID: \"62422ac3-d47e-40a0-8997-90a3fba2a424\") "
Oct 30 01:18:51.600659 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:51.599541    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62422ac3-d47e-40a0-8997-90a3fba2a424-istio-data" pod "62422ac3-d47e-40a0-8997-90a3fba2a424" (UID: "62422ac3-d47e-40a0-8997-90a3fba2a424"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:18:51.600804 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:51.600648    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62422ac3-d47e-40a0-8997-90a3fba2a424-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62422ac3-d47e-40a0-8997-90a3fba2a424-workload-certs\") pod \"62422ac3-d47e-40a0-8997-90a3fba2a424\" (UID: \"62422ac3-d47e-40a0-8997-90a3fba2a424\") "
Oct 30 01:18:51.600804 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:51.600760    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/62422ac3-d47e-40a0-8997-90a3fba2a424-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62422ac3-d47e-40a0-8997-90a3fba2a424-istio-podinfo\") pod \"62422ac3-d47e-40a0-8997-90a3fba2a424\" (UID: \"62422ac3-d47e-40a0-8997-90a3fba2a424\") "
Oct 30 01:18:51.600930 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:51.600898    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62422ac3-d47e-40a0-8997-90a3fba2a424-workload-certs" pod "62422ac3-d47e-40a0-8997-90a3fba2a424" (UID: "62422ac3-d47e-40a0-8997-90a3fba2a424"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:18:51.600930 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:51.600915    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62422ac3-d47e-40a0-8997-90a3fba2a424-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62422ac3-d47e-40a0-8997-90a3fba2a424-credential-socket\") pod \"62422ac3-d47e-40a0-8997-90a3fba2a424\" (UID: \"62422ac3-d47e-40a0-8997-90a3fba2a424\") "
Oct 30 01:18:51.601040 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:51.600962    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62422ac3-d47e-40a0-8997-90a3fba2a424-kube-api-access-n5nvh\" (UniqueName: \"kubernetes.io/projected/62422ac3-d47e-40a0-8997-90a3fba2a424-kube-api-access-n5nvh\") pod \"62422ac3-d47e-40a0-8997-90a3fba2a424\" (UID: \"62422ac3-d47e-40a0-8997-90a3fba2a424\") "
Oct 30 01:18:51.601103 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:51.601057    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62422ac3-d47e-40a0-8997-90a3fba2a424-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62422ac3-d47e-40a0-8997-90a3fba2a424-istiod-ca-cert\") pod \"62422ac3-d47e-40a0-8997-90a3fba2a424\" (UID: \"62422ac3-d47e-40a0-8997-90a3fba2a424\") "
Oct 30 01:18:51.601103 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:51.601089    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62422ac3-d47e-40a0-8997-90a3fba2a424-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62422ac3-d47e-40a0-8997-90a3fba2a424-workload-socket\") pod \"62422ac3-d47e-40a0-8997-90a3fba2a424\" (UID: \"62422ac3-d47e-40a0-8997-90a3fba2a424\") "
Oct 30 01:18:51.601208 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:51.601122    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62422ac3-d47e-40a0-8997-90a3fba2a424-istio-token\" (UniqueName: \"kubernetes.io/projected/62422ac3-d47e-40a0-8997-90a3fba2a424-istio-token\") pod \"62422ac3-d47e-40a0-8997-90a3fba2a424\" (UID: \"62422ac3-d47e-40a0-8997-90a3fba2a424\") "
Oct 30 01:18:51.601208 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:51.601172    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62422ac3-d47e-40a0-8997-90a3fba2a424-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62422ac3-d47e-40a0-8997-90a3fba2a424-istio-envoy\") pod \"62422ac3-d47e-40a0-8997-90a3fba2a424\" (UID: \"62422ac3-d47e-40a0-8997-90a3fba2a424\") "
Oct 30 01:18:51.601681 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:51.601657    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62422ac3-d47e-40a0-8997-90a3fba2a424-istiod-ca-cert" pod "62422ac3-d47e-40a0-8997-90a3fba2a424" (UID: "62422ac3-d47e-40a0-8997-90a3fba2a424"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:18:51.601813 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:51.601793    5002 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62422ac3-d47e-40a0-8997-90a3fba2a424-istio-data\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:18:51.602352 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:51.602332    5002 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62422ac3-d47e-40a0-8997-90a3fba2a424-workload-certs\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:18:51.603329 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:51.602051    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62422ac3-d47e-40a0-8997-90a3fba2a424-credential-socket" pod "62422ac3-d47e-40a0-8997-90a3fba2a424" (UID: "62422ac3-d47e-40a0-8997-90a3fba2a424"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:18:51.603537 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:51.602063    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62422ac3-d47e-40a0-8997-90a3fba2a424-workload-socket" pod "62422ac3-d47e-40a0-8997-90a3fba2a424" (UID: "62422ac3-d47e-40a0-8997-90a3fba2a424"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:18:51.603758 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:51.603735    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/62422ac3-d47e-40a0-8997-90a3fba2a424-istio-podinfo" pod "62422ac3-d47e-40a0-8997-90a3fba2a424" (UID: "62422ac3-d47e-40a0-8997-90a3fba2a424"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:18:51.604993 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:51.604969    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62422ac3-d47e-40a0-8997-90a3fba2a424-istio-envoy" pod "62422ac3-d47e-40a0-8997-90a3fba2a424" (UID: "62422ac3-d47e-40a0-8997-90a3fba2a424"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:18:51.605851 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:51.605816    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62422ac3-d47e-40a0-8997-90a3fba2a424-istio-token" pod "62422ac3-d47e-40a0-8997-90a3fba2a424" (UID: "62422ac3-d47e-40a0-8997-90a3fba2a424"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:18:51.606910 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:51.606885    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62422ac3-d47e-40a0-8997-90a3fba2a424-kube-api-access-n5nvh" pod "62422ac3-d47e-40a0-8997-90a3fba2a424" (UID: "62422ac3-d47e-40a0-8997-90a3fba2a424"). InnerVolumeSpecName "kube-api-access-n5nvh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:18:51.704216 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:51.704173    5002 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62422ac3-d47e-40a0-8997-90a3fba2a424-istio-podinfo\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:18:51.704216 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:51.704209    5002 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62422ac3-d47e-40a0-8997-90a3fba2a424-credential-socket\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:18:51.704216 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:51.704219    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n5nvh\" (UniqueName: \"kubernetes.io/projected/62422ac3-d47e-40a0-8997-90a3fba2a424-kube-api-access-n5nvh\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:18:51.704725 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:51.704230    5002 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62422ac3-d47e-40a0-8997-90a3fba2a424-istiod-ca-cert\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:18:51.704725 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:51.704240    5002 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62422ac3-d47e-40a0-8997-90a3fba2a424-workload-socket\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:18:51.704725 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:51.704247    5002 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62422ac3-d47e-40a0-8997-90a3fba2a424-istio-token\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:18:51.704725 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:51.704256    5002 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62422ac3-d47e-40a0-8997-90a3fba2a424-istio-envoy\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:18:52.510738 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:52.510704    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-5tcsp"
Oct 30 01:18:52.530571 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:52.530532    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-5tcsp" event={"ID":"62422ac3-d47e-40a0-8997-90a3fba2a424","Type":"ContainerDied","Data":"81c4fe4b7128a339eaca861bf79401abd0a20577405c2177e9ef05fdf14ff4ac"}
Oct 30 01:18:52.530897 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:52.530868    5002 scope.go:122] "RemoveContainer" containerID="8fc8dd904bcddbd6ed9a21b6d6a5543ab1cd58452f15e943d350738b78f81f16"
Oct 30 01:18:52.610029 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:52.609991    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-5tcsp"]
Oct 30 01:18:52.613456 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:52.613427    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-5tcsp"]
Oct 30 01:18:54.495975 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:18:54.495939    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62422ac3-d47e-40a0-8997-90a3fba2a424" path="/var/lib/kubelet/pods/62422ac3-d47e-40a0-8997-90a3fba2a424/volumes"
Oct 30 01:19:00.115196 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:00.115106    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-01215f-tq8mq"]
Oct 30 01:19:00.115967 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:00.115942    5002 cpu_manager.go:414] "containerMap: removing container" podUID="05abbb95-d56a-4abb-a43f-c043b2266219" containerName="revoke-keys"
Oct 30 01:19:00.115967 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:00.115966    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05abbb95-d56a-4abb-a43f-c043b2266219" containerName="revoke-keys"
Oct 30 01:19:00.116157 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:00.115998    5002 cpu_manager.go:414] "containerMap: removing container" podUID="62422ac3-d47e-40a0-8997-90a3fba2a424" containerName="istio-proxy"
Oct 30 01:19:00.116157 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:00.116006    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62422ac3-d47e-40a0-8997-90a3fba2a424" containerName="istio-proxy"
Oct 30 01:19:00.116277 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:00.116187    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="05abbb95-d56a-4abb-a43f-c043b2266219" containerName="revoke-keys"
Oct 30 01:19:00.116277 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:00.116206    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="62422ac3-d47e-40a0-8997-90a3fba2a424" containerName="istio-proxy"
Oct 30 01:19:00.121180 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:00.121159    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-01215f-tq8mq"
Oct 30 01:19:00.142856 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:00.142470    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-01215f-tq8mq"]
Oct 30 01:19:00.166401 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:00.154680    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:19:00.166401 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:00.154752    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:19:00.166401 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:00.154856    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5phsh\""
Oct 30 01:19:00.166401 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:00.154936    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:19:00.166401 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:00.156530    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhzjq\" (UniqueName: \"kubernetes.io/projected/e353f412-169a-4b5b-920f-eddbfb1af82e-kube-api-access-bhzjq\") pod \"maas-api-revoke-keys-e2e-ipp-a-01215f-tq8mq\" (UID: \"e353f412-169a-4b5b-920f-eddbfb1af82e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-01215f-tq8mq"
Oct 30 01:19:00.166401 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:00.156625    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e353f412-169a-4b5b-920f-eddbfb1af82e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-01215f-tq8mq\" (UID: \"e353f412-169a-4b5b-920f-eddbfb1af82e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-01215f-tq8mq"
Oct 30 01:19:00.166401 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:00.156770    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e353f412-169a-4b5b-920f-eddbfb1af82e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-01215f-tq8mq\" (UID: \"e353f412-169a-4b5b-920f-eddbfb1af82e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-01215f-tq8mq"
Oct 30 01:19:00.272353 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:00.272314    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e353f412-169a-4b5b-920f-eddbfb1af82e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-01215f-tq8mq\" (UID: \"e353f412-169a-4b5b-920f-eddbfb1af82e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-01215f-tq8mq"
Oct 30 01:19:00.282757 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:00.282720    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e353f412-169a-4b5b-920f-eddbfb1af82e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-01215f-tq8mq\" (UID: \"e353f412-169a-4b5b-920f-eddbfb1af82e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-01215f-tq8mq"
Oct 30 01:19:00.283153 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:00.283131    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhzjq\" (UniqueName: \"kubernetes.io/projected/e353f412-169a-4b5b-920f-eddbfb1af82e-kube-api-access-bhzjq\") pod \"maas-api-revoke-keys-e2e-ipp-a-01215f-tq8mq\" (UID: \"e353f412-169a-4b5b-920f-eddbfb1af82e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-01215f-tq8mq"
Oct 30 01:19:00.283674 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:00.283624    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e353f412-169a-4b5b-920f-eddbfb1af82e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-01215f-tq8mq\" (UID: \"e353f412-169a-4b5b-920f-eddbfb1af82e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-01215f-tq8mq"
Oct 30 01:19:00.286299 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:00.286274    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e353f412-169a-4b5b-920f-eddbfb1af82e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-01215f-tq8mq\" (UID: \"e353f412-169a-4b5b-920f-eddbfb1af82e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-01215f-tq8mq"
Oct 30 01:19:00.308251 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:00.308220    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhzjq\" (UniqueName: \"kubernetes.io/projected/e353f412-169a-4b5b-920f-eddbfb1af82e-kube-api-access-bhzjq\") pod \"maas-api-revoke-keys-e2e-ipp-a-01215f-tq8mq\" (UID: \"e353f412-169a-4b5b-920f-eddbfb1af82e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-01215f-tq8mq"
Oct 30 01:19:00.440621 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:00.440581    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-01215f-tq8mq"
Oct 30 01:19:00.761775 ip-10-0-128-7 kubenswrapper[5002]: W1030 01:19:00.761729    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode353f412_169a_4b5b_920f_eddbfb1af82e.slice/crio-ab70cdc2464bb8c6b8b593db6a34a67078b246d85957b59370bf49f43fdab28b WatchSource:0}: Error finding container ab70cdc2464bb8c6b8b593db6a34a67078b246d85957b59370bf49f43fdab28b: Status 404 returned error can't find the container with id ab70cdc2464bb8c6b8b593db6a34a67078b246d85957b59370bf49f43fdab28b
Oct 30 01:19:00.761918 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:00.761876    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-01215f-tq8mq"]
Oct 30 01:19:01.725048 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:01.725004    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-01215f-tq8mq" event={"ID":"e353f412-169a-4b5b-920f-eddbfb1af82e","Type":"ContainerStarted","Data":"d91c918077d8323d5ae7581e02ce29afc5f1d956547b9196d73aa6930b3369bf"}
Oct 30 01:19:01.725574 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:01.725108    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-01215f-tq8mq" event={"ID":"e353f412-169a-4b5b-920f-eddbfb1af82e","Type":"ContainerStarted","Data":"ab70cdc2464bb8c6b8b593db6a34a67078b246d85957b59370bf49f43fdab28b"}
Oct 30 01:19:01.772504 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:01.772433    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-01215f-tq8mq" podStartSLOduration=1.772412358 podStartE2EDuration="1.772412358s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:19:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:19:01.768569941 +0000 UTC m=+2837.385002821" watchObservedRunningTime="2026-10-30 01:19:01.772412358 +0000 UTC m=+2837.388845239"
Oct 30 01:19:02.737110 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:02.737060    5002 generic.go:356] "Generic (PLEG): container finished" podID="e353f412-169a-4b5b-920f-eddbfb1af82e" containerID="d91c918077d8323d5ae7581e02ce29afc5f1d956547b9196d73aa6930b3369bf" exitCode=0
Oct 30 01:19:02.737569 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:02.737150    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-01215f-tq8mq" event={"ID":"e353f412-169a-4b5b-920f-eddbfb1af82e","Type":"ContainerDied","Data":"d91c918077d8323d5ae7581e02ce29afc5f1d956547b9196d73aa6930b3369bf"}
Oct 30 01:19:03.030637 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:03.030549    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-172c4d2b-wkgr7"]
Oct 30 01:19:03.034079 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:03.034050    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-172c4d2b-wkgr7"]
Oct 30 01:19:03.388627 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:03.388524    5002 patch_prober.go:29] interesting pod/e2e-ipp-a-b73232-openshift-default-59c5464ccc-2kpc7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.1:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 01:19:03.388627 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:03.388598    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-b73232-openshift-default-59c5464ccc-2kpc7" podUID="8c2a594b-3bbf-4c94-8a9f-7b79c663f135" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.1:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:19:03.631293 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:03.624814    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-plcp4"]
Oct 30 01:19:03.631293 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:03.625087    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-plcp4" podUID="3b369691-a360-4ec3-ac10-b55af864f470" containerName="istio-proxy" containerID="cri-o://e8c759c36a821ad9cfc29a7edf354f13c5b03ea0d7ffebe4ace4d2ab16c0f3ed" gracePeriod=30
Oct 30 01:19:04.006141 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:04.006113    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-01215f-tq8mq"
Oct 30 01:19:04.176274 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:04.176172    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e353f412-169a-4b5b-920f-eddbfb1af82e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e353f412-169a-4b5b-920f-eddbfb1af82e-maas-api-openshift-ca\") pod \"e353f412-169a-4b5b-920f-eddbfb1af82e\" (UID: \"e353f412-169a-4b5b-920f-eddbfb1af82e\") "
Oct 30 01:19:04.176274 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:04.176261    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e353f412-169a-4b5b-920f-eddbfb1af82e-kube-api-access-bhzjq\" (UniqueName: \"kubernetes.io/projected/e353f412-169a-4b5b-920f-eddbfb1af82e-kube-api-access-bhzjq\") pod \"e353f412-169a-4b5b-920f-eddbfb1af82e\" (UID: \"e353f412-169a-4b5b-920f-eddbfb1af82e\") "
Oct 30 01:19:04.176675 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:04.176641    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e353f412-169a-4b5b-920f-eddbfb1af82e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e353f412-169a-4b5b-920f-eddbfb1af82e-maas-api-xks-ca\") pod \"e353f412-169a-4b5b-920f-eddbfb1af82e\" (UID: \"e353f412-169a-4b5b-920f-eddbfb1af82e\") "
Oct 30 01:19:04.176675 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:04.176670    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e353f412-169a-4b5b-920f-eddbfb1af82e-maas-api-openshift-ca" pod "e353f412-169a-4b5b-920f-eddbfb1af82e" (UID: "e353f412-169a-4b5b-920f-eddbfb1af82e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:19:04.177366 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:04.177341    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e353f412-169a-4b5b-920f-eddbfb1af82e-maas-api-openshift-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:19:04.179522 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:04.179180    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e353f412-169a-4b5b-920f-eddbfb1af82e-kube-api-access-bhzjq" pod "e353f412-169a-4b5b-920f-eddbfb1af82e" (UID: "e353f412-169a-4b5b-920f-eddbfb1af82e"). InnerVolumeSpecName "kube-api-access-bhzjq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:19:04.179522 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:04.179462    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e353f412-169a-4b5b-920f-eddbfb1af82e-maas-api-xks-ca" pod "e353f412-169a-4b5b-920f-eddbfb1af82e" (UID: "e353f412-169a-4b5b-920f-eddbfb1af82e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:19:04.278797 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:04.278747    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bhzjq\" (UniqueName: \"kubernetes.io/projected/e353f412-169a-4b5b-920f-eddbfb1af82e-kube-api-access-bhzjq\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:19:04.278797 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:04.278795    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e353f412-169a-4b5b-920f-eddbfb1af82e-maas-api-xks-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:19:04.507730 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:04.499022    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ad005b6-91dd-458c-9be3-7696b9ae2e15" path="/var/lib/kubelet/pods/0ad005b6-91dd-458c-9be3-7696b9ae2e15/volumes"
Oct 30 01:19:04.801807 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:04.801722    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-01215f-tq8mq"
Oct 30 01:19:04.801807 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:04.801757    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-01215f-tq8mq" event={"ID":"e353f412-169a-4b5b-920f-eddbfb1af82e","Type":"ContainerDied","Data":"ab70cdc2464bb8c6b8b593db6a34a67078b246d85957b59370bf49f43fdab28b"}
Oct 30 01:19:04.802053 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:04.801808    5002 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ab70cdc2464bb8c6b8b593db6a34a67078b246d85957b59370bf49f43fdab28b"
Oct 30 01:19:04.803662 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:04.803637    5002 generic.go:356] "Generic (PLEG): container finished" podID="8c2a594b-3bbf-4c94-8a9f-7b79c663f135" containerID="bbec892f4b363921edf12e2be788aa12e7a2b4eda9645e63b5ecb9fdadb3d557" exitCode=0
Oct 30 01:19:04.803806 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:04.803715    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-b73232-openshift-default-59c5464ccc-2kpc7" event={"ID":"8c2a594b-3bbf-4c94-8a9f-7b79c663f135","Type":"ContainerDied","Data":"bbec892f4b363921edf12e2be788aa12e7a2b4eda9645e63b5ecb9fdadb3d557"}
Oct 30 01:19:04.804372 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:04.804207    5002 scope.go:122] "RemoveContainer" containerID="bbec892f4b363921edf12e2be788aa12e7a2b4eda9645e63b5ecb9fdadb3d557"
Oct 30 01:19:04.805879 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:04.805854    5002 generic.go:356] "Generic (PLEG): container finished" podID="d1c17dcc-9b86-41c5-bea2-e284d0ec960c" containerID="69cd1b815a3940aefd4ac4b378500a4e582337393a76bd80c0d630e223325db5" exitCode=0
Oct 30 01:19:04.805993 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:04.805921    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-16c30f-openshift-default-dffb9ddc9-6jkdj" event={"ID":"d1c17dcc-9b86-41c5-bea2-e284d0ec960c","Type":"ContainerDied","Data":"69cd1b815a3940aefd4ac4b378500a4e582337393a76bd80c0d630e223325db5"}
Oct 30 01:19:04.806342 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:04.806326    5002 scope.go:122] "RemoveContainer" containerID="69cd1b815a3940aefd4ac4b378500a4e582337393a76bd80c0d630e223325db5"
Oct 30 01:19:04.806725 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:04.806692    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:19:04.806856 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:04.806806    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:19:04.806923 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:04.806863    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:19:04.808466 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:04.808430    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:19:04.808584 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:04.808507    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:19:04.808584 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:04.808550    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:19:05.821586 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:05.821549    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-b73232-openshift-default-59c5464ccc-2kpc7" event={"ID":"8c2a594b-3bbf-4c94-8a9f-7b79c663f135","Type":"ContainerStarted","Data":"24749d74a59c84ce731b6945d64b5d7397191f25a67a3b0836dfac7a2c54ff95"}
Oct 30 01:19:05.823671 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:05.823646    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-16c30f-openshift-default-dffb9ddc9-6jkdj" event={"ID":"d1c17dcc-9b86-41c5-bea2-e284d0ec960c","Type":"ContainerStarted","Data":"fc33b85c11faab6a91031b034174bee3ac3d14507e1bdec2e5015614cecef3f6"}
Oct 30 01:19:05.991402 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:05.976440    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-b73232-7f6ff6589f-nmvjx"]
Oct 30 01:19:06.082465 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:06.081810    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-b73232-openshift-default-59c5464ccc-2kpc7"
Oct 30 01:19:06.082465 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:06.081936    5002 patch_prober.go:29] interesting pod/e2e-ipp-a-b73232-openshift-default-59c5464ccc-2kpc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused" start-of-body=
Oct 30 01:19:06.082465 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:06.081994    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-b73232-openshift-default-59c5464ccc-2kpc7" podUID="8c2a594b-3bbf-4c94-8a9f-7b79c663f135" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused"
Oct 30 01:19:06.262406 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:06.261892    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-16c30f-openshift-default-dffb9ddc9-6jkdj"
Oct 30 01:19:06.262406 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:06.262022    5002 patch_prober.go:29] interesting pod/e2e-ipp-b-16c30f-openshift-default-dffb9ddc9-6jkdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.4:15021/healthz/ready\": dial tcp 10.132.1.4:15021: connect: connection refused" start-of-body=
Oct 30 01:19:06.262406 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:06.262077    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-16c30f-openshift-default-dffb9ddc9-6jkdj" podUID="d1c17dcc-9b86-41c5-bea2-e284d0ec960c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.4:15021/healthz/ready\": dial tcp 10.132.1.4:15021: connect: connection refused"
Oct 30 01:19:07.086406 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:07.072231    5002 patch_prober.go:29] interesting pod/e2e-ipp-a-b73232-openshift-default-59c5464ccc-2kpc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused" start-of-body=
Oct 30 01:19:07.086406 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:07.072307    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-b73232-openshift-default-59c5464ccc-2kpc7" podUID="8c2a594b-3bbf-4c94-8a9f-7b79c663f135" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused"
Oct 30 01:19:07.248666 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:07.248629    5002 patch_prober.go:29] interesting pod/e2e-ipp-b-16c30f-openshift-default-dffb9ddc9-6jkdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.4:15021/healthz/ready\": dial tcp 10.132.1.4:15021: connect: connection refused" start-of-body=
Oct 30 01:19:07.248878 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:07.248688    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-16c30f-openshift-default-dffb9ddc9-6jkdj" podUID="d1c17dcc-9b86-41c5-bea2-e284d0ec960c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.4:15021/healthz/ready\": dial tcp 10.132.1.4:15021: connect: connection refused"
Oct 30 01:19:08.067405 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:08.066797    5002 patch_prober.go:29] interesting pod/e2e-ipp-a-b73232-openshift-default-59c5464ccc-2kpc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused" start-of-body=
Oct 30 01:19:08.067405 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:08.066876    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-b73232-openshift-default-59c5464ccc-2kpc7" podUID="8c2a594b-3bbf-4c94-8a9f-7b79c663f135" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused"
Oct 30 01:19:08.248678 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:08.248643    5002 patch_prober.go:29] interesting pod/e2e-ipp-b-16c30f-openshift-default-dffb9ddc9-6jkdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.4:15021/healthz/ready\": dial tcp 10.132.1.4:15021: connect: connection refused" start-of-body=
Oct 30 01:19:08.249075 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:08.248702    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-16c30f-openshift-default-dffb9ddc9-6jkdj" podUID="d1c17dcc-9b86-41c5-bea2-e284d0ec960c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.4:15021/healthz/ready\": dial tcp 10.132.1.4:15021: connect: connection refused"
Oct 30 01:19:08.936797 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:08.936759    5002 generic.go:356] "Generic (PLEG): container finished" podID="3b369691-a360-4ec3-ac10-b55af864f470" containerID="e8c759c36a821ad9cfc29a7edf354f13c5b03ea0d7ffebe4ace4d2ab16c0f3ed" exitCode=0
Oct 30 01:19:08.937078 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:08.936960    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-plcp4" event={"ID":"3b369691-a360-4ec3-ac10-b55af864f470","Type":"ContainerDied","Data":"e8c759c36a821ad9cfc29a7edf354f13c5b03ea0d7ffebe4ace4d2ab16c0f3ed"}
Oct 30 01:19:09.071406 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:09.067865    5002 patch_prober.go:29] interesting pod/e2e-ipp-a-b73232-openshift-default-59c5464ccc-2kpc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused" start-of-body=
Oct 30 01:19:09.071406 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:09.067952    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-b73232-openshift-default-59c5464ccc-2kpc7" podUID="8c2a594b-3bbf-4c94-8a9f-7b79c663f135" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused"
Oct 30 01:19:09.211681 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:09.211496    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-plcp4"
Oct 30 01:19:09.240567 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:09.240532    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b369691-a360-4ec3-ac10-b55af864f470-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b369691-a360-4ec3-ac10-b55af864f470-credential-socket\") pod \"3b369691-a360-4ec3-ac10-b55af864f470\" (UID: \"3b369691-a360-4ec3-ac10-b55af864f470\") "
Oct 30 01:19:09.240761 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:09.240645    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b369691-a360-4ec3-ac10-b55af864f470-kube-api-access-wdh74\" (UniqueName: \"kubernetes.io/projected/3b369691-a360-4ec3-ac10-b55af864f470-kube-api-access-wdh74\") pod \"3b369691-a360-4ec3-ac10-b55af864f470\" (UID: \"3b369691-a360-4ec3-ac10-b55af864f470\") "
Oct 30 01:19:09.240761 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:09.240697    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b369691-a360-4ec3-ac10-b55af864f470-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b369691-a360-4ec3-ac10-b55af864f470-workload-certs\") pod \"3b369691-a360-4ec3-ac10-b55af864f470\" (UID: \"3b369691-a360-4ec3-ac10-b55af864f470\") "
Oct 30 01:19:09.240761 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:09.240717    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b369691-a360-4ec3-ac10-b55af864f470-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b369691-a360-4ec3-ac10-b55af864f470-istiod-ca-cert\") pod \"3b369691-a360-4ec3-ac10-b55af864f470\" (UID: \"3b369691-a360-4ec3-ac10-b55af864f470\") "
Oct 30 01:19:09.240945 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:09.240778    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b369691-a360-4ec3-ac10-b55af864f470-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b369691-a360-4ec3-ac10-b55af864f470-istio-data\") pod \"3b369691-a360-4ec3-ac10-b55af864f470\" (UID: \"3b369691-a360-4ec3-ac10-b55af864f470\") "
Oct 30 01:19:09.240945 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:09.240809    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b369691-a360-4ec3-ac10-b55af864f470-credential-socket" pod "3b369691-a360-4ec3-ac10-b55af864f470" (UID: "3b369691-a360-4ec3-ac10-b55af864f470"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:19:09.240945 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:09.240851    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b369691-a360-4ec3-ac10-b55af864f470-istio-token\" (UniqueName: \"kubernetes.io/projected/3b369691-a360-4ec3-ac10-b55af864f470-istio-token\") pod \"3b369691-a360-4ec3-ac10-b55af864f470\" (UID: \"3b369691-a360-4ec3-ac10-b55af864f470\") "
Oct 30 01:19:09.240945 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:09.240940    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b369691-a360-4ec3-ac10-b55af864f470-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b369691-a360-4ec3-ac10-b55af864f470-istio-envoy\") pod \"3b369691-a360-4ec3-ac10-b55af864f470\" (UID: \"3b369691-a360-4ec3-ac10-b55af864f470\") "
Oct 30 01:19:09.241157 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:09.240966    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b369691-a360-4ec3-ac10-b55af864f470-workload-certs" pod "3b369691-a360-4ec3-ac10-b55af864f470" (UID: "3b369691-a360-4ec3-ac10-b55af864f470"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:19:09.241157 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:09.241034    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b369691-a360-4ec3-ac10-b55af864f470-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b369691-a360-4ec3-ac10-b55af864f470-workload-socket\") pod \"3b369691-a360-4ec3-ac10-b55af864f470\" (UID: \"3b369691-a360-4ec3-ac10-b55af864f470\") "
Oct 30 01:19:09.241157 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:09.241072    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b369691-a360-4ec3-ac10-b55af864f470-istio-data" pod "3b369691-a360-4ec3-ac10-b55af864f470" (UID: "3b369691-a360-4ec3-ac10-b55af864f470"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:19:09.241157 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:09.241089    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3b369691-a360-4ec3-ac10-b55af864f470-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b369691-a360-4ec3-ac10-b55af864f470-istio-podinfo\") pod \"3b369691-a360-4ec3-ac10-b55af864f470\" (UID: \"3b369691-a360-4ec3-ac10-b55af864f470\") "
Oct 30 01:19:09.241321 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:09.241183    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b369691-a360-4ec3-ac10-b55af864f470-istiod-ca-cert" pod "3b369691-a360-4ec3-ac10-b55af864f470" (UID: "3b369691-a360-4ec3-ac10-b55af864f470"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:19:09.241321 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:09.241291    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b369691-a360-4ec3-ac10-b55af864f470-workload-socket" pod "3b369691-a360-4ec3-ac10-b55af864f470" (UID: "3b369691-a360-4ec3-ac10-b55af864f470"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:19:09.241790 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:09.241764    5002 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b369691-a360-4ec3-ac10-b55af864f470-istio-data\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:19:09.241904 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:09.241794    5002 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b369691-a360-4ec3-ac10-b55af864f470-workload-socket\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:19:09.241904 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:09.241812    5002 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b369691-a360-4ec3-ac10-b55af864f470-credential-socket\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:19:09.241904 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:09.241827    5002 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b369691-a360-4ec3-ac10-b55af864f470-workload-certs\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:19:09.241904 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:09.241841    5002 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b369691-a360-4ec3-ac10-b55af864f470-istiod-ca-cert\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:19:09.243338 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:09.243310    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b369691-a360-4ec3-ac10-b55af864f470-kube-api-access-wdh74" pod "3b369691-a360-4ec3-ac10-b55af864f470" (UID: "3b369691-a360-4ec3-ac10-b55af864f470"). InnerVolumeSpecName "kube-api-access-wdh74". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:19:09.243338 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:09.243324    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b369691-a360-4ec3-ac10-b55af864f470-istio-token" pod "3b369691-a360-4ec3-ac10-b55af864f470" (UID: "3b369691-a360-4ec3-ac10-b55af864f470"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:19:09.243516 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:09.243351    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3b369691-a360-4ec3-ac10-b55af864f470-istio-podinfo" pod "3b369691-a360-4ec3-ac10-b55af864f470" (UID: "3b369691-a360-4ec3-ac10-b55af864f470"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:19:09.243516 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:09.243419    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b369691-a360-4ec3-ac10-b55af864f470-istio-envoy" pod "3b369691-a360-4ec3-ac10-b55af864f470" (UID: "3b369691-a360-4ec3-ac10-b55af864f470"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:19:09.248777 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:09.248756    5002 patch_prober.go:29] interesting pod/e2e-ipp-b-16c30f-openshift-default-dffb9ddc9-6jkdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.4:15021/healthz/ready\": dial tcp 10.132.1.4:15021: connect: connection refused" start-of-body=
Oct 30 01:19:09.249077 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:09.248797    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-16c30f-openshift-default-dffb9ddc9-6jkdj" podUID="d1c17dcc-9b86-41c5-bea2-e284d0ec960c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.4:15021/healthz/ready\": dial tcp 10.132.1.4:15021: connect: connection refused"
Oct 30 01:19:09.342994 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:09.342955    5002 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b369691-a360-4ec3-ac10-b55af864f470-istio-token\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:19:09.342994 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:09.342988    5002 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b369691-a360-4ec3-ac10-b55af864f470-istio-envoy\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:19:09.342994 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:09.342998    5002 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b369691-a360-4ec3-ac10-b55af864f470-istio-podinfo\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:19:09.343956 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:09.343009    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wdh74\" (UniqueName: \"kubernetes.io/projected/3b369691-a360-4ec3-ac10-b55af864f470-kube-api-access-wdh74\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:19:09.947520 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:09.947487    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-plcp4"
Oct 30 01:19:09.947707 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:09.947491    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-plcp4" event={"ID":"3b369691-a360-4ec3-ac10-b55af864f470","Type":"ContainerDied","Data":"e6c052d4d4d1fceeead0ae63edb2066e42d5fe067b81d38974b66c56a7379dd6"}
Oct 30 01:19:09.947707 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:09.947619    5002 scope.go:122] "RemoveContainer" containerID="e8c759c36a821ad9cfc29a7edf354f13c5b03ea0d7ffebe4ace4d2ab16c0f3ed"
Oct 30 01:19:10.080768 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:10.080731    5002 patch_prober.go:29] interesting pod/e2e-ipp-a-b73232-openshift-default-59c5464ccc-2kpc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused" start-of-body=
Oct 30 01:19:10.081055 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:10.081023    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-b73232-openshift-default-59c5464ccc-2kpc7" podUID="8c2a594b-3bbf-4c94-8a9f-7b79c663f135" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused"
Oct 30 01:19:10.102953 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:10.102914    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-plcp4"]
Oct 30 01:19:10.117509 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:10.116762    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-plcp4"]
Oct 30 01:19:10.248515 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:10.248425    5002 patch_prober.go:29] interesting pod/e2e-ipp-b-16c30f-openshift-default-dffb9ddc9-6jkdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.4:15021/healthz/ready\": dial tcp 10.132.1.4:15021: connect: connection refused" start-of-body=
Oct 30 01:19:10.248515 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:10.248490    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-16c30f-openshift-default-dffb9ddc9-6jkdj" podUID="d1c17dcc-9b86-41c5-bea2-e284d0ec960c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.4:15021/healthz/ready\": dial tcp 10.132.1.4:15021: connect: connection refused"
Oct 30 01:19:10.510199 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:10.510112    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b369691-a360-4ec3-ac10-b55af864f470" path="/var/lib/kubelet/pods/3b369691-a360-4ec3-ac10-b55af864f470/volumes"
Oct 30 01:19:10.984247 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:10.984202    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b73232-7f6ff6589f-nmvjx" podUID="9359f1a2-ec9e-43de-a324-ab2aee55800d" containerName="payload-pre-processing" containerID="cri-o://5c321c052697792f422ee1cce04fb55f3c0b622ceb953c3a3a6796fa66ba90c1" gracePeriod=25
Oct 30 01:19:11.071425 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:11.069747    5002 patch_prober.go:29] interesting pod/e2e-ipp-a-b73232-openshift-default-59c5464ccc-2kpc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused" start-of-body=
Oct 30 01:19:11.071425 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:11.069810    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-b73232-openshift-default-59c5464ccc-2kpc7" podUID="8c2a594b-3bbf-4c94-8a9f-7b79c663f135" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused"
Oct 30 01:19:11.260454 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:11.249258    5002 patch_prober.go:29] interesting pod/e2e-ipp-b-16c30f-openshift-default-dffb9ddc9-6jkdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.4:15021/healthz/ready\": dial tcp 10.132.1.4:15021: connect: connection refused" start-of-body=
Oct 30 01:19:11.260454 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:11.249364    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-16c30f-openshift-default-dffb9ddc9-6jkdj" podUID="d1c17dcc-9b86-41c5-bea2-e284d0ec960c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.4:15021/healthz/ready\": dial tcp 10.132.1.4:15021: connect: connection refused"
Oct 30 01:19:11.453091 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:11.453062    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b73232-7f6ff6589f-nmvjx"
Oct 30 01:19:11.538488 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:11.530006    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9359f1a2-ec9e-43de-a324-ab2aee55800d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9359f1a2-ec9e-43de-a324-ab2aee55800d-plugins-config-volume\") pod \"9359f1a2-ec9e-43de-a324-ab2aee55800d\" (UID: \"9359f1a2-ec9e-43de-a324-ab2aee55800d\") "
Oct 30 01:19:11.538488 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:11.530112    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9359f1a2-ec9e-43de-a324-ab2aee55800d-kube-api-access-xgvvj\" (UniqueName: \"kubernetes.io/projected/9359f1a2-ec9e-43de-a324-ab2aee55800d-kube-api-access-xgvvj\") pod \"9359f1a2-ec9e-43de-a324-ab2aee55800d\" (UID: \"9359f1a2-ec9e-43de-a324-ab2aee55800d\") "
Oct 30 01:19:11.539491 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:11.539438    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9359f1a2-ec9e-43de-a324-ab2aee55800d-plugins-config-volume" pod "9359f1a2-ec9e-43de-a324-ab2aee55800d" (UID: "9359f1a2-ec9e-43de-a324-ab2aee55800d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:19:11.569228 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:11.567665    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9359f1a2-ec9e-43de-a324-ab2aee55800d-kube-api-access-xgvvj" pod "9359f1a2-ec9e-43de-a324-ab2aee55800d" (UID: "9359f1a2-ec9e-43de-a324-ab2aee55800d"). InnerVolumeSpecName "kube-api-access-xgvvj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:19:11.631953 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:11.631906    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xgvvj\" (UniqueName: \"kubernetes.io/projected/9359f1a2-ec9e-43de-a324-ab2aee55800d-kube-api-access-xgvvj\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:19:11.631953 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:11.631954    5002 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9359f1a2-ec9e-43de-a324-ab2aee55800d-plugins-config-volume\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:19:12.025673 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:12.025639    5002 generic.go:356] "Generic (PLEG): container finished" podID="9359f1a2-ec9e-43de-a324-ab2aee55800d" containerID="5c321c052697792f422ee1cce04fb55f3c0b622ceb953c3a3a6796fa66ba90c1" exitCode=0
Oct 30 01:19:12.025855 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:12.025694    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b73232-7f6ff6589f-nmvjx"
Oct 30 01:19:12.025855 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:12.025727    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b73232-7f6ff6589f-nmvjx" event={"ID":"9359f1a2-ec9e-43de-a324-ab2aee55800d","Type":"ContainerDied","Data":"5c321c052697792f422ee1cce04fb55f3c0b622ceb953c3a3a6796fa66ba90c1"}
Oct 30 01:19:12.025855 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:12.025785    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b73232-7f6ff6589f-nmvjx" event={"ID":"9359f1a2-ec9e-43de-a324-ab2aee55800d","Type":"ContainerDied","Data":"0d22960afd66ae44c388b19100ea160a9f05ae04342a259c0c691b2f90dd098d"}
Oct 30 01:19:12.025855 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:12.025814    5002 scope.go:122] "RemoveContainer" containerID="5c321c052697792f422ee1cce04fb55f3c0b622ceb953c3a3a6796fa66ba90c1"
Oct 30 01:19:12.037738 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:12.037716    5002 scope.go:122] "RemoveContainer" containerID="5c321c052697792f422ee1cce04fb55f3c0b622ceb953c3a3a6796fa66ba90c1"
Oct 30 01:19:12.038021 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:19:12.038000    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c321c052697792f422ee1cce04fb55f3c0b622ceb953c3a3a6796fa66ba90c1\": container with ID starting with 5c321c052697792f422ee1cce04fb55f3c0b622ceb953c3a3a6796fa66ba90c1 not found: ID does not exist" containerID="5c321c052697792f422ee1cce04fb55f3c0b622ceb953c3a3a6796fa66ba90c1"
Oct 30 01:19:12.038098 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:12.038034    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c321c052697792f422ee1cce04fb55f3c0b622ceb953c3a3a6796fa66ba90c1"} err="failed to get container status \"5c321c052697792f422ee1cce04fb55f3c0b622ceb953c3a3a6796fa66ba90c1\": rpc error: code = NotFound desc = could not find container \"5c321c052697792f422ee1cce04fb55f3c0b622ceb953c3a3a6796fa66ba90c1\": container with ID starting with 5c321c052697792f422ee1cce04fb55f3c0b622ceb953c3a3a6796fa66ba90c1 not found: ID does not exist"
Oct 30 01:19:12.071619 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:12.071589    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-b73232-openshift-default-59c5464ccc-2kpc7"
Oct 30 01:19:12.071912 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:12.071884    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-b73232-openshift-default-59c5464ccc-2kpc7"
Oct 30 01:19:12.072635 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:12.072613    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-b73232-openshift-default-59c5464ccc-2kpc7"
Oct 30 01:19:12.083795 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:12.083769    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-b73232-7f6ff6589f-nmvjx"]
Oct 30 01:19:12.098956 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:12.098930    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-b73232-7f6ff6589f-nmvjx"]
Oct 30 01:19:12.261185 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:12.261152    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-16c30f-openshift-default-dffb9ddc9-6jkdj"
Oct 30 01:19:12.261533 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:12.261515    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-16c30f-openshift-default-dffb9ddc9-6jkdj"
Oct 30 01:19:12.262364 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:12.262339    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-16c30f-openshift-default-dffb9ddc9-6jkdj"
Oct 30 01:19:12.495365 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:12.495328    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9359f1a2-ec9e-43de-a324-ab2aee55800d" path="/var/lib/kubelet/pods/9359f1a2-ec9e-43de-a324-ab2aee55800d/volumes"
Oct 30 01:19:12.862047 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:19:12.861962    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbjdq" containerName="extractor"
Oct 30 01:19:13.030873 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:13.030840    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24acd507-28chc"]
Oct 30 01:19:13.037842 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:13.037438    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24acd507-28chc"]
Oct 30 01:19:13.787700 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:13.787660    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-1e8c0e-openshift-default-6fb676f64f-ld62q"]
Oct 30 01:19:13.788883 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:13.788848    5002 cpu_manager.go:414] "containerMap: removing container" podUID="3b369691-a360-4ec3-ac10-b55af864f470" containerName="istio-proxy"
Oct 30 01:19:13.788883 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:13.788881    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b369691-a360-4ec3-ac10-b55af864f470" containerName="istio-proxy"
Oct 30 01:19:13.789100 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:13.788900    5002 cpu_manager.go:414] "containerMap: removing container" podUID="9359f1a2-ec9e-43de-a324-ab2aee55800d" containerName="payload-pre-processing"
Oct 30 01:19:13.789100 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:13.788909    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9359f1a2-ec9e-43de-a324-ab2aee55800d" containerName="payload-pre-processing"
Oct 30 01:19:13.789100 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:13.788954    5002 cpu_manager.go:414] "containerMap: removing container" podUID="e353f412-169a-4b5b-920f-eddbfb1af82e" containerName="revoke-keys"
Oct 30 01:19:13.789100 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:13.788962    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e353f412-169a-4b5b-920f-eddbfb1af82e" containerName="revoke-keys"
Oct 30 01:19:13.789323 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:13.789117    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="e353f412-169a-4b5b-920f-eddbfb1af82e" containerName="revoke-keys"
Oct 30 01:19:13.789323 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:13.789140    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b369691-a360-4ec3-ac10-b55af864f470" containerName="istio-proxy"
Oct 30 01:19:13.789323 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:13.789152    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="9359f1a2-ec9e-43de-a324-ab2aee55800d" containerName="payload-pre-processing"
Oct 30 01:19:13.792345 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:13.792322    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-1e8c0e-openshift-default-6fb676f64f-ld62q"
Oct 30 01:19:13.818058 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:13.818014    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-1e8c0e-openshift-default-6fb676f64f-ld62q"]
Oct 30 01:19:13.826960 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:13.824693    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-1e8c0e-openshift-default-dockercfg-w6hqm\""
Oct 30 01:19:13.866404 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:13.861369    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e09703cf-6ee3-45e9-893d-5a626ca636c9-credential-socket\") pod \"e2e-ipp-b-1e8c0e-openshift-default-6fb676f64f-ld62q\" (UID: \"e09703cf-6ee3-45e9-893d-5a626ca636c9\") " pod="openshift-ingress/e2e-ipp-b-1e8c0e-openshift-default-6fb676f64f-ld62q"
Oct 30 01:19:13.866404 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:13.861543    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e09703cf-6ee3-45e9-893d-5a626ca636c9-istio-podinfo\") pod \"e2e-ipp-b-1e8c0e-openshift-default-6fb676f64f-ld62q\" (UID: \"e09703cf-6ee3-45e9-893d-5a626ca636c9\") " pod="openshift-ingress/e2e-ipp-b-1e8c0e-openshift-default-6fb676f64f-ld62q"
Oct 30 01:19:13.866404 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:13.861622    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vz89v\" (UniqueName: \"kubernetes.io/projected/e09703cf-6ee3-45e9-893d-5a626ca636c9-kube-api-access-vz89v\") pod \"e2e-ipp-b-1e8c0e-openshift-default-6fb676f64f-ld62q\" (UID: \"e09703cf-6ee3-45e9-893d-5a626ca636c9\") " pod="openshift-ingress/e2e-ipp-b-1e8c0e-openshift-default-6fb676f64f-ld62q"
Oct 30 01:19:13.866404 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:13.861766    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e09703cf-6ee3-45e9-893d-5a626ca636c9-istio-data\") pod \"e2e-ipp-b-1e8c0e-openshift-default-6fb676f64f-ld62q\" (UID: \"e09703cf-6ee3-45e9-893d-5a626ca636c9\") " pod="openshift-ingress/e2e-ipp-b-1e8c0e-openshift-default-6fb676f64f-ld62q"
Oct 30 01:19:13.866404 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:13.861805    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e09703cf-6ee3-45e9-893d-5a626ca636c9-workload-socket\") pod \"e2e-ipp-b-1e8c0e-openshift-default-6fb676f64f-ld62q\" (UID: \"e09703cf-6ee3-45e9-893d-5a626ca636c9\") " pod="openshift-ingress/e2e-ipp-b-1e8c0e-openshift-default-6fb676f64f-ld62q"
Oct 30 01:19:13.866404 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:13.861989    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e09703cf-6ee3-45e9-893d-5a626ca636c9-istiod-ca-cert\") pod \"e2e-ipp-b-1e8c0e-openshift-default-6fb676f64f-ld62q\" (UID: \"e09703cf-6ee3-45e9-893d-5a626ca636c9\") " pod="openshift-ingress/e2e-ipp-b-1e8c0e-openshift-default-6fb676f64f-ld62q"
Oct 30 01:19:13.866404 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:13.862090    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e09703cf-6ee3-45e9-893d-5a626ca636c9-workload-certs\") pod \"e2e-ipp-b-1e8c0e-openshift-default-6fb676f64f-ld62q\" (UID: \"e09703cf-6ee3-45e9-893d-5a626ca636c9\") " pod="openshift-ingress/e2e-ipp-b-1e8c0e-openshift-default-6fb676f64f-ld62q"
Oct 30 01:19:13.866404 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:13.862134    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e09703cf-6ee3-45e9-893d-5a626ca636c9-istio-token\") pod \"e2e-ipp-b-1e8c0e-openshift-default-6fb676f64f-ld62q\" (UID: \"e09703cf-6ee3-45e9-893d-5a626ca636c9\") " pod="openshift-ingress/e2e-ipp-b-1e8c0e-openshift-default-6fb676f64f-ld62q"
Oct 30 01:19:13.866404 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:13.862429    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e09703cf-6ee3-45e9-893d-5a626ca636c9-istio-envoy\") pod \"e2e-ipp-b-1e8c0e-openshift-default-6fb676f64f-ld62q\" (UID: \"e09703cf-6ee3-45e9-893d-5a626ca636c9\") " pod="openshift-ingress/e2e-ipp-b-1e8c0e-openshift-default-6fb676f64f-ld62q"
Oct 30 01:19:13.975402 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:13.967687    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e09703cf-6ee3-45e9-893d-5a626ca636c9-workload-certs\") pod \"e2e-ipp-b-1e8c0e-openshift-default-6fb676f64f-ld62q\" (UID: \"e09703cf-6ee3-45e9-893d-5a626ca636c9\") " pod="openshift-ingress/e2e-ipp-b-1e8c0e-openshift-default-6fb676f64f-ld62q"
Oct 30 01:19:13.975402 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:13.967762    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e09703cf-6ee3-45e9-893d-5a626ca636c9-istio-token\") pod \"e2e-ipp-b-1e8c0e-openshift-default-6fb676f64f-ld62q\" (UID: \"e09703cf-6ee3-45e9-893d-5a626ca636c9\") " pod="openshift-ingress/e2e-ipp-b-1e8c0e-openshift-default-6fb676f64f-ld62q"
Oct 30 01:19:13.984408 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:13.983429    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e09703cf-6ee3-45e9-893d-5a626ca636c9-istio-envoy\") pod \"e2e-ipp-b-1e8c0e-openshift-default-6fb676f64f-ld62q\" (UID: \"e09703cf-6ee3-45e9-893d-5a626ca636c9\") " pod="openshift-ingress/e2e-ipp-b-1e8c0e-openshift-default-6fb676f64f-ld62q"
Oct 30 01:19:13.984408 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:13.983528    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e09703cf-6ee3-45e9-893d-5a626ca636c9-credential-socket\") pod \"e2e-ipp-b-1e8c0e-openshift-default-6fb676f64f-ld62q\" (UID: \"e09703cf-6ee3-45e9-893d-5a626ca636c9\") " pod="openshift-ingress/e2e-ipp-b-1e8c0e-openshift-default-6fb676f64f-ld62q"
Oct 30 01:19:13.984408 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:13.983623    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e09703cf-6ee3-45e9-893d-5a626ca636c9-istio-podinfo\") pod \"e2e-ipp-b-1e8c0e-openshift-default-6fb676f64f-ld62q\" (UID: \"e09703cf-6ee3-45e9-893d-5a626ca636c9\") " pod="openshift-ingress/e2e-ipp-b-1e8c0e-openshift-default-6fb676f64f-ld62q"
Oct 30 01:19:13.984408 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:13.983652    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vz89v\" (UniqueName: \"kubernetes.io/projected/e09703cf-6ee3-45e9-893d-5a626ca636c9-kube-api-access-vz89v\") pod \"e2e-ipp-b-1e8c0e-openshift-default-6fb676f64f-ld62q\" (UID: \"e09703cf-6ee3-45e9-893d-5a626ca636c9\") " pod="openshift-ingress/e2e-ipp-b-1e8c0e-openshift-default-6fb676f64f-ld62q"
Oct 30 01:19:13.984408 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:13.983751    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e09703cf-6ee3-45e9-893d-5a626ca636c9-istio-data\") pod \"e2e-ipp-b-1e8c0e-openshift-default-6fb676f64f-ld62q\" (UID: \"e09703cf-6ee3-45e9-893d-5a626ca636c9\") " pod="openshift-ingress/e2e-ipp-b-1e8c0e-openshift-default-6fb676f64f-ld62q"
Oct 30 01:19:13.984408 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:13.983782    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e09703cf-6ee3-45e9-893d-5a626ca636c9-workload-socket\") pod \"e2e-ipp-b-1e8c0e-openshift-default-6fb676f64f-ld62q\" (UID: \"e09703cf-6ee3-45e9-893d-5a626ca636c9\") " pod="openshift-ingress/e2e-ipp-b-1e8c0e-openshift-default-6fb676f64f-ld62q"
Oct 30 01:19:13.984408 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:13.983891    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e09703cf-6ee3-45e9-893d-5a626ca636c9-istiod-ca-cert\") pod \"e2e-ipp-b-1e8c0e-openshift-default-6fb676f64f-ld62q\" (UID: \"e09703cf-6ee3-45e9-893d-5a626ca636c9\") " pod="openshift-ingress/e2e-ipp-b-1e8c0e-openshift-default-6fb676f64f-ld62q"
Oct 30 01:19:13.997413 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:13.996130    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e09703cf-6ee3-45e9-893d-5a626ca636c9-istio-envoy\") pod \"e2e-ipp-b-1e8c0e-openshift-default-6fb676f64f-ld62q\" (UID: \"e09703cf-6ee3-45e9-893d-5a626ca636c9\") " pod="openshift-ingress/e2e-ipp-b-1e8c0e-openshift-default-6fb676f64f-ld62q"
Oct 30 01:19:13.997413 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:13.996489    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e09703cf-6ee3-45e9-893d-5a626ca636c9-workload-certs\") pod \"e2e-ipp-b-1e8c0e-openshift-default-6fb676f64f-ld62q\" (UID: \"e09703cf-6ee3-45e9-893d-5a626ca636c9\") " pod="openshift-ingress/e2e-ipp-b-1e8c0e-openshift-default-6fb676f64f-ld62q"
Oct 30 01:19:13.997413 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:13.997299    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e09703cf-6ee3-45e9-893d-5a626ca636c9-credential-socket\") pod \"e2e-ipp-b-1e8c0e-openshift-default-6fb676f64f-ld62q\" (UID: \"e09703cf-6ee3-45e9-893d-5a626ca636c9\") " pod="openshift-ingress/e2e-ipp-b-1e8c0e-openshift-default-6fb676f64f-ld62q"
Oct 30 01:19:14.002018 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:13.999570    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e09703cf-6ee3-45e9-893d-5a626ca636c9-workload-socket\") pod \"e2e-ipp-b-1e8c0e-openshift-default-6fb676f64f-ld62q\" (UID: \"e09703cf-6ee3-45e9-893d-5a626ca636c9\") " pod="openshift-ingress/e2e-ipp-b-1e8c0e-openshift-default-6fb676f64f-ld62q"
Oct 30 01:19:14.005400 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:14.004321    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e09703cf-6ee3-45e9-893d-5a626ca636c9-istio-podinfo\") pod \"e2e-ipp-b-1e8c0e-openshift-default-6fb676f64f-ld62q\" (UID: \"e09703cf-6ee3-45e9-893d-5a626ca636c9\") " pod="openshift-ingress/e2e-ipp-b-1e8c0e-openshift-default-6fb676f64f-ld62q"
Oct 30 01:19:14.011006 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:14.009706    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e09703cf-6ee3-45e9-893d-5a626ca636c9-istio-data\") pod \"e2e-ipp-b-1e8c0e-openshift-default-6fb676f64f-ld62q\" (UID: \"e09703cf-6ee3-45e9-893d-5a626ca636c9\") " pod="openshift-ingress/e2e-ipp-b-1e8c0e-openshift-default-6fb676f64f-ld62q"
Oct 30 01:19:14.011006 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:14.010431    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e09703cf-6ee3-45e9-893d-5a626ca636c9-istiod-ca-cert\") pod \"e2e-ipp-b-1e8c0e-openshift-default-6fb676f64f-ld62q\" (UID: \"e09703cf-6ee3-45e9-893d-5a626ca636c9\") " pod="openshift-ingress/e2e-ipp-b-1e8c0e-openshift-default-6fb676f64f-ld62q"
Oct 30 01:19:14.047526 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:14.036213    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vz89v\" (UniqueName: \"kubernetes.io/projected/e09703cf-6ee3-45e9-893d-5a626ca636c9-kube-api-access-vz89v\") pod \"e2e-ipp-b-1e8c0e-openshift-default-6fb676f64f-ld62q\" (UID: \"e09703cf-6ee3-45e9-893d-5a626ca636c9\") " pod="openshift-ingress/e2e-ipp-b-1e8c0e-openshift-default-6fb676f64f-ld62q"
Oct 30 01:19:14.057434 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:14.050262    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e09703cf-6ee3-45e9-893d-5a626ca636c9-istio-token\") pod \"e2e-ipp-b-1e8c0e-openshift-default-6fb676f64f-ld62q\" (UID: \"e09703cf-6ee3-45e9-893d-5a626ca636c9\") " pod="openshift-ingress/e2e-ipp-b-1e8c0e-openshift-default-6fb676f64f-ld62q"
Oct 30 01:19:14.115413 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:14.112660    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-1e8c0e-openshift-default-6fb676f64f-ld62q"
Oct 30 01:19:14.497431 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:14.497362    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46dd358d-4793-4fa7-ad99-06c2e2147c0f" path="/var/lib/kubelet/pods/46dd358d-4793-4fa7-ad99-06c2e2147c0f/volumes"
Oct 30 01:19:14.516125 ip-10-0-128-7 kubenswrapper[5002]: W1030 01:19:14.516087    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode09703cf_6ee3_45e9_893d_5a626ca636c9.slice/crio-d018ff30bfa75b77c6559a8d2e3b8554b923eb11f396a978ffd9f6d9662bb134 WatchSource:0}: Error finding container d018ff30bfa75b77c6559a8d2e3b8554b923eb11f396a978ffd9f6d9662bb134: Status 404 returned error can't find the container with id d018ff30bfa75b77c6559a8d2e3b8554b923eb11f396a978ffd9f6d9662bb134
Oct 30 01:19:14.519414 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:14.519361    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-1e8c0e-openshift-default-6fb676f64f-ld62q"]
Oct 30 01:19:14.519529 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:14.519358    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:19:14.519529 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:14.519508    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:19:14.519627 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:14.519560    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:19:15.090465 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:15.090426    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-1e8c0e-openshift-default-6fb676f64f-ld62q" event={"ID":"e09703cf-6ee3-45e9-893d-5a626ca636c9","Type":"ContainerStarted","Data":"4ffab9d9a6fd83e8b2639aaf218cbf94607909f3ff720ba0d2aa6083b0d37bcd"}
Oct 30 01:19:15.090849 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:15.090489    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-1e8c0e-openshift-default-6fb676f64f-ld62q" event={"ID":"e09703cf-6ee3-45e9-893d-5a626ca636c9","Type":"ContainerStarted","Data":"d018ff30bfa75b77c6559a8d2e3b8554b923eb11f396a978ffd9f6d9662bb134"}
Oct 30 01:19:15.115776 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:15.115741    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-1e8c0e-openshift-default-6fb676f64f-ld62q"
Oct 30 01:19:15.122846 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:15.122805    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-1e8c0e-openshift-default-6fb676f64f-ld62q"
Oct 30 01:19:15.153630 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:15.153571    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-1e8c0e-openshift-default-6fb676f64f-ld62q" podStartSLOduration=2.153553107 podStartE2EDuration="2.153553107s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:19:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:19:15.152913311 +0000 UTC m=+2850.769346192" watchObservedRunningTime="2026-10-30 01:19:15.153553107 +0000 UTC m=+2850.769986055"
Oct 30 01:19:16.098972 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:16.098942    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-1e8c0e-openshift-default-6fb676f64f-ld62q"
Oct 30 01:19:16.100219 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:16.100197    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-1e8c0e-openshift-default-6fb676f64f-ld62q"
Oct 30 01:19:19.614487 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:19.614448    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-1e8c0e-857cdc4678-hxkmw"]
Oct 30 01:19:19.619013 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:19.618994    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-1e8c0e-857cdc4678-hxkmw"
Oct 30 01:19:19.636521 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:19.636479    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-1e8c0e\""
Oct 30 01:19:19.641139 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:19.641113    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-1e8c0e-857cdc4678-hxkmw"]
Oct 30 01:19:19.650462 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:19.650438    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-1e8c0e-dockercfg-k7qm8\""
Oct 30 01:19:19.702893 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:19.702854    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7178b0c4-ce99-4aa0-8c35-bddc5936b53f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-1e8c0e-857cdc4678-hxkmw\" (UID: \"7178b0c4-ce99-4aa0-8c35-bddc5936b53f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-1e8c0e-857cdc4678-hxkmw"
Oct 30 01:19:19.703136 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:19.703102    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrfvg\" (UniqueName: \"kubernetes.io/projected/7178b0c4-ce99-4aa0-8c35-bddc5936b53f-kube-api-access-jrfvg\") pod \"payload-processing-e2e-ipp-b-1e8c0e-857cdc4678-hxkmw\" (UID: \"7178b0c4-ce99-4aa0-8c35-bddc5936b53f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-1e8c0e-857cdc4678-hxkmw"
Oct 30 01:19:19.811294 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:19.811249    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrfvg\" (UniqueName: \"kubernetes.io/projected/7178b0c4-ce99-4aa0-8c35-bddc5936b53f-kube-api-access-jrfvg\") pod \"payload-processing-e2e-ipp-b-1e8c0e-857cdc4678-hxkmw\" (UID: \"7178b0c4-ce99-4aa0-8c35-bddc5936b53f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-1e8c0e-857cdc4678-hxkmw"
Oct 30 01:19:19.834491 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:19.820500    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7178b0c4-ce99-4aa0-8c35-bddc5936b53f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-1e8c0e-857cdc4678-hxkmw\" (UID: \"7178b0c4-ce99-4aa0-8c35-bddc5936b53f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-1e8c0e-857cdc4678-hxkmw"
Oct 30 01:19:19.842651 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:19.837769    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7178b0c4-ce99-4aa0-8c35-bddc5936b53f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-1e8c0e-857cdc4678-hxkmw\" (UID: \"7178b0c4-ce99-4aa0-8c35-bddc5936b53f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-1e8c0e-857cdc4678-hxkmw"
Oct 30 01:19:19.862239 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:19.862206    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrfvg\" (UniqueName: \"kubernetes.io/projected/7178b0c4-ce99-4aa0-8c35-bddc5936b53f-kube-api-access-jrfvg\") pod \"payload-processing-e2e-ipp-b-1e8c0e-857cdc4678-hxkmw\" (UID: \"7178b0c4-ce99-4aa0-8c35-bddc5936b53f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-1e8c0e-857cdc4678-hxkmw"
Oct 30 01:19:19.936411 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:19.935033    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-1e8c0e-857cdc4678-hxkmw"
Oct 30 01:19:20.281654 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:20.281623    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-1e8c0e-857cdc4678-hxkmw"]
Oct 30 01:19:20.281819 ip-10-0-128-7 kubenswrapper[5002]: W1030 01:19:20.281741    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7178b0c4_ce99_4aa0_8c35_bddc5936b53f.slice/crio-15bea8aa7dc6f8b0a69973aaf07918ea8af2a3a76d5ab76bb7fade0b8e7dda81 WatchSource:0}: Error finding container 15bea8aa7dc6f8b0a69973aaf07918ea8af2a3a76d5ab76bb7fade0b8e7dda81: Status 404 returned error can't find the container with id 15bea8aa7dc6f8b0a69973aaf07918ea8af2a3a76d5ab76bb7fade0b8e7dda81
Oct 30 01:19:21.139198 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:21.139156    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-1e8c0e-857cdc4678-hxkmw" event={"ID":"7178b0c4-ce99-4aa0-8c35-bddc5936b53f","Type":"ContainerStarted","Data":"e768837311e3c1b3e946fa09062d99042cf865c3da507190f2c95bd2387e657e"}
Oct 30 01:19:21.139673 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:21.139227    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-1e8c0e-857cdc4678-hxkmw"
Oct 30 01:19:21.139673 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:21.139249    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-1e8c0e-857cdc4678-hxkmw" event={"ID":"7178b0c4-ce99-4aa0-8c35-bddc5936b53f","Type":"ContainerStarted","Data":"15bea8aa7dc6f8b0a69973aaf07918ea8af2a3a76d5ab76bb7fade0b8e7dda81"}
Oct 30 01:19:21.182708 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:21.182637    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-1e8c0e-857cdc4678-hxkmw" podStartSLOduration=2.182618537 podStartE2EDuration="2.182618537s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:19:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:19:21.182320616 +0000 UTC m=+2856.798753496" watchObservedRunningTime="2026-10-30 01:19:21.182618537 +0000 UTC m=+2856.799051417"
Oct 30 01:19:23.829893 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:23.829860    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7d8z4"]
Oct 30 01:19:26.033414 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:26.033357    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-635e17f5-v97r9"]
Oct 30 01:19:26.046815 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:26.046780    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-635e17f5-v97r9"]
Oct 30 01:19:26.495560 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:26.495527    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22278f91-d2c8-4808-a572-17290bda0150" path="/var/lib/kubelet/pods/22278f91-d2c8-4808-a572-17290bda0150/volumes"
Oct 30 01:19:32.150209 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:32.150176    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-1e8c0e-857cdc4678-hxkmw"
Oct 30 01:19:36.031303 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:36.031263    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b3ba534-lzjvc"]
Oct 30 01:19:36.045143 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:36.045108    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b3ba534-lzjvc"]
Oct 30 01:19:36.495423 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:36.495367    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="477a7d12-e9bf-4fe4-877b-03f469167592" path="/var/lib/kubelet/pods/477a7d12-e9bf-4fe4-877b-03f469167592/volumes"
Oct 30 01:19:37.760957 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:37.760919    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a7ca1f-kvl7v"]
Oct 30 01:19:37.775346 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:37.775304    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a7ca1f-kvl7v"]
Oct 30 01:19:38.506490 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:38.506456    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="613b69b3-822c-4468-b9bb-18b34cca8ba5" path="/var/lib/kubelet/pods/613b69b3-822c-4468-b9bb-18b34cca8ba5/volumes"
Oct 30 01:19:42.863497 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:19:42.863452    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbjdq" containerName="extractor"
Oct 30 01:19:44.324613 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:44.319480    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-01215f-tq8mq"]
Oct 30 01:19:44.328865 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:44.328824    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-01215f-tq8mq"]
Oct 30 01:19:44.495922 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:44.495887    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e353f412-169a-4b5b-920f-eddbfb1af82e" path="/var/lib/kubelet/pods/e353f412-169a-4b5b-920f-eddbfb1af82e/volumes"
Oct 30 01:19:49.258544 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:49.258438    5002 scope.go:122] "RemoveContainer" containerID="fdbfd27f271bb4bb7f162253facf1a62f4aa110d21f1edef991f48d517e602e1"
Oct 30 01:19:49.271363 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:49.271325    5002 scope.go:122] "RemoveContainer" containerID="9b58c57b966194ffbd9e312be31029a8102154ea337b61d8a2935a137b09b0cb"
Oct 30 01:19:49.281740 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:49.281715    5002 scope.go:122] "RemoveContainer" containerID="0c6df505545d38121ca68480bdea497c91715e6386d580ab1b30103e32e654c8"
Oct 30 01:19:51.102827 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:51.102787    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-01215f-openshift-default-95bfd666f-rkm78"]
Oct 30 01:19:51.103666 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:51.103607    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-01215f-openshift-default-95bfd666f-rkm78" podUID="4c140dcb-40bd-4cd2-ac16-df81ed774c16" containerName="istio-proxy" containerID="cri-o://45fe47611a2d211358b789fd525622a10e810b445708603137038cbe3b5ccc34" gracePeriod=30
Oct 30 01:19:51.694306 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:51.694257    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-9a8001-hjs8l"]
Oct 30 01:19:51.701224 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:51.701200    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-9a8001-hjs8l"
Oct 30 01:19:51.717875 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:51.717844    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:19:51.718033 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:51.717998    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-9a8001-hjs8l"]
Oct 30 01:19:51.725476 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:51.725446    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:19:51.763586 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:51.763546    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bn8kr\" (UniqueName: \"kubernetes.io/projected/b55fafdf-43af-45d6-a7bb-fb38659c3fe4-kube-api-access-bn8kr\") pod \"maas-api-revoke-keys-e2e-ipp-b-9a8001-hjs8l\" (UID: \"b55fafdf-43af-45d6-a7bb-fb38659c3fe4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-9a8001-hjs8l"
Oct 30 01:19:51.763776 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:51.763615    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b55fafdf-43af-45d6-a7bb-fb38659c3fe4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-9a8001-hjs8l\" (UID: \"b55fafdf-43af-45d6-a7bb-fb38659c3fe4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-9a8001-hjs8l"
Oct 30 01:19:51.763879 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:51.763845    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b55fafdf-43af-45d6-a7bb-fb38659c3fe4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-9a8001-hjs8l\" (UID: \"b55fafdf-43af-45d6-a7bb-fb38659c3fe4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-9a8001-hjs8l"
Oct 30 01:19:51.765576 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:51.765552    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5phsh\""
Oct 30 01:19:51.765745 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:51.765638    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:19:51.865185 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:51.865146    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b55fafdf-43af-45d6-a7bb-fb38659c3fe4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-9a8001-hjs8l\" (UID: \"b55fafdf-43af-45d6-a7bb-fb38659c3fe4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-9a8001-hjs8l"
Oct 30 01:19:51.865369 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:51.865284    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bn8kr\" (UniqueName: \"kubernetes.io/projected/b55fafdf-43af-45d6-a7bb-fb38659c3fe4-kube-api-access-bn8kr\") pod \"maas-api-revoke-keys-e2e-ipp-b-9a8001-hjs8l\" (UID: \"b55fafdf-43af-45d6-a7bb-fb38659c3fe4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-9a8001-hjs8l"
Oct 30 01:19:51.865369 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:51.865337    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b55fafdf-43af-45d6-a7bb-fb38659c3fe4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-9a8001-hjs8l\" (UID: \"b55fafdf-43af-45d6-a7bb-fb38659c3fe4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-9a8001-hjs8l"
Oct 30 01:19:51.866200 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:51.866169    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b55fafdf-43af-45d6-a7bb-fb38659c3fe4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-9a8001-hjs8l\" (UID: \"b55fafdf-43af-45d6-a7bb-fb38659c3fe4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-9a8001-hjs8l"
Oct 30 01:19:51.867749 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:51.867717    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b55fafdf-43af-45d6-a7bb-fb38659c3fe4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-9a8001-hjs8l\" (UID: \"b55fafdf-43af-45d6-a7bb-fb38659c3fe4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-9a8001-hjs8l"
Oct 30 01:19:51.890615 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:51.890579    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bn8kr\" (UniqueName: \"kubernetes.io/projected/b55fafdf-43af-45d6-a7bb-fb38659c3fe4-kube-api-access-bn8kr\") pod \"maas-api-revoke-keys-e2e-ipp-b-9a8001-hjs8l\" (UID: \"b55fafdf-43af-45d6-a7bb-fb38659c3fe4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-9a8001-hjs8l"
Oct 30 01:19:52.017210 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:52.017118    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-9a8001-hjs8l"
Oct 30 01:19:52.326334 ip-10-0-128-7 kubenswrapper[5002]: W1030 01:19:52.326301    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb55fafdf_43af_45d6_a7bb_fb38659c3fe4.slice/crio-b5427bc659374132254edac182e95d138eb7ab792f5d9332f40343625b4c661a WatchSource:0}: Error finding container b5427bc659374132254edac182e95d138eb7ab792f5d9332f40343625b4c661a: Status 404 returned error can't find the container with id b5427bc659374132254edac182e95d138eb7ab792f5d9332f40343625b4c661a
Oct 30 01:19:52.327762 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:52.327736    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-9a8001-hjs8l"]
Oct 30 01:19:52.401965 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:52.401924    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-x49mt"]
Oct 30 01:19:52.402180 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:52.402134    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-x49mt" podUID="19abf60c-9f0e-47c8-9679-970496e6772c" containerName="istio-proxy" containerID="cri-o://15f26cf4361f3eea12c3580685bc363574cce9f0653ff0122924c0552072b762" gracePeriod=30
Oct 30 01:19:52.482603 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:52.482556    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-9a8001-hjs8l" event={"ID":"b55fafdf-43af-45d6-a7bb-fb38659c3fe4","Type":"ContainerStarted","Data":"b5427bc659374132254edac182e95d138eb7ab792f5d9332f40343625b4c661a"}
Oct 30 01:19:53.492948 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:53.492913    5002 generic.go:356] "Generic (PLEG): container finished" podID="b55fafdf-43af-45d6-a7bb-fb38659c3fe4" containerID="86ae70df480ea8c844011ee7d41692d8c9cef62ee847dfdae0e26114d8a4a995" exitCode=0
Oct 30 01:19:53.493478 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:53.492990    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-9a8001-hjs8l" event={"ID":"b55fafdf-43af-45d6-a7bb-fb38659c3fe4","Type":"ContainerDied","Data":"86ae70df480ea8c844011ee7d41692d8c9cef62ee847dfdae0e26114d8a4a995"}
Oct 30 01:19:54.677363 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:54.677324    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-9a8001-hjs8l"
Oct 30 01:19:54.840299 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:54.840203    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b55fafdf-43af-45d6-a7bb-fb38659c3fe4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b55fafdf-43af-45d6-a7bb-fb38659c3fe4-maas-api-openshift-ca\") pod \"b55fafdf-43af-45d6-a7bb-fb38659c3fe4\" (UID: \"b55fafdf-43af-45d6-a7bb-fb38659c3fe4\") "
Oct 30 01:19:54.840537 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:54.840325    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b55fafdf-43af-45d6-a7bb-fb38659c3fe4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b55fafdf-43af-45d6-a7bb-fb38659c3fe4-maas-api-xks-ca\") pod \"b55fafdf-43af-45d6-a7bb-fb38659c3fe4\" (UID: \"b55fafdf-43af-45d6-a7bb-fb38659c3fe4\") "
Oct 30 01:19:54.840537 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:54.840458    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b55fafdf-43af-45d6-a7bb-fb38659c3fe4-kube-api-access-bn8kr\" (UniqueName: \"kubernetes.io/projected/b55fafdf-43af-45d6-a7bb-fb38659c3fe4-kube-api-access-bn8kr\") pod \"b55fafdf-43af-45d6-a7bb-fb38659c3fe4\" (UID: \"b55fafdf-43af-45d6-a7bb-fb38659c3fe4\") "
Oct 30 01:19:54.840668 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:54.840625    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b55fafdf-43af-45d6-a7bb-fb38659c3fe4-maas-api-openshift-ca" pod "b55fafdf-43af-45d6-a7bb-fb38659c3fe4" (UID: "b55fafdf-43af-45d6-a7bb-fb38659c3fe4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:19:54.841087 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:54.841060    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b55fafdf-43af-45d6-a7bb-fb38659c3fe4-maas-api-openshift-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:19:54.843298 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:54.843265    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b55fafdf-43af-45d6-a7bb-fb38659c3fe4-maas-api-xks-ca" pod "b55fafdf-43af-45d6-a7bb-fb38659c3fe4" (UID: "b55fafdf-43af-45d6-a7bb-fb38659c3fe4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:19:54.844735 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:54.844709    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b55fafdf-43af-45d6-a7bb-fb38659c3fe4-kube-api-access-bn8kr" pod "b55fafdf-43af-45d6-a7bb-fb38659c3fe4" (UID: "b55fafdf-43af-45d6-a7bb-fb38659c3fe4"). InnerVolumeSpecName "kube-api-access-bn8kr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:19:54.942245 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:54.942208    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b55fafdf-43af-45d6-a7bb-fb38659c3fe4-maas-api-xks-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:19:54.942245 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:54.942244    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bn8kr\" (UniqueName: \"kubernetes.io/projected/b55fafdf-43af-45d6-a7bb-fb38659c3fe4-kube-api-access-bn8kr\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:19:55.511590 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:55.511558    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-9a8001-hjs8l"
Oct 30 01:19:55.511775 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:55.511591    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-9a8001-hjs8l" event={"ID":"b55fafdf-43af-45d6-a7bb-fb38659c3fe4","Type":"ContainerDied","Data":"b5427bc659374132254edac182e95d138eb7ab792f5d9332f40343625b4c661a"}
Oct 30 01:19:55.511775 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:55.511640    5002 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b5427bc659374132254edac182e95d138eb7ab792f5d9332f40343625b4c661a"
Oct 30 01:19:56.403508 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:56.403473    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-01215f-openshift-default-95bfd666f-rkm78"
Oct 30 01:19:56.521507 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:56.521466    5002 generic.go:356] "Generic (PLEG): container finished" podID="4c140dcb-40bd-4cd2-ac16-df81ed774c16" containerID="45fe47611a2d211358b789fd525622a10e810b445708603137038cbe3b5ccc34" exitCode=0
Oct 30 01:19:56.521695 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:56.521513    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-01215f-openshift-default-95bfd666f-rkm78"
Oct 30 01:19:56.521756 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:56.521714    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-01215f-openshift-default-95bfd666f-rkm78" event={"ID":"4c140dcb-40bd-4cd2-ac16-df81ed774c16","Type":"ContainerDied","Data":"45fe47611a2d211358b789fd525622a10e810b445708603137038cbe3b5ccc34"}
Oct 30 01:19:56.521814 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:56.521762    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-01215f-openshift-default-95bfd666f-rkm78" event={"ID":"4c140dcb-40bd-4cd2-ac16-df81ed774c16","Type":"ContainerDied","Data":"c2be7eab5e8eb8158fe1f0994652da997ce772600055919c694ddbe01d8404e6"}
Oct 30 01:19:56.521814 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:56.521793    5002 scope.go:122] "RemoveContainer" containerID="45fe47611a2d211358b789fd525622a10e810b445708603137038cbe3b5ccc34"
Oct 30 01:19:56.535425 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:56.535363    5002 scope.go:122] "RemoveContainer" containerID="d8efe79ab9d9d2cee14f023376facf07f6bf3d680db7bd4489893d8432f8bc3d"
Oct 30 01:19:56.548129 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:56.548104    5002 scope.go:122] "RemoveContainer" containerID="45fe47611a2d211358b789fd525622a10e810b445708603137038cbe3b5ccc34"
Oct 30 01:19:56.548466 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:19:56.548434    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45fe47611a2d211358b789fd525622a10e810b445708603137038cbe3b5ccc34\": container with ID starting with 45fe47611a2d211358b789fd525622a10e810b445708603137038cbe3b5ccc34 not found: ID does not exist" containerID="45fe47611a2d211358b789fd525622a10e810b445708603137038cbe3b5ccc34"
Oct 30 01:19:56.548565 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:56.548476    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45fe47611a2d211358b789fd525622a10e810b445708603137038cbe3b5ccc34"} err="failed to get container status \"45fe47611a2d211358b789fd525622a10e810b445708603137038cbe3b5ccc34\": rpc error: code = NotFound desc = could not find container \"45fe47611a2d211358b789fd525622a10e810b445708603137038cbe3b5ccc34\": container with ID starting with 45fe47611a2d211358b789fd525622a10e810b445708603137038cbe3b5ccc34 not found: ID does not exist"
Oct 30 01:19:56.548565 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:56.548494    5002 scope.go:122] "RemoveContainer" containerID="d8efe79ab9d9d2cee14f023376facf07f6bf3d680db7bd4489893d8432f8bc3d"
Oct 30 01:19:56.548801 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:19:56.548777    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8efe79ab9d9d2cee14f023376facf07f6bf3d680db7bd4489893d8432f8bc3d\": container with ID starting with d8efe79ab9d9d2cee14f023376facf07f6bf3d680db7bd4489893d8432f8bc3d not found: ID does not exist" containerID="d8efe79ab9d9d2cee14f023376facf07f6bf3d680db7bd4489893d8432f8bc3d"
Oct 30 01:19:56.548853 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:56.548809    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8efe79ab9d9d2cee14f023376facf07f6bf3d680db7bd4489893d8432f8bc3d"} err="failed to get container status \"d8efe79ab9d9d2cee14f023376facf07f6bf3d680db7bd4489893d8432f8bc3d\": rpc error: code = NotFound desc = could not find container \"d8efe79ab9d9d2cee14f023376facf07f6bf3d680db7bd4489893d8432f8bc3d\": container with ID starting with d8efe79ab9d9d2cee14f023376facf07f6bf3d680db7bd4489893d8432f8bc3d not found: ID does not exist"
Oct 30 01:19:56.564166 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:56.564143    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c140dcb-40bd-4cd2-ac16-df81ed774c16-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c140dcb-40bd-4cd2-ac16-df81ed774c16-istio-data\") pod \"4c140dcb-40bd-4cd2-ac16-df81ed774c16\" (UID: \"4c140dcb-40bd-4cd2-ac16-df81ed774c16\") "
Oct 30 01:19:56.564295 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:56.564279    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4c140dcb-40bd-4cd2-ac16-df81ed774c16-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c140dcb-40bd-4cd2-ac16-df81ed774c16-istio-podinfo\") pod \"4c140dcb-40bd-4cd2-ac16-df81ed774c16\" (UID: \"4c140dcb-40bd-4cd2-ac16-df81ed774c16\") "
Oct 30 01:19:56.564352 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:56.564322    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c140dcb-40bd-4cd2-ac16-df81ed774c16-istio-token\" (UniqueName: \"kubernetes.io/projected/4c140dcb-40bd-4cd2-ac16-df81ed774c16-istio-token\") pod \"4c140dcb-40bd-4cd2-ac16-df81ed774c16\" (UID: \"4c140dcb-40bd-4cd2-ac16-df81ed774c16\") "
Oct 30 01:19:56.564414 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:56.564391    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c140dcb-40bd-4cd2-ac16-df81ed774c16-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c140dcb-40bd-4cd2-ac16-df81ed774c16-credential-socket\") pod \"4c140dcb-40bd-4cd2-ac16-df81ed774c16\" (UID: \"4c140dcb-40bd-4cd2-ac16-df81ed774c16\") "
Oct 30 01:19:56.564483 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:56.564460    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c140dcb-40bd-4cd2-ac16-df81ed774c16-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c140dcb-40bd-4cd2-ac16-df81ed774c16-workload-socket\") pod \"4c140dcb-40bd-4cd2-ac16-df81ed774c16\" (UID: \"4c140dcb-40bd-4cd2-ac16-df81ed774c16\") "
Oct 30 01:19:56.564483 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:56.564472    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c140dcb-40bd-4cd2-ac16-df81ed774c16-istio-data" pod "4c140dcb-40bd-4cd2-ac16-df81ed774c16" (UID: "4c140dcb-40bd-4cd2-ac16-df81ed774c16"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:19:56.564608 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:56.564531    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c140dcb-40bd-4cd2-ac16-df81ed774c16-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c140dcb-40bd-4cd2-ac16-df81ed774c16-istiod-ca-cert\") pod \"4c140dcb-40bd-4cd2-ac16-df81ed774c16\" (UID: \"4c140dcb-40bd-4cd2-ac16-df81ed774c16\") "
Oct 30 01:19:56.564608 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:56.564565    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c140dcb-40bd-4cd2-ac16-df81ed774c16-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c140dcb-40bd-4cd2-ac16-df81ed774c16-workload-certs\") pod \"4c140dcb-40bd-4cd2-ac16-df81ed774c16\" (UID: \"4c140dcb-40bd-4cd2-ac16-df81ed774c16\") "
Oct 30 01:19:56.564608 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:56.564595    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c140dcb-40bd-4cd2-ac16-df81ed774c16-kube-api-access-txhw8\" (UniqueName: \"kubernetes.io/projected/4c140dcb-40bd-4cd2-ac16-df81ed774c16-kube-api-access-txhw8\") pod \"4c140dcb-40bd-4cd2-ac16-df81ed774c16\" (UID: \"4c140dcb-40bd-4cd2-ac16-df81ed774c16\") "
Oct 30 01:19:56.564777 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:56.564670    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c140dcb-40bd-4cd2-ac16-df81ed774c16-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c140dcb-40bd-4cd2-ac16-df81ed774c16-istio-envoy\") pod \"4c140dcb-40bd-4cd2-ac16-df81ed774c16\" (UID: \"4c140dcb-40bd-4cd2-ac16-df81ed774c16\") "
Oct 30 01:19:56.564876 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:56.564846    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c140dcb-40bd-4cd2-ac16-df81ed774c16-workload-socket" pod "4c140dcb-40bd-4cd2-ac16-df81ed774c16" (UID: "4c140dcb-40bd-4cd2-ac16-df81ed774c16"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:19:56.565138 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:56.565087    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c140dcb-40bd-4cd2-ac16-df81ed774c16-credential-socket" pod "4c140dcb-40bd-4cd2-ac16-df81ed774c16" (UID: "4c140dcb-40bd-4cd2-ac16-df81ed774c16"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:19:56.565138 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:56.565099    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c140dcb-40bd-4cd2-ac16-df81ed774c16-workload-certs" pod "4c140dcb-40bd-4cd2-ac16-df81ed774c16" (UID: "4c140dcb-40bd-4cd2-ac16-df81ed774c16"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:19:56.565557 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:56.565334    5002 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c140dcb-40bd-4cd2-ac16-df81ed774c16-credential-socket\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:19:56.565557 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:56.565358    5002 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c140dcb-40bd-4cd2-ac16-df81ed774c16-workload-socket\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:19:56.565557 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:56.565372    5002 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c140dcb-40bd-4cd2-ac16-df81ed774c16-workload-certs\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:19:56.565557 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:56.565397    5002 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c140dcb-40bd-4cd2-ac16-df81ed774c16-istio-data\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:19:56.565557 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:56.565524    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c140dcb-40bd-4cd2-ac16-df81ed774c16-istiod-ca-cert" pod "4c140dcb-40bd-4cd2-ac16-df81ed774c16" (UID: "4c140dcb-40bd-4cd2-ac16-df81ed774c16"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:19:56.566943 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:56.566910    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4c140dcb-40bd-4cd2-ac16-df81ed774c16-istio-podinfo" pod "4c140dcb-40bd-4cd2-ac16-df81ed774c16" (UID: "4c140dcb-40bd-4cd2-ac16-df81ed774c16"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:19:56.567222 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:56.567201    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c140dcb-40bd-4cd2-ac16-df81ed774c16-istio-token" pod "4c140dcb-40bd-4cd2-ac16-df81ed774c16" (UID: "4c140dcb-40bd-4cd2-ac16-df81ed774c16"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:19:56.567360 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:56.567338    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c140dcb-40bd-4cd2-ac16-df81ed774c16-istio-envoy" pod "4c140dcb-40bd-4cd2-ac16-df81ed774c16" (UID: "4c140dcb-40bd-4cd2-ac16-df81ed774c16"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:19:56.567422 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:56.567342    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c140dcb-40bd-4cd2-ac16-df81ed774c16-kube-api-access-txhw8" pod "4c140dcb-40bd-4cd2-ac16-df81ed774c16" (UID: "4c140dcb-40bd-4cd2-ac16-df81ed774c16"). InnerVolumeSpecName "kube-api-access-txhw8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:19:56.667873 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:56.667709    5002 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c140dcb-40bd-4cd2-ac16-df81ed774c16-istio-podinfo\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:19:56.667873 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:56.667747    5002 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c140dcb-40bd-4cd2-ac16-df81ed774c16-istio-token\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:19:56.667873 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:56.667761    5002 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c140dcb-40bd-4cd2-ac16-df81ed774c16-istiod-ca-cert\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:19:56.667873 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:56.667774    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-txhw8\" (UniqueName: \"kubernetes.io/projected/4c140dcb-40bd-4cd2-ac16-df81ed774c16-kube-api-access-txhw8\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:19:56.667873 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:56.667788    5002 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c140dcb-40bd-4cd2-ac16-df81ed774c16-istio-envoy\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:19:56.709404 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:56.692987    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7d8z4"]
Oct 30 01:19:56.873248 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:56.873210    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-01215f-openshift-default-95bfd666f-rkm78"]
Oct 30 01:19:56.876488 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:56.876458    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-01215f-openshift-default-95bfd666f-rkm78"]
Oct 30 01:19:57.530186 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:57.530151    5002 generic.go:356] "Generic (PLEG): container finished" podID="19abf60c-9f0e-47c8-9679-970496e6772c" containerID="15f26cf4361f3eea12c3580685bc363574cce9f0653ff0122924c0552072b762" exitCode=0
Oct 30 01:19:57.530587 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:57.530228    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-x49mt" event={"ID":"19abf60c-9f0e-47c8-9679-970496e6772c","Type":"ContainerDied","Data":"15f26cf4361f3eea12c3580685bc363574cce9f0653ff0122924c0552072b762"}
Oct 30 01:19:57.680035 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:57.679968    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-x49mt"
Oct 30 01:19:57.792886 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:57.792854    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19abf60c-9f0e-47c8-9679-970496e6772c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19abf60c-9f0e-47c8-9679-970496e6772c-credential-socket\") pod \"19abf60c-9f0e-47c8-9679-970496e6772c\" (UID: \"19abf60c-9f0e-47c8-9679-970496e6772c\") "
Oct 30 01:19:57.793073 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:57.792922    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19abf60c-9f0e-47c8-9679-970496e6772c-kube-api-access-dlc9t\" (UniqueName: \"kubernetes.io/projected/19abf60c-9f0e-47c8-9679-970496e6772c-kube-api-access-dlc9t\") pod \"19abf60c-9f0e-47c8-9679-970496e6772c\" (UID: \"19abf60c-9f0e-47c8-9679-970496e6772c\") "
Oct 30 01:19:57.793073 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:57.792965    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19abf60c-9f0e-47c8-9679-970496e6772c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19abf60c-9f0e-47c8-9679-970496e6772c-workload-socket\") pod \"19abf60c-9f0e-47c8-9679-970496e6772c\" (UID: \"19abf60c-9f0e-47c8-9679-970496e6772c\") "
Oct 30 01:19:57.793073 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:57.792999    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19abf60c-9f0e-47c8-9679-970496e6772c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19abf60c-9f0e-47c8-9679-970496e6772c-workload-certs\") pod \"19abf60c-9f0e-47c8-9679-970496e6772c\" (UID: \"19abf60c-9f0e-47c8-9679-970496e6772c\") "
Oct 30 01:19:57.793073 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:57.793058    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19abf60c-9f0e-47c8-9679-970496e6772c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19abf60c-9f0e-47c8-9679-970496e6772c-istio-data\") pod \"19abf60c-9f0e-47c8-9679-970496e6772c\" (UID: \"19abf60c-9f0e-47c8-9679-970496e6772c\") "
Oct 30 01:19:57.793322 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:57.793096    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19abf60c-9f0e-47c8-9679-970496e6772c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19abf60c-9f0e-47c8-9679-970496e6772c-istio-envoy\") pod \"19abf60c-9f0e-47c8-9679-970496e6772c\" (UID: \"19abf60c-9f0e-47c8-9679-970496e6772c\") "
Oct 30 01:19:57.793322 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:57.793154    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19abf60c-9f0e-47c8-9679-970496e6772c-credential-socket" pod "19abf60c-9f0e-47c8-9679-970496e6772c" (UID: "19abf60c-9f0e-47c8-9679-970496e6772c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:19:57.793322 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:57.793170    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19abf60c-9f0e-47c8-9679-970496e6772c-istio-token\" (UniqueName: \"kubernetes.io/projected/19abf60c-9f0e-47c8-9679-970496e6772c-istio-token\") pod \"19abf60c-9f0e-47c8-9679-970496e6772c\" (UID: \"19abf60c-9f0e-47c8-9679-970496e6772c\") "
Oct 30 01:19:57.793322 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:57.793254    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19abf60c-9f0e-47c8-9679-970496e6772c-workload-socket" pod "19abf60c-9f0e-47c8-9679-970496e6772c" (UID: "19abf60c-9f0e-47c8-9679-970496e6772c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:19:57.793322 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:57.793318    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/19abf60c-9f0e-47c8-9679-970496e6772c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19abf60c-9f0e-47c8-9679-970496e6772c-istio-podinfo\") pod \"19abf60c-9f0e-47c8-9679-970496e6772c\" (UID: \"19abf60c-9f0e-47c8-9679-970496e6772c\") "
Oct 30 01:19:57.793631 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:57.793354    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19abf60c-9f0e-47c8-9679-970496e6772c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19abf60c-9f0e-47c8-9679-970496e6772c-istiod-ca-cert\") pod \"19abf60c-9f0e-47c8-9679-970496e6772c\" (UID: \"19abf60c-9f0e-47c8-9679-970496e6772c\") "
Oct 30 01:19:57.793723 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:57.793688    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19abf60c-9f0e-47c8-9679-970496e6772c-workload-certs" pod "19abf60c-9f0e-47c8-9679-970496e6772c" (UID: "19abf60c-9f0e-47c8-9679-970496e6772c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:19:57.793790 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:57.793723    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19abf60c-9f0e-47c8-9679-970496e6772c-istio-data" pod "19abf60c-9f0e-47c8-9679-970496e6772c" (UID: "19abf60c-9f0e-47c8-9679-970496e6772c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:19:57.794022 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:57.793997    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19abf60c-9f0e-47c8-9679-970496e6772c-istiod-ca-cert" pod "19abf60c-9f0e-47c8-9679-970496e6772c" (UID: "19abf60c-9f0e-47c8-9679-970496e6772c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:19:57.794139 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:57.794122    5002 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19abf60c-9f0e-47c8-9679-970496e6772c-istiod-ca-cert\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:19:57.794213 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:57.794141    5002 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19abf60c-9f0e-47c8-9679-970496e6772c-credential-socket\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:19:57.794213 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:57.794150    5002 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19abf60c-9f0e-47c8-9679-970496e6772c-workload-socket\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:19:57.794213 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:57.794160    5002 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19abf60c-9f0e-47c8-9679-970496e6772c-workload-certs\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:19:57.794213 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:57.794167    5002 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19abf60c-9f0e-47c8-9679-970496e6772c-istio-data\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:19:57.795445 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:57.795419    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19abf60c-9f0e-47c8-9679-970496e6772c-kube-api-access-dlc9t" pod "19abf60c-9f0e-47c8-9679-970496e6772c" (UID: "19abf60c-9f0e-47c8-9679-970496e6772c"). InnerVolumeSpecName "kube-api-access-dlc9t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:19:57.795716 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:57.795699    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19abf60c-9f0e-47c8-9679-970496e6772c-istio-envoy" pod "19abf60c-9f0e-47c8-9679-970496e6772c" (UID: "19abf60c-9f0e-47c8-9679-970496e6772c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:19:57.795884 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:57.795865    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/19abf60c-9f0e-47c8-9679-970496e6772c-istio-podinfo" pod "19abf60c-9f0e-47c8-9679-970496e6772c" (UID: "19abf60c-9f0e-47c8-9679-970496e6772c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:19:57.795925 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:57.795895    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19abf60c-9f0e-47c8-9679-970496e6772c-istio-token" pod "19abf60c-9f0e-47c8-9679-970496e6772c" (UID: "19abf60c-9f0e-47c8-9679-970496e6772c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:19:57.896001 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:57.895961    5002 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19abf60c-9f0e-47c8-9679-970496e6772c-istio-token\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:19:57.896001 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:57.895997    5002 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19abf60c-9f0e-47c8-9679-970496e6772c-istio-podinfo\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:19:57.896001 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:57.896007    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dlc9t\" (UniqueName: \"kubernetes.io/projected/19abf60c-9f0e-47c8-9679-970496e6772c-kube-api-access-dlc9t\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:19:57.896245 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:57.896018    5002 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19abf60c-9f0e-47c8-9679-970496e6772c-istio-envoy\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:19:58.515405 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:58.515333    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c140dcb-40bd-4cd2-ac16-df81ed774c16" path="/var/lib/kubelet/pods/4c140dcb-40bd-4cd2-ac16-df81ed774c16/volumes"
Oct 30 01:19:58.548622 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:58.548582    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-x49mt" event={"ID":"19abf60c-9f0e-47c8-9679-970496e6772c","Type":"ContainerDied","Data":"6a0a417e628fd6686455d0688856fb7d1cff6213eaa53155eb30366f0a6c3eff"}
Oct 30 01:19:58.549142 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:58.548625    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-x49mt"
Oct 30 01:19:58.549142 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:58.548674    5002 scope.go:122] "RemoveContainer" containerID="15f26cf4361f3eea12c3580685bc363574cce9f0653ff0122924c0552072b762"
Oct 30 01:19:58.607239 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:58.607200    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-x49mt"]
Oct 30 01:19:58.621014 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:19:58.620977    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-x49mt"]
Oct 30 01:20:00.496842 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:00.496802    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19abf60c-9f0e-47c8-9679-970496e6772c" path="/var/lib/kubelet/pods/19abf60c-9f0e-47c8-9679-970496e6772c/volumes"
Oct 30 01:20:04.532061 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:04.532023    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-658c99-openshift-default-54d78694df-cfhwj"]
Oct 30 01:20:04.532994 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:04.532967    5002 cpu_manager.go:414] "containerMap: removing container" podUID="4c140dcb-40bd-4cd2-ac16-df81ed774c16" containerName="istio-proxy"
Oct 30 01:20:04.532994 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:04.532993    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c140dcb-40bd-4cd2-ac16-df81ed774c16" containerName="istio-proxy"
Oct 30 01:20:04.533161 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:04.533024    5002 cpu_manager.go:414] "containerMap: removing container" podUID="19abf60c-9f0e-47c8-9679-970496e6772c" containerName="istio-proxy"
Oct 30 01:20:04.533161 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:04.533032    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19abf60c-9f0e-47c8-9679-970496e6772c" containerName="istio-proxy"
Oct 30 01:20:04.533161 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:04.533064    5002 cpu_manager.go:414] "containerMap: removing container" podUID="b55fafdf-43af-45d6-a7bb-fb38659c3fe4" containerName="revoke-keys"
Oct 30 01:20:04.533161 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:04.533071    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b55fafdf-43af-45d6-a7bb-fb38659c3fe4" containerName="revoke-keys"
Oct 30 01:20:04.533294 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:04.533241    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c140dcb-40bd-4cd2-ac16-df81ed774c16" containerName="istio-proxy"
Oct 30 01:20:04.533294 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:04.533256    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="19abf60c-9f0e-47c8-9679-970496e6772c" containerName="istio-proxy"
Oct 30 01:20:04.533294 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:04.533271    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="b55fafdf-43af-45d6-a7bb-fb38659c3fe4" containerName="revoke-keys"
Oct 30 01:20:04.538733 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:04.538701    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-658c99-openshift-default-54d78694df-cfhwj"
Oct 30 01:20:04.566143 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:04.566106    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-658c99-openshift-default-dockercfg-lzf9t\""
Oct 30 01:20:04.571791 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:04.571758    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-658c99-openshift-default-54d78694df-cfhwj"]
Oct 30 01:20:04.636064 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:04.636021    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7077b8b8-1f35-49c4-9e58-762297c046cf-istiod-ca-cert\") pod \"e2e-ipp-b-658c99-openshift-default-54d78694df-cfhwj\" (UID: \"7077b8b8-1f35-49c4-9e58-762297c046cf\") " pod="openshift-ingress/e2e-ipp-b-658c99-openshift-default-54d78694df-cfhwj"
Oct 30 01:20:04.636241 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:04.636142    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7077b8b8-1f35-49c4-9e58-762297c046cf-istio-token\") pod \"e2e-ipp-b-658c99-openshift-default-54d78694df-cfhwj\" (UID: \"7077b8b8-1f35-49c4-9e58-762297c046cf\") " pod="openshift-ingress/e2e-ipp-b-658c99-openshift-default-54d78694df-cfhwj"
Oct 30 01:20:04.636311 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:04.636235    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7077b8b8-1f35-49c4-9e58-762297c046cf-workload-socket\") pod \"e2e-ipp-b-658c99-openshift-default-54d78694df-cfhwj\" (UID: \"7077b8b8-1f35-49c4-9e58-762297c046cf\") " pod="openshift-ingress/e2e-ipp-b-658c99-openshift-default-54d78694df-cfhwj"
Oct 30 01:20:04.636373 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:04.636335    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7077b8b8-1f35-49c4-9e58-762297c046cf-credential-socket\") pod \"e2e-ipp-b-658c99-openshift-default-54d78694df-cfhwj\" (UID: \"7077b8b8-1f35-49c4-9e58-762297c046cf\") " pod="openshift-ingress/e2e-ipp-b-658c99-openshift-default-54d78694df-cfhwj"
Oct 30 01:20:04.636441 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:04.636374    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7077b8b8-1f35-49c4-9e58-762297c046cf-istio-podinfo\") pod \"e2e-ipp-b-658c99-openshift-default-54d78694df-cfhwj\" (UID: \"7077b8b8-1f35-49c4-9e58-762297c046cf\") " pod="openshift-ingress/e2e-ipp-b-658c99-openshift-default-54d78694df-cfhwj"
Oct 30 01:20:04.636516 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:04.636497    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7077b8b8-1f35-49c4-9e58-762297c046cf-istio-envoy\") pod \"e2e-ipp-b-658c99-openshift-default-54d78694df-cfhwj\" (UID: \"7077b8b8-1f35-49c4-9e58-762297c046cf\") " pod="openshift-ingress/e2e-ipp-b-658c99-openshift-default-54d78694df-cfhwj"
Oct 30 01:20:04.636562 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:04.636528    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7077b8b8-1f35-49c4-9e58-762297c046cf-workload-certs\") pod \"e2e-ipp-b-658c99-openshift-default-54d78694df-cfhwj\" (UID: \"7077b8b8-1f35-49c4-9e58-762297c046cf\") " pod="openshift-ingress/e2e-ipp-b-658c99-openshift-default-54d78694df-cfhwj"
Oct 30 01:20:04.636638 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:04.636621    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7077b8b8-1f35-49c4-9e58-762297c046cf-istio-data\") pod \"e2e-ipp-b-658c99-openshift-default-54d78694df-cfhwj\" (UID: \"7077b8b8-1f35-49c4-9e58-762297c046cf\") " pod="openshift-ingress/e2e-ipp-b-658c99-openshift-default-54d78694df-cfhwj"
Oct 30 01:20:04.636675 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:04.636657    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8w4t4\" (UniqueName: \"kubernetes.io/projected/7077b8b8-1f35-49c4-9e58-762297c046cf-kube-api-access-8w4t4\") pod \"e2e-ipp-b-658c99-openshift-default-54d78694df-cfhwj\" (UID: \"7077b8b8-1f35-49c4-9e58-762297c046cf\") " pod="openshift-ingress/e2e-ipp-b-658c99-openshift-default-54d78694df-cfhwj"
Oct 30 01:20:04.739406 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:04.739153    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7077b8b8-1f35-49c4-9e58-762297c046cf-istiod-ca-cert\") pod \"e2e-ipp-b-658c99-openshift-default-54d78694df-cfhwj\" (UID: \"7077b8b8-1f35-49c4-9e58-762297c046cf\") " pod="openshift-ingress/e2e-ipp-b-658c99-openshift-default-54d78694df-cfhwj"
Oct 30 01:20:04.739406 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:04.739252    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7077b8b8-1f35-49c4-9e58-762297c046cf-istio-token\") pod \"e2e-ipp-b-658c99-openshift-default-54d78694df-cfhwj\" (UID: \"7077b8b8-1f35-49c4-9e58-762297c046cf\") " pod="openshift-ingress/e2e-ipp-b-658c99-openshift-default-54d78694df-cfhwj"
Oct 30 01:20:04.739406 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:04.739285    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7077b8b8-1f35-49c4-9e58-762297c046cf-workload-socket\") pod \"e2e-ipp-b-658c99-openshift-default-54d78694df-cfhwj\" (UID: \"7077b8b8-1f35-49c4-9e58-762297c046cf\") " pod="openshift-ingress/e2e-ipp-b-658c99-openshift-default-54d78694df-cfhwj"
Oct 30 01:20:04.739406 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:04.739351    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7077b8b8-1f35-49c4-9e58-762297c046cf-credential-socket\") pod \"e2e-ipp-b-658c99-openshift-default-54d78694df-cfhwj\" (UID: \"7077b8b8-1f35-49c4-9e58-762297c046cf\") " pod="openshift-ingress/e2e-ipp-b-658c99-openshift-default-54d78694df-cfhwj"
Oct 30 01:20:04.739406 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:04.739408    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7077b8b8-1f35-49c4-9e58-762297c046cf-istio-podinfo\") pod \"e2e-ipp-b-658c99-openshift-default-54d78694df-cfhwj\" (UID: \"7077b8b8-1f35-49c4-9e58-762297c046cf\") " pod="openshift-ingress/e2e-ipp-b-658c99-openshift-default-54d78694df-cfhwj"
Oct 30 01:20:04.739914 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:04.739470    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7077b8b8-1f35-49c4-9e58-762297c046cf-istio-envoy\") pod \"e2e-ipp-b-658c99-openshift-default-54d78694df-cfhwj\" (UID: \"7077b8b8-1f35-49c4-9e58-762297c046cf\") " pod="openshift-ingress/e2e-ipp-b-658c99-openshift-default-54d78694df-cfhwj"
Oct 30 01:20:04.739914 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:04.739501    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7077b8b8-1f35-49c4-9e58-762297c046cf-workload-certs\") pod \"e2e-ipp-b-658c99-openshift-default-54d78694df-cfhwj\" (UID: \"7077b8b8-1f35-49c4-9e58-762297c046cf\") " pod="openshift-ingress/e2e-ipp-b-658c99-openshift-default-54d78694df-cfhwj"
Oct 30 01:20:04.739914 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:04.739605    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7077b8b8-1f35-49c4-9e58-762297c046cf-istio-data\") pod \"e2e-ipp-b-658c99-openshift-default-54d78694df-cfhwj\" (UID: \"7077b8b8-1f35-49c4-9e58-762297c046cf\") " pod="openshift-ingress/e2e-ipp-b-658c99-openshift-default-54d78694df-cfhwj"
Oct 30 01:20:04.739914 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:04.739641    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8w4t4\" (UniqueName: \"kubernetes.io/projected/7077b8b8-1f35-49c4-9e58-762297c046cf-kube-api-access-8w4t4\") pod \"e2e-ipp-b-658c99-openshift-default-54d78694df-cfhwj\" (UID: \"7077b8b8-1f35-49c4-9e58-762297c046cf\") " pod="openshift-ingress/e2e-ipp-b-658c99-openshift-default-54d78694df-cfhwj"
Oct 30 01:20:04.747403 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:04.740871    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7077b8b8-1f35-49c4-9e58-762297c046cf-istiod-ca-cert\") pod \"e2e-ipp-b-658c99-openshift-default-54d78694df-cfhwj\" (UID: \"7077b8b8-1f35-49c4-9e58-762297c046cf\") " pod="openshift-ingress/e2e-ipp-b-658c99-openshift-default-54d78694df-cfhwj"
Oct 30 01:20:04.747403 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:04.741432    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7077b8b8-1f35-49c4-9e58-762297c046cf-workload-socket\") pod \"e2e-ipp-b-658c99-openshift-default-54d78694df-cfhwj\" (UID: \"7077b8b8-1f35-49c4-9e58-762297c046cf\") " pod="openshift-ingress/e2e-ipp-b-658c99-openshift-default-54d78694df-cfhwj"
Oct 30 01:20:04.747403 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:04.741672    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7077b8b8-1f35-49c4-9e58-762297c046cf-credential-socket\") pod \"e2e-ipp-b-658c99-openshift-default-54d78694df-cfhwj\" (UID: \"7077b8b8-1f35-49c4-9e58-762297c046cf\") " pod="openshift-ingress/e2e-ipp-b-658c99-openshift-default-54d78694df-cfhwj"
Oct 30 01:20:04.747403 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:04.745396    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7077b8b8-1f35-49c4-9e58-762297c046cf-workload-certs\") pod \"e2e-ipp-b-658c99-openshift-default-54d78694df-cfhwj\" (UID: \"7077b8b8-1f35-49c4-9e58-762297c046cf\") " pod="openshift-ingress/e2e-ipp-b-658c99-openshift-default-54d78694df-cfhwj"
Oct 30 01:20:04.747403 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:04.746208    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7077b8b8-1f35-49c4-9e58-762297c046cf-istio-data\") pod \"e2e-ipp-b-658c99-openshift-default-54d78694df-cfhwj\" (UID: \"7077b8b8-1f35-49c4-9e58-762297c046cf\") " pod="openshift-ingress/e2e-ipp-b-658c99-openshift-default-54d78694df-cfhwj"
Oct 30 01:20:04.755409 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:04.748219    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7077b8b8-1f35-49c4-9e58-762297c046cf-istio-podinfo\") pod \"e2e-ipp-b-658c99-openshift-default-54d78694df-cfhwj\" (UID: \"7077b8b8-1f35-49c4-9e58-762297c046cf\") " pod="openshift-ingress/e2e-ipp-b-658c99-openshift-default-54d78694df-cfhwj"
Oct 30 01:20:04.755409 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:04.754652    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7077b8b8-1f35-49c4-9e58-762297c046cf-istio-envoy\") pod \"e2e-ipp-b-658c99-openshift-default-54d78694df-cfhwj\" (UID: \"7077b8b8-1f35-49c4-9e58-762297c046cf\") " pod="openshift-ingress/e2e-ipp-b-658c99-openshift-default-54d78694df-cfhwj"
Oct 30 01:20:04.778009 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:04.772431    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8w4t4\" (UniqueName: \"kubernetes.io/projected/7077b8b8-1f35-49c4-9e58-762297c046cf-kube-api-access-8w4t4\") pod \"e2e-ipp-b-658c99-openshift-default-54d78694df-cfhwj\" (UID: \"7077b8b8-1f35-49c4-9e58-762297c046cf\") " pod="openshift-ingress/e2e-ipp-b-658c99-openshift-default-54d78694df-cfhwj"
Oct 30 01:20:04.799306 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:04.783104    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7077b8b8-1f35-49c4-9e58-762297c046cf-istio-token\") pod \"e2e-ipp-b-658c99-openshift-default-54d78694df-cfhwj\" (UID: \"7077b8b8-1f35-49c4-9e58-762297c046cf\") " pod="openshift-ingress/e2e-ipp-b-658c99-openshift-default-54d78694df-cfhwj"
Oct 30 01:20:04.863416 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:04.863059    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-658c99-openshift-default-54d78694df-cfhwj"
Oct 30 01:20:05.298941 ip-10-0-128-7 kubenswrapper[5002]: W1030 01:20:05.289118    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7077b8b8_1f35_49c4_9e58_762297c046cf.slice/crio-ee0e51aaa09d576ce44d698d46b37c6d636b14d6b7ca14d940fcb43cc53f4306 WatchSource:0}: Error finding container ee0e51aaa09d576ce44d698d46b37c6d636b14d6b7ca14d940fcb43cc53f4306: Status 404 returned error can't find the container with id ee0e51aaa09d576ce44d698d46b37c6d636b14d6b7ca14d940fcb43cc53f4306
Oct 30 01:20:05.298941 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:05.291767    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-658c99-openshift-default-54d78694df-cfhwj"]
Oct 30 01:20:05.298941 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:05.292169    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:20:05.298941 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:05.292237    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:20:05.298941 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:05.292271    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:20:05.641654 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:05.641553    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-658c99-openshift-default-54d78694df-cfhwj" event={"ID":"7077b8b8-1f35-49c4-9e58-762297c046cf","Type":"ContainerStarted","Data":"3f0d5174659d934cb002b76b41d644fd6133deca3d9b55042a24a21b138cf830"}
Oct 30 01:20:05.641654 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:05.641619    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-658c99-openshift-default-54d78694df-cfhwj" event={"ID":"7077b8b8-1f35-49c4-9e58-762297c046cf","Type":"ContainerStarted","Data":"ee0e51aaa09d576ce44d698d46b37c6d636b14d6b7ca14d940fcb43cc53f4306"}
Oct 30 01:20:05.695321 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:05.695244    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-658c99-openshift-default-54d78694df-cfhwj" podStartSLOduration=1.695226093 podStartE2EDuration="1.695226093s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:20:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:20:05.693846624 +0000 UTC m=+2901.310279517" watchObservedRunningTime="2026-10-30 01:20:05.695226093 +0000 UTC m=+2901.311658973"
Oct 30 01:20:05.864405 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:05.863584    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-658c99-openshift-default-54d78694df-cfhwj"
Oct 30 01:20:06.870654 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:06.870613    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-658c99-openshift-default-54d78694df-cfhwj"
Oct 30 01:20:07.658157 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:07.658117    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-658c99-openshift-default-54d78694df-cfhwj"
Oct 30 01:20:07.659202 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:07.659181    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-658c99-openshift-default-54d78694df-cfhwj"
Oct 30 01:20:12.324085 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:12.324049    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-9a8001-hjs8l"]
Oct 30 01:20:12.337437 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:12.337408    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-9a8001-hjs8l"]
Oct 30 01:20:12.498957 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:12.498915    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b55fafdf-43af-45d6-a7bb-fb38659c3fe4" path="/var/lib/kubelet/pods/b55fafdf-43af-45d6-a7bb-fb38659c3fe4/volumes"
Oct 30 01:20:12.861762 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:20:12.861720    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbjdq" containerName="extractor"
Oct 30 01:20:19.347637 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:19.347595    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-9a8001-openshift-default-d885f8f7f-dt8r5"]
Oct 30 01:20:19.348138 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:19.347811    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-9a8001-openshift-default-d885f8f7f-dt8r5" podUID="423cd7a9-8a7c-4b26-b7c2-ba0e4053c042" containerName="istio-proxy" containerID="cri-o://8c0c4062cdc81472706ee2c877619232632302e6cbf645e3f4e36bf61b11b94a" gracePeriod=30
Oct 30 01:20:19.923882 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:19.923842    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-93079c-66kbl"]
Oct 30 01:20:19.924767 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:19.924744    5002 cpu_manager.go:414] "containerMap: removing container" podUID="4c140dcb-40bd-4cd2-ac16-df81ed774c16" containerName="istio-proxy"
Oct 30 01:20:19.924835 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:19.924771    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c140dcb-40bd-4cd2-ac16-df81ed774c16" containerName="istio-proxy"
Oct 30 01:20:19.924964 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:19.924951    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c140dcb-40bd-4cd2-ac16-df81ed774c16" containerName="istio-proxy"
Oct 30 01:20:19.929161 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:19.929137    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-93079c-66kbl"
Oct 30 01:20:19.946229 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:19.946206    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:20:19.946429 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:19.946279    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:20:19.947615 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:19.947585    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-93079c-66kbl"]
Oct 30 01:20:19.981295 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:19.976598    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:20:19.981295 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:19.976628    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5phsh\""
Oct 30 01:20:19.993857 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:19.993829    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f6d5b8-554bh"]
Oct 30 01:20:19.998439 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:19.998418    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f6d5b8-554bh"
Oct 30 01:20:20.084676 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:20.084639    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f6d5b8-554bh"]
Oct 30 01:20:20.094420 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:20.094359    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rnqv\" (UniqueName: \"kubernetes.io/projected/cd051518-6e55-4dd2-91a4-052386ed2328-kube-api-access-7rnqv\") pod \"maas-api-revoke-keys-e2e-ipp-a-f6d5b8-554bh\" (UID: \"cd051518-6e55-4dd2-91a4-052386ed2328\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f6d5b8-554bh"
Oct 30 01:20:20.094725 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:20.094698    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drlms\" (UniqueName: \"kubernetes.io/projected/87a79eb3-d09c-4519-9c1a-fa8c7fdef2e6-kube-api-access-drlms\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-93079c-66kbl\" (UID: \"87a79eb3-d09c-4519-9c1a-fa8c7fdef2e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-93079c-66kbl"
Oct 30 01:20:20.094827 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:20.094765    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87a79eb3-d09c-4519-9c1a-fa8c7fdef2e6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-93079c-66kbl\" (UID: \"87a79eb3-d09c-4519-9c1a-fa8c7fdef2e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-93079c-66kbl"
Oct 30 01:20:20.094827 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:20.094800    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd051518-6e55-4dd2-91a4-052386ed2328-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f6d5b8-554bh\" (UID: \"cd051518-6e55-4dd2-91a4-052386ed2328\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f6d5b8-554bh"
Oct 30 01:20:20.094948 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:20.094831    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87a79eb3-d09c-4519-9c1a-fa8c7fdef2e6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-93079c-66kbl\" (UID: \"87a79eb3-d09c-4519-9c1a-fa8c7fdef2e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-93079c-66kbl"
Oct 30 01:20:20.095010 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:20.094942    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd051518-6e55-4dd2-91a4-052386ed2328-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f6d5b8-554bh\" (UID: \"cd051518-6e55-4dd2-91a4-052386ed2328\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f6d5b8-554bh"
Oct 30 01:20:20.196691 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:20.196655    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drlms\" (UniqueName: \"kubernetes.io/projected/87a79eb3-d09c-4519-9c1a-fa8c7fdef2e6-kube-api-access-drlms\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-93079c-66kbl\" (UID: \"87a79eb3-d09c-4519-9c1a-fa8c7fdef2e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-93079c-66kbl"
Oct 30 01:20:20.196881 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:20.196728    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87a79eb3-d09c-4519-9c1a-fa8c7fdef2e6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-93079c-66kbl\" (UID: \"87a79eb3-d09c-4519-9c1a-fa8c7fdef2e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-93079c-66kbl"
Oct 30 01:20:20.196881 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:20.196756    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd051518-6e55-4dd2-91a4-052386ed2328-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f6d5b8-554bh\" (UID: \"cd051518-6e55-4dd2-91a4-052386ed2328\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f6d5b8-554bh"
Oct 30 01:20:20.196881 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:20.196779    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87a79eb3-d09c-4519-9c1a-fa8c7fdef2e6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-93079c-66kbl\" (UID: \"87a79eb3-d09c-4519-9c1a-fa8c7fdef2e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-93079c-66kbl"
Oct 30 01:20:20.196881 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:20.196868    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd051518-6e55-4dd2-91a4-052386ed2328-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f6d5b8-554bh\" (UID: \"cd051518-6e55-4dd2-91a4-052386ed2328\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f6d5b8-554bh"
Oct 30 01:20:20.197131 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:20.196901    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rnqv\" (UniqueName: \"kubernetes.io/projected/cd051518-6e55-4dd2-91a4-052386ed2328-kube-api-access-7rnqv\") pod \"maas-api-revoke-keys-e2e-ipp-a-f6d5b8-554bh\" (UID: \"cd051518-6e55-4dd2-91a4-052386ed2328\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f6d5b8-554bh"
Oct 30 01:20:20.197572 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:20.197539    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87a79eb3-d09c-4519-9c1a-fa8c7fdef2e6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-93079c-66kbl\" (UID: \"87a79eb3-d09c-4519-9c1a-fa8c7fdef2e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-93079c-66kbl"
Oct 30 01:20:20.197891 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:20.197862    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd051518-6e55-4dd2-91a4-052386ed2328-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f6d5b8-554bh\" (UID: \"cd051518-6e55-4dd2-91a4-052386ed2328\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f6d5b8-554bh"
Oct 30 01:20:20.199704 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:20.199673    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd051518-6e55-4dd2-91a4-052386ed2328-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f6d5b8-554bh\" (UID: \"cd051518-6e55-4dd2-91a4-052386ed2328\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f6d5b8-554bh"
Oct 30 01:20:20.199820 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:20.199702    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87a79eb3-d09c-4519-9c1a-fa8c7fdef2e6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-93079c-66kbl\" (UID: \"87a79eb3-d09c-4519-9c1a-fa8c7fdef2e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-93079c-66kbl"
Oct 30 01:20:20.221852 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:20.221823    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rnqv\" (UniqueName: \"kubernetes.io/projected/cd051518-6e55-4dd2-91a4-052386ed2328-kube-api-access-7rnqv\") pod \"maas-api-revoke-keys-e2e-ipp-a-f6d5b8-554bh\" (UID: \"cd051518-6e55-4dd2-91a4-052386ed2328\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f6d5b8-554bh"
Oct 30 01:20:20.221980 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:20.221901    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-drlms\" (UniqueName: \"kubernetes.io/projected/87a79eb3-d09c-4519-9c1a-fa8c7fdef2e6-kube-api-access-drlms\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-93079c-66kbl\" (UID: \"87a79eb3-d09c-4519-9c1a-fa8c7fdef2e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-93079c-66kbl"
Oct 30 01:20:20.251780 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:20.251747    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-93079c-66kbl"
Oct 30 01:20:20.384705 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:20.384671    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f6d5b8-554bh"
Oct 30 01:20:20.655988 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:20.655941    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-93079c-66kbl"]
Oct 30 01:20:20.745593 ip-10-0-128-7 kubenswrapper[5002]: W1030 01:20:20.745561    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd051518_6e55_4dd2_91a4_052386ed2328.slice/crio-9c117899d64874dc6c54ece7b5fc76a9b0e70d2f48666dbb912432ca5083ae4d WatchSource:0}: Error finding container 9c117899d64874dc6c54ece7b5fc76a9b0e70d2f48666dbb912432ca5083ae4d: Status 404 returned error can't find the container with id 9c117899d64874dc6c54ece7b5fc76a9b0e70d2f48666dbb912432ca5083ae4d
Oct 30 01:20:20.746597 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:20.746571    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f6d5b8-554bh"]
Oct 30 01:20:20.767495 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:20.767450    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f6d5b8-554bh" event={"ID":"cd051518-6e55-4dd2-91a4-052386ed2328","Type":"ContainerStarted","Data":"9c117899d64874dc6c54ece7b5fc76a9b0e70d2f48666dbb912432ca5083ae4d"}
Oct 30 01:20:20.768982 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:20.768940    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-93079c-66kbl" event={"ID":"87a79eb3-d09c-4519-9c1a-fa8c7fdef2e6","Type":"ContainerStarted","Data":"81b2d37217e3ca67fab1b0660b1551495fc6f37df76265a78c63dddf58049b9d"}
Oct 30 01:20:21.779390 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:21.779335    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-93079c-66kbl" event={"ID":"87a79eb3-d09c-4519-9c1a-fa8c7fdef2e6","Type":"ContainerStarted","Data":"c1cbe437652412f8cf52c5ee01bf4aa497dea6c8629ea009a1d43bbff18a9282"}
Oct 30 01:20:21.781641 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:21.781607    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f6d5b8-554bh" event={"ID":"cd051518-6e55-4dd2-91a4-052386ed2328","Type":"ContainerStarted","Data":"80d96b84010ed176848a534e6b53080569a82ff93f3323ee9ab470ea64e34a65"}
Oct 30 01:20:21.826049 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:21.825990    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-93079c-66kbl" podStartSLOduration=2.825975899 podStartE2EDuration="2.825975899s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:20:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:20:21.822803991 +0000 UTC m=+2917.439236869" watchObservedRunningTime="2026-10-30 01:20:21.825975899 +0000 UTC m=+2917.442408778"
Oct 30 01:20:21.867809 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:21.867738    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f6d5b8-554bh" podStartSLOduration=2.867720278 podStartE2EDuration="2.867720278s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:20:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:20:21.865527233 +0000 UTC m=+2917.481960112" watchObservedRunningTime="2026-10-30 01:20:21.867720278 +0000 UTC m=+2917.484153158"
Oct 30 01:20:22.791924 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:22.791889    5002 generic.go:356] "Generic (PLEG): container finished" podID="87a79eb3-d09c-4519-9c1a-fa8c7fdef2e6" containerID="c1cbe437652412f8cf52c5ee01bf4aa497dea6c8629ea009a1d43bbff18a9282" exitCode=0
Oct 30 01:20:22.792981 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:22.791973    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-93079c-66kbl" event={"ID":"87a79eb3-d09c-4519-9c1a-fa8c7fdef2e6","Type":"ContainerDied","Data":"c1cbe437652412f8cf52c5ee01bf4aa497dea6c8629ea009a1d43bbff18a9282"}
Oct 30 01:20:22.794086 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:22.794057    5002 generic.go:356] "Generic (PLEG): container finished" podID="cd051518-6e55-4dd2-91a4-052386ed2328" containerID="80d96b84010ed176848a534e6b53080569a82ff93f3323ee9ab470ea64e34a65" exitCode=0
Oct 30 01:20:22.794200 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:22.794102    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f6d5b8-554bh" event={"ID":"cd051518-6e55-4dd2-91a4-052386ed2328","Type":"ContainerDied","Data":"80d96b84010ed176848a534e6b53080569a82ff93f3323ee9ab470ea64e34a65"}
Oct 30 01:20:24.043253 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:24.043217    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f6d5b8-554bh"
Oct 30 01:20:24.090838 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:24.090432    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-93079c-66kbl"
Oct 30 01:20:24.117954 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:24.117920    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87a79eb3-d09c-4519-9c1a-fa8c7fdef2e6-kube-api-access-drlms\" (UniqueName: \"kubernetes.io/projected/87a79eb3-d09c-4519-9c1a-fa8c7fdef2e6-kube-api-access-drlms\") pod \"87a79eb3-d09c-4519-9c1a-fa8c7fdef2e6\" (UID: \"87a79eb3-d09c-4519-9c1a-fa8c7fdef2e6\") "
Oct 30 01:20:24.117954 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:24.117956    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/87a79eb3-d09c-4519-9c1a-fa8c7fdef2e6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87a79eb3-d09c-4519-9c1a-fa8c7fdef2e6-maas-api-xks-ca\") pod \"87a79eb3-d09c-4519-9c1a-fa8c7fdef2e6\" (UID: \"87a79eb3-d09c-4519-9c1a-fa8c7fdef2e6\") "
Oct 30 01:20:24.118222 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:24.118023    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cd051518-6e55-4dd2-91a4-052386ed2328-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd051518-6e55-4dd2-91a4-052386ed2328-maas-api-xks-ca\") pod \"cd051518-6e55-4dd2-91a4-052386ed2328\" (UID: \"cd051518-6e55-4dd2-91a4-052386ed2328\") "
Oct 30 01:20:24.118222 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:24.118087    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87a79eb3-d09c-4519-9c1a-fa8c7fdef2e6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87a79eb3-d09c-4519-9c1a-fa8c7fdef2e6-maas-api-openshift-ca\") pod \"87a79eb3-d09c-4519-9c1a-fa8c7fdef2e6\" (UID: \"87a79eb3-d09c-4519-9c1a-fa8c7fdef2e6\") "
Oct 30 01:20:24.118589 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:24.118554    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87a79eb3-d09c-4519-9c1a-fa8c7fdef2e6-maas-api-openshift-ca" pod "87a79eb3-d09c-4519-9c1a-fa8c7fdef2e6" (UID: "87a79eb3-d09c-4519-9c1a-fa8c7fdef2e6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:20:24.118748 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:24.118711    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87a79eb3-d09c-4519-9c1a-fa8c7fdef2e6-maas-api-openshift-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:20:24.120654 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:24.120620    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd051518-6e55-4dd2-91a4-052386ed2328-maas-api-xks-ca" pod "cd051518-6e55-4dd2-91a4-052386ed2328" (UID: "cd051518-6e55-4dd2-91a4-052386ed2328"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:20:24.120932 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:24.120898    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87a79eb3-d09c-4519-9c1a-fa8c7fdef2e6-kube-api-access-drlms" pod "87a79eb3-d09c-4519-9c1a-fa8c7fdef2e6" (UID: "87a79eb3-d09c-4519-9c1a-fa8c7fdef2e6"). InnerVolumeSpecName "kube-api-access-drlms". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:20:24.121043 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:24.120926    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87a79eb3-d09c-4519-9c1a-fa8c7fdef2e6-maas-api-xks-ca" pod "87a79eb3-d09c-4519-9c1a-fa8c7fdef2e6" (UID: "87a79eb3-d09c-4519-9c1a-fa8c7fdef2e6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:20:24.219365 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:24.219323    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd051518-6e55-4dd2-91a4-052386ed2328-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd051518-6e55-4dd2-91a4-052386ed2328-maas-api-openshift-ca\") pod \"cd051518-6e55-4dd2-91a4-052386ed2328\" (UID: \"cd051518-6e55-4dd2-91a4-052386ed2328\") "
Oct 30 01:20:24.219365 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:24.219395    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd051518-6e55-4dd2-91a4-052386ed2328-kube-api-access-7rnqv\" (UniqueName: \"kubernetes.io/projected/cd051518-6e55-4dd2-91a4-052386ed2328-kube-api-access-7rnqv\") pod \"cd051518-6e55-4dd2-91a4-052386ed2328\" (UID: \"cd051518-6e55-4dd2-91a4-052386ed2328\") "
Oct 30 01:20:24.219717 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:24.219693    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd051518-6e55-4dd2-91a4-052386ed2328-maas-api-openshift-ca" pod "cd051518-6e55-4dd2-91a4-052386ed2328" (UID: "cd051518-6e55-4dd2-91a4-052386ed2328"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:20:24.219942 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:24.219928    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-drlms\" (UniqueName: \"kubernetes.io/projected/87a79eb3-d09c-4519-9c1a-fa8c7fdef2e6-kube-api-access-drlms\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:20:24.219996 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:24.219945    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87a79eb3-d09c-4519-9c1a-fa8c7fdef2e6-maas-api-xks-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:20:24.219996 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:24.219954    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd051518-6e55-4dd2-91a4-052386ed2328-maas-api-xks-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:20:24.219996 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:24.219962    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd051518-6e55-4dd2-91a4-052386ed2328-maas-api-openshift-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:20:24.221548 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:24.221527    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd051518-6e55-4dd2-91a4-052386ed2328-kube-api-access-7rnqv" pod "cd051518-6e55-4dd2-91a4-052386ed2328" (UID: "cd051518-6e55-4dd2-91a4-052386ed2328"). InnerVolumeSpecName "kube-api-access-7rnqv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:20:24.320782 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:24.320704    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7rnqv\" (UniqueName: \"kubernetes.io/projected/cd051518-6e55-4dd2-91a4-052386ed2328-kube-api-access-7rnqv\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:20:24.610453 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:24.610427    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-9a8001-openshift-default-d885f8f7f-dt8r5"
Oct 30 01:20:24.623679 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:24.623654    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/423cd7a9-8a7c-4b26-b7c2-ba0e4053c042-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/423cd7a9-8a7c-4b26-b7c2-ba0e4053c042-istio-envoy\") pod \"423cd7a9-8a7c-4b26-b7c2-ba0e4053c042\" (UID: \"423cd7a9-8a7c-4b26-b7c2-ba0e4053c042\") "
Oct 30 01:20:24.623776 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:24.623699    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/423cd7a9-8a7c-4b26-b7c2-ba0e4053c042-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/423cd7a9-8a7c-4b26-b7c2-ba0e4053c042-istiod-ca-cert\") pod \"423cd7a9-8a7c-4b26-b7c2-ba0e4053c042\" (UID: \"423cd7a9-8a7c-4b26-b7c2-ba0e4053c042\") "
Oct 30 01:20:24.623841 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:24.623726    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/423cd7a9-8a7c-4b26-b7c2-ba0e4053c042-istio-token\" (UniqueName: \"kubernetes.io/projected/423cd7a9-8a7c-4b26-b7c2-ba0e4053c042-istio-token\") pod \"423cd7a9-8a7c-4b26-b7c2-ba0e4053c042\" (UID: \"423cd7a9-8a7c-4b26-b7c2-ba0e4053c042\") "
Oct 30 01:20:24.623905 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:24.623868    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/423cd7a9-8a7c-4b26-b7c2-ba0e4053c042-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/423cd7a9-8a7c-4b26-b7c2-ba0e4053c042-istio-podinfo\") pod \"423cd7a9-8a7c-4b26-b7c2-ba0e4053c042\" (UID: \"423cd7a9-8a7c-4b26-b7c2-ba0e4053c042\") "
Oct 30 01:20:24.623964 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:24.623921    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/423cd7a9-8a7c-4b26-b7c2-ba0e4053c042-kube-api-access-5rsb4\" (UniqueName: \"kubernetes.io/projected/423cd7a9-8a7c-4b26-b7c2-ba0e4053c042-kube-api-access-5rsb4\") pod \"423cd7a9-8a7c-4b26-b7c2-ba0e4053c042\" (UID: \"423cd7a9-8a7c-4b26-b7c2-ba0e4053c042\") "
Oct 30 01:20:24.624021 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:24.624003    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/423cd7a9-8a7c-4b26-b7c2-ba0e4053c042-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/423cd7a9-8a7c-4b26-b7c2-ba0e4053c042-workload-socket\") pod \"423cd7a9-8a7c-4b26-b7c2-ba0e4053c042\" (UID: \"423cd7a9-8a7c-4b26-b7c2-ba0e4053c042\") "
Oct 30 01:20:24.624083 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:24.624033    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/423cd7a9-8a7c-4b26-b7c2-ba0e4053c042-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/423cd7a9-8a7c-4b26-b7c2-ba0e4053c042-credential-socket\") pod \"423cd7a9-8a7c-4b26-b7c2-ba0e4053c042\" (UID: \"423cd7a9-8a7c-4b26-b7c2-ba0e4053c042\") "
Oct 30 01:20:24.624083 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:24.624069    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/423cd7a9-8a7c-4b26-b7c2-ba0e4053c042-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/423cd7a9-8a7c-4b26-b7c2-ba0e4053c042-istio-data\") pod \"423cd7a9-8a7c-4b26-b7c2-ba0e4053c042\" (UID: \"423cd7a9-8a7c-4b26-b7c2-ba0e4053c042\") "
Oct 30 01:20:24.624209 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:24.624133    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/423cd7a9-8a7c-4b26-b7c2-ba0e4053c042-istiod-ca-cert" pod "423cd7a9-8a7c-4b26-b7c2-ba0e4053c042" (UID: "423cd7a9-8a7c-4b26-b7c2-ba0e4053c042"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:20:24.624209 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:24.624168    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/423cd7a9-8a7c-4b26-b7c2-ba0e4053c042-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/423cd7a9-8a7c-4b26-b7c2-ba0e4053c042-workload-certs\") pod \"423cd7a9-8a7c-4b26-b7c2-ba0e4053c042\" (UID: \"423cd7a9-8a7c-4b26-b7c2-ba0e4053c042\") "
Oct 30 01:20:24.624320 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:24.624294    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/423cd7a9-8a7c-4b26-b7c2-ba0e4053c042-workload-socket" pod "423cd7a9-8a7c-4b26-b7c2-ba0e4053c042" (UID: "423cd7a9-8a7c-4b26-b7c2-ba0e4053c042"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:20:24.624521 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:24.624423    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/423cd7a9-8a7c-4b26-b7c2-ba0e4053c042-credential-socket" pod "423cd7a9-8a7c-4b26-b7c2-ba0e4053c042" (UID: "423cd7a9-8a7c-4b26-b7c2-ba0e4053c042"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:20:24.624521 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:24.624437    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/423cd7a9-8a7c-4b26-b7c2-ba0e4053c042-istio-data" pod "423cd7a9-8a7c-4b26-b7c2-ba0e4053c042" (UID: "423cd7a9-8a7c-4b26-b7c2-ba0e4053c042"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:20:24.624725 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:24.624562    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/423cd7a9-8a7c-4b26-b7c2-ba0e4053c042-workload-certs" pod "423cd7a9-8a7c-4b26-b7c2-ba0e4053c042" (UID: "423cd7a9-8a7c-4b26-b7c2-ba0e4053c042"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:20:24.625158 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:24.625132    5002 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/423cd7a9-8a7c-4b26-b7c2-ba0e4053c042-workload-certs\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:20:24.625232 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:24.625164    5002 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/423cd7a9-8a7c-4b26-b7c2-ba0e4053c042-istiod-ca-cert\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:20:24.625232 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:24.625180    5002 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/423cd7a9-8a7c-4b26-b7c2-ba0e4053c042-workload-socket\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:20:24.625232 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:24.625194    5002 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/423cd7a9-8a7c-4b26-b7c2-ba0e4053c042-credential-socket\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:20:24.625232 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:24.625207    5002 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/423cd7a9-8a7c-4b26-b7c2-ba0e4053c042-istio-data\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:20:24.626108 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:24.626080    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/423cd7a9-8a7c-4b26-b7c2-ba0e4053c042-kube-api-access-5rsb4" pod "423cd7a9-8a7c-4b26-b7c2-ba0e4053c042" (UID: "423cd7a9-8a7c-4b26-b7c2-ba0e4053c042"). InnerVolumeSpecName "kube-api-access-5rsb4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:20:24.626108 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:24.626095    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/423cd7a9-8a7c-4b26-b7c2-ba0e4053c042-istio-token" pod "423cd7a9-8a7c-4b26-b7c2-ba0e4053c042" (UID: "423cd7a9-8a7c-4b26-b7c2-ba0e4053c042"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:20:24.626362 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:24.626336    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/423cd7a9-8a7c-4b26-b7c2-ba0e4053c042-istio-podinfo" pod "423cd7a9-8a7c-4b26-b7c2-ba0e4053c042" (UID: "423cd7a9-8a7c-4b26-b7c2-ba0e4053c042"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:20:24.626625 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:24.626607    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/423cd7a9-8a7c-4b26-b7c2-ba0e4053c042-istio-envoy" pod "423cd7a9-8a7c-4b26-b7c2-ba0e4053c042" (UID: "423cd7a9-8a7c-4b26-b7c2-ba0e4053c042"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:20:24.726295 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:24.726238    5002 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/423cd7a9-8a7c-4b26-b7c2-ba0e4053c042-istio-envoy\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:20:24.726295 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:24.726284    5002 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/423cd7a9-8a7c-4b26-b7c2-ba0e4053c042-istio-token\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:20:24.726295 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:24.726298    5002 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/423cd7a9-8a7c-4b26-b7c2-ba0e4053c042-istio-podinfo\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:20:24.726295 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:24.726310    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5rsb4\" (UniqueName: \"kubernetes.io/projected/423cd7a9-8a7c-4b26-b7c2-ba0e4053c042-kube-api-access-5rsb4\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:20:24.810154 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:24.810126    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f6d5b8-554bh"
Oct 30 01:20:24.810345 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:24.810172    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f6d5b8-554bh" event={"ID":"cd051518-6e55-4dd2-91a4-052386ed2328","Type":"ContainerDied","Data":"9c117899d64874dc6c54ece7b5fc76a9b0e70d2f48666dbb912432ca5083ae4d"}
Oct 30 01:20:24.810345 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:24.810223    5002 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9c117899d64874dc6c54ece7b5fc76a9b0e70d2f48666dbb912432ca5083ae4d"
Oct 30 01:20:24.812076 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:24.812049    5002 generic.go:356] "Generic (PLEG): container finished" podID="423cd7a9-8a7c-4b26-b7c2-ba0e4053c042" containerID="8c0c4062cdc81472706ee2c877619232632302e6cbf645e3f4e36bf61b11b94a" exitCode=0
Oct 30 01:20:24.812234 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:24.812122    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-9a8001-openshift-default-d885f8f7f-dt8r5"
Oct 30 01:20:24.812234 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:24.812178    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-9a8001-openshift-default-d885f8f7f-dt8r5" event={"ID":"423cd7a9-8a7c-4b26-b7c2-ba0e4053c042","Type":"ContainerDied","Data":"8c0c4062cdc81472706ee2c877619232632302e6cbf645e3f4e36bf61b11b94a"}
Oct 30 01:20:24.812395 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:24.812233    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-9a8001-openshift-default-d885f8f7f-dt8r5" event={"ID":"423cd7a9-8a7c-4b26-b7c2-ba0e4053c042","Type":"ContainerDied","Data":"f0c81b38a38643a44df636e1313ac85aac2babb0ed20be04ad28707cc73aae67"}
Oct 30 01:20:24.812395 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:24.812262    5002 scope.go:122] "RemoveContainer" containerID="8c0c4062cdc81472706ee2c877619232632302e6cbf645e3f4e36bf61b11b94a"
Oct 30 01:20:24.813837 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:24.813818    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-93079c-66kbl" event={"ID":"87a79eb3-d09c-4519-9c1a-fa8c7fdef2e6","Type":"ContainerDied","Data":"81b2d37217e3ca67fab1b0660b1551495fc6f37df76265a78c63dddf58049b9d"}
Oct 30 01:20:24.813930 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:24.813834    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-93079c-66kbl"
Oct 30 01:20:24.813930 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:24.813857    5002 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="81b2d37217e3ca67fab1b0660b1551495fc6f37df76265a78c63dddf58049b9d"
Oct 30 01:20:24.823629 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:24.823612    5002 scope.go:122] "RemoveContainer" containerID="f0c46b72fed307bda296524757f0270115e4ab84456c2250aa124fbb87458199"
Oct 30 01:20:24.833486 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:24.833465    5002 scope.go:122] "RemoveContainer" containerID="8c0c4062cdc81472706ee2c877619232632302e6cbf645e3f4e36bf61b11b94a"
Oct 30 01:20:24.833785 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:20:24.833764    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c0c4062cdc81472706ee2c877619232632302e6cbf645e3f4e36bf61b11b94a\": container with ID starting with 8c0c4062cdc81472706ee2c877619232632302e6cbf645e3f4e36bf61b11b94a not found: ID does not exist" containerID="8c0c4062cdc81472706ee2c877619232632302e6cbf645e3f4e36bf61b11b94a"
Oct 30 01:20:24.833863 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:24.833796    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c0c4062cdc81472706ee2c877619232632302e6cbf645e3f4e36bf61b11b94a"} err="failed to get container status \"8c0c4062cdc81472706ee2c877619232632302e6cbf645e3f4e36bf61b11b94a\": rpc error: code = NotFound desc = could not find container \"8c0c4062cdc81472706ee2c877619232632302e6cbf645e3f4e36bf61b11b94a\": container with ID starting with 8c0c4062cdc81472706ee2c877619232632302e6cbf645e3f4e36bf61b11b94a not found: ID does not exist"
Oct 30 01:20:24.833863 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:24.833823    5002 scope.go:122] "RemoveContainer" containerID="f0c46b72fed307bda296524757f0270115e4ab84456c2250aa124fbb87458199"
Oct 30 01:20:24.834105 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:20:24.834086    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0c46b72fed307bda296524757f0270115e4ab84456c2250aa124fbb87458199\": container with ID starting with f0c46b72fed307bda296524757f0270115e4ab84456c2250aa124fbb87458199 not found: ID does not exist" containerID="f0c46b72fed307bda296524757f0270115e4ab84456c2250aa124fbb87458199"
Oct 30 01:20:24.834155 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:24.834112    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0c46b72fed307bda296524757f0270115e4ab84456c2250aa124fbb87458199"} err="failed to get container status \"f0c46b72fed307bda296524757f0270115e4ab84456c2250aa124fbb87458199\": rpc error: code = NotFound desc = could not find container \"f0c46b72fed307bda296524757f0270115e4ab84456c2250aa124fbb87458199\": container with ID starting with f0c46b72fed307bda296524757f0270115e4ab84456c2250aa124fbb87458199 not found: ID does not exist"
Oct 30 01:20:24.875447 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:24.874478    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-9a8001-openshift-default-d885f8f7f-dt8r5"]
Oct 30 01:20:24.879221 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:24.879192    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-9a8001-openshift-default-d885f8f7f-dt8r5"]
Oct 30 01:20:25.649519 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:25.649470    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7d8z4"]
Oct 30 01:20:26.495364 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:26.495322    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="423cd7a9-8a7c-4b26-b7c2-ba0e4053c042" path="/var/lib/kubelet/pods/423cd7a9-8a7c-4b26-b7c2-ba0e4053c042/volumes"
Oct 30 01:20:27.619760 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:27.616542    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-a7ca1f-openshift-default-77b7bf58df-l8qlx"]
Oct 30 01:20:27.619760 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:27.616805    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-a7ca1f-openshift-default-77b7bf58df-l8qlx" podUID="6c75a860-08fd-4293-8c2d-9aa04e4348fa" containerName="istio-proxy" containerID="cri-o://7e12842879ac1d33d95b87234c9f4a05a54894efc661d754b0f43ad0562306a8" gracePeriod=30
Oct 30 01:20:28.110647 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:28.110604    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-cf4567-42nwr"]
Oct 30 01:20:28.111475 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:28.111451    5002 cpu_manager.go:414] "containerMap: removing container" podUID="87a79eb3-d09c-4519-9c1a-fa8c7fdef2e6" containerName="revoke-keys"
Oct 30 01:20:28.111475 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:28.111472    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87a79eb3-d09c-4519-9c1a-fa8c7fdef2e6" containerName="revoke-keys"
Oct 30 01:20:28.111695 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:28.111493    5002 cpu_manager.go:414] "containerMap: removing container" podUID="cd051518-6e55-4dd2-91a4-052386ed2328" containerName="revoke-keys"
Oct 30 01:20:28.111695 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:28.111498    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd051518-6e55-4dd2-91a4-052386ed2328" containerName="revoke-keys"
Oct 30 01:20:28.111695 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:28.111532    5002 cpu_manager.go:414] "containerMap: removing container" podUID="423cd7a9-8a7c-4b26-b7c2-ba0e4053c042" containerName="istio-proxy"
Oct 30 01:20:28.111695 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:28.111538    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="423cd7a9-8a7c-4b26-b7c2-ba0e4053c042" containerName="istio-proxy"
Oct 30 01:20:28.111695 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:28.111547    5002 cpu_manager.go:414] "containerMap: removing container" podUID="423cd7a9-8a7c-4b26-b7c2-ba0e4053c042" containerName="istio-proxy"
Oct 30 01:20:28.111695 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:28.111553    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="423cd7a9-8a7c-4b26-b7c2-ba0e4053c042" containerName="istio-proxy"
Oct 30 01:20:28.111695 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:28.111642    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="423cd7a9-8a7c-4b26-b7c2-ba0e4053c042" containerName="istio-proxy"
Oct 30 01:20:28.111695 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:28.111650    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="423cd7a9-8a7c-4b26-b7c2-ba0e4053c042" containerName="istio-proxy"
Oct 30 01:20:28.111695 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:28.111657    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="87a79eb3-d09c-4519-9c1a-fa8c7fdef2e6" containerName="revoke-keys"
Oct 30 01:20:28.111695 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:28.111665    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd051518-6e55-4dd2-91a4-052386ed2328" containerName="revoke-keys"
Oct 30 01:20:28.116805 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:28.116782    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-cf4567-42nwr"
Oct 30 01:20:28.135217 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:28.133515    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:20:28.135217 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:28.133647    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:20:28.135217 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:28.133974    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-cf4567-42nwr"]
Oct 30 01:20:28.181738 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:28.181688    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:20:28.183582 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:28.182119    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5phsh\""
Oct 30 01:20:28.310441 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:28.310225    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9a6030a-1f44-41f4-87a7-760d1c5ac463-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-cf4567-42nwr\" (UID: \"f9a6030a-1f44-41f4-87a7-760d1c5ac463\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-cf4567-42nwr"
Oct 30 01:20:28.315416 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:28.310930    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9a6030a-1f44-41f4-87a7-760d1c5ac463-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-cf4567-42nwr\" (UID: \"f9a6030a-1f44-41f4-87a7-760d1c5ac463\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-cf4567-42nwr"
Oct 30 01:20:28.315764 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:28.315731    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcdg7\" (UniqueName: \"kubernetes.io/projected/f9a6030a-1f44-41f4-87a7-760d1c5ac463-kube-api-access-kcdg7\") pod \"maas-api-revoke-keys-e2e-ipp-b-cf4567-42nwr\" (UID: \"f9a6030a-1f44-41f4-87a7-760d1c5ac463\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-cf4567-42nwr"
Oct 30 01:20:28.417124 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:28.417030    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcdg7\" (UniqueName: \"kubernetes.io/projected/f9a6030a-1f44-41f4-87a7-760d1c5ac463-kube-api-access-kcdg7\") pod \"maas-api-revoke-keys-e2e-ipp-b-cf4567-42nwr\" (UID: \"f9a6030a-1f44-41f4-87a7-760d1c5ac463\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-cf4567-42nwr"
Oct 30 01:20:28.417297 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:28.417145    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9a6030a-1f44-41f4-87a7-760d1c5ac463-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-cf4567-42nwr\" (UID: \"f9a6030a-1f44-41f4-87a7-760d1c5ac463\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-cf4567-42nwr"
Oct 30 01:20:28.417435 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:28.417314    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9a6030a-1f44-41f4-87a7-760d1c5ac463-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-cf4567-42nwr\" (UID: \"f9a6030a-1f44-41f4-87a7-760d1c5ac463\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-cf4567-42nwr"
Oct 30 01:20:28.417946 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:28.417922    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9a6030a-1f44-41f4-87a7-760d1c5ac463-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-cf4567-42nwr\" (UID: \"f9a6030a-1f44-41f4-87a7-760d1c5ac463\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-cf4567-42nwr"
Oct 30 01:20:28.420846 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:28.420821    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9a6030a-1f44-41f4-87a7-760d1c5ac463-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-cf4567-42nwr\" (UID: \"f9a6030a-1f44-41f4-87a7-760d1c5ac463\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-cf4567-42nwr"
Oct 30 01:20:28.441562 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:28.441539    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcdg7\" (UniqueName: \"kubernetes.io/projected/f9a6030a-1f44-41f4-87a7-760d1c5ac463-kube-api-access-kcdg7\") pod \"maas-api-revoke-keys-e2e-ipp-b-cf4567-42nwr\" (UID: \"f9a6030a-1f44-41f4-87a7-760d1c5ac463\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-cf4567-42nwr"
Oct 30 01:20:28.444248 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:28.444215    5002 patch_prober.go:29] interesting pod/e2e-ipp-a-a7ca1f-openshift-default-77b7bf58df-l8qlx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 01:20:28.444370 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:28.444269    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-a7ca1f-openshift-default-77b7bf58df-l8qlx" podUID="6c75a860-08fd-4293-8c2d-9aa04e4348fa" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 01:20:28.732162 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:28.732126    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-cf4567-42nwr"
Oct 30 01:20:29.081288 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:29.081259    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-cf4567-42nwr"]
Oct 30 01:20:29.084070 ip-10-0-128-7 kubenswrapper[5002]: W1030 01:20:29.084002    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9a6030a_1f44_41f4_87a7_760d1c5ac463.slice/crio-443e2622e765a221f1fffa1a3951693d680acb30a9a75968de0f0bed3bf4a4fc WatchSource:0}: Error finding container 443e2622e765a221f1fffa1a3951693d680acb30a9a75968de0f0bed3bf4a4fc: Status 404 returned error can't find the container with id 443e2622e765a221f1fffa1a3951693d680acb30a9a75968de0f0bed3bf4a4fc
Oct 30 01:20:29.936558 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:29.935915    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-cf4567-42nwr" event={"ID":"f9a6030a-1f44-41f4-87a7-760d1c5ac463","Type":"ContainerStarted","Data":"11d649d9fa32c403be495fce07741e2c3988d49744e247bd10a44082bbbd99bb"}
Oct 30 01:20:29.936558 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:29.935974    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-cf4567-42nwr" event={"ID":"f9a6030a-1f44-41f4-87a7-760d1c5ac463","Type":"ContainerStarted","Data":"443e2622e765a221f1fffa1a3951693d680acb30a9a75968de0f0bed3bf4a4fc"}
Oct 30 01:20:29.990875 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:29.988373    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-cf4567-42nwr" podStartSLOduration=1.988352806 podStartE2EDuration="1.988352806s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:20:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:20:29.985368585 +0000 UTC m=+2925.601801493" watchObservedRunningTime="2026-10-30 01:20:29.988352806 +0000 UTC m=+2925.604785687"
Oct 30 01:20:31.021416 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:31.017638    5002 generic.go:356] "Generic (PLEG): container finished" podID="f9a6030a-1f44-41f4-87a7-760d1c5ac463" containerID="11d649d9fa32c403be495fce07741e2c3988d49744e247bd10a44082bbbd99bb" exitCode=0
Oct 30 01:20:31.021416 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:31.017717    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-cf4567-42nwr" event={"ID":"f9a6030a-1f44-41f4-87a7-760d1c5ac463","Type":"ContainerDied","Data":"11d649d9fa32c403be495fce07741e2c3988d49744e247bd10a44082bbbd99bb"}
Oct 30 01:20:32.291145 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:32.291075    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-cf4567-42nwr"
Oct 30 01:20:32.408838 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:32.408801    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9a6030a-1f44-41f4-87a7-760d1c5ac463-kube-api-access-kcdg7\" (UniqueName: \"kubernetes.io/projected/f9a6030a-1f44-41f4-87a7-760d1c5ac463-kube-api-access-kcdg7\") pod \"f9a6030a-1f44-41f4-87a7-760d1c5ac463\" (UID: \"f9a6030a-1f44-41f4-87a7-760d1c5ac463\") "
Oct 30 01:20:32.409009 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:32.408857    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f9a6030a-1f44-41f4-87a7-760d1c5ac463-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9a6030a-1f44-41f4-87a7-760d1c5ac463-maas-api-xks-ca\") pod \"f9a6030a-1f44-41f4-87a7-760d1c5ac463\" (UID: \"f9a6030a-1f44-41f4-87a7-760d1c5ac463\") "
Oct 30 01:20:32.409009 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:32.408915    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9a6030a-1f44-41f4-87a7-760d1c5ac463-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9a6030a-1f44-41f4-87a7-760d1c5ac463-maas-api-openshift-ca\") pod \"f9a6030a-1f44-41f4-87a7-760d1c5ac463\" (UID: \"f9a6030a-1f44-41f4-87a7-760d1c5ac463\") "
Oct 30 01:20:32.410360 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:32.410306    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9a6030a-1f44-41f4-87a7-760d1c5ac463-maas-api-openshift-ca" pod "f9a6030a-1f44-41f4-87a7-760d1c5ac463" (UID: "f9a6030a-1f44-41f4-87a7-760d1c5ac463"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:20:32.412190 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:32.412159    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9a6030a-1f44-41f4-87a7-760d1c5ac463-maas-api-xks-ca" pod "f9a6030a-1f44-41f4-87a7-760d1c5ac463" (UID: "f9a6030a-1f44-41f4-87a7-760d1c5ac463"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:20:32.413291 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:32.413264    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9a6030a-1f44-41f4-87a7-760d1c5ac463-kube-api-access-kcdg7" pod "f9a6030a-1f44-41f4-87a7-760d1c5ac463" (UID: "f9a6030a-1f44-41f4-87a7-760d1c5ac463"). InnerVolumeSpecName "kube-api-access-kcdg7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:20:32.510968 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:32.510936    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kcdg7\" (UniqueName: \"kubernetes.io/projected/f9a6030a-1f44-41f4-87a7-760d1c5ac463-kube-api-access-kcdg7\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:20:32.511117 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:32.510972    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9a6030a-1f44-41f4-87a7-760d1c5ac463-maas-api-xks-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:20:32.511117 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:32.510987    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9a6030a-1f44-41f4-87a7-760d1c5ac463-maas-api-openshift-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:20:32.987261 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:32.987224    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-a7ca1f-openshift-default-77b7bf58df-l8qlx"
Oct 30 01:20:33.018059 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:33.018014    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c75a860-08fd-4293-8c2d-9aa04e4348fa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c75a860-08fd-4293-8c2d-9aa04e4348fa-workload-socket\") pod \"6c75a860-08fd-4293-8c2d-9aa04e4348fa\" (UID: \"6c75a860-08fd-4293-8c2d-9aa04e4348fa\") "
Oct 30 01:20:33.018059 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:33.018055    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c75a860-08fd-4293-8c2d-9aa04e4348fa-kube-api-access-lt4jg\" (UniqueName: \"kubernetes.io/projected/6c75a860-08fd-4293-8c2d-9aa04e4348fa-kube-api-access-lt4jg\") pod \"6c75a860-08fd-4293-8c2d-9aa04e4348fa\" (UID: \"6c75a860-08fd-4293-8c2d-9aa04e4348fa\") "
Oct 30 01:20:33.018375 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:33.018077    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c75a860-08fd-4293-8c2d-9aa04e4348fa-istio-token\" (UniqueName: \"kubernetes.io/projected/6c75a860-08fd-4293-8c2d-9aa04e4348fa-istio-token\") pod \"6c75a860-08fd-4293-8c2d-9aa04e4348fa\" (UID: \"6c75a860-08fd-4293-8c2d-9aa04e4348fa\") "
Oct 30 01:20:33.018375 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:33.018209    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6c75a860-08fd-4293-8c2d-9aa04e4348fa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c75a860-08fd-4293-8c2d-9aa04e4348fa-istio-podinfo\") pod \"6c75a860-08fd-4293-8c2d-9aa04e4348fa\" (UID: \"6c75a860-08fd-4293-8c2d-9aa04e4348fa\") "
Oct 30 01:20:33.018375 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:33.018277    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c75a860-08fd-4293-8c2d-9aa04e4348fa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c75a860-08fd-4293-8c2d-9aa04e4348fa-istiod-ca-cert\") pod \"6c75a860-08fd-4293-8c2d-9aa04e4348fa\" (UID: \"6c75a860-08fd-4293-8c2d-9aa04e4348fa\") "
Oct 30 01:20:33.018375 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:33.018350    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c75a860-08fd-4293-8c2d-9aa04e4348fa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c75a860-08fd-4293-8c2d-9aa04e4348fa-workload-certs\") pod \"6c75a860-08fd-4293-8c2d-9aa04e4348fa\" (UID: \"6c75a860-08fd-4293-8c2d-9aa04e4348fa\") "
Oct 30 01:20:33.019020 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:33.018449    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c75a860-08fd-4293-8c2d-9aa04e4348fa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c75a860-08fd-4293-8c2d-9aa04e4348fa-credential-socket\") pod \"6c75a860-08fd-4293-8c2d-9aa04e4348fa\" (UID: \"6c75a860-08fd-4293-8c2d-9aa04e4348fa\") "
Oct 30 01:20:33.019020 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:33.018484    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c75a860-08fd-4293-8c2d-9aa04e4348fa-workload-socket" pod "6c75a860-08fd-4293-8c2d-9aa04e4348fa" (UID: "6c75a860-08fd-4293-8c2d-9aa04e4348fa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:20:33.019020 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:33.018532    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c75a860-08fd-4293-8c2d-9aa04e4348fa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c75a860-08fd-4293-8c2d-9aa04e4348fa-istio-envoy\") pod \"6c75a860-08fd-4293-8c2d-9aa04e4348fa\" (UID: \"6c75a860-08fd-4293-8c2d-9aa04e4348fa\") "
Oct 30 01:20:33.019020 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:33.018582    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c75a860-08fd-4293-8c2d-9aa04e4348fa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c75a860-08fd-4293-8c2d-9aa04e4348fa-istio-data\") pod \"6c75a860-08fd-4293-8c2d-9aa04e4348fa\" (UID: \"6c75a860-08fd-4293-8c2d-9aa04e4348fa\") "
Oct 30 01:20:33.019020 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:33.018715    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c75a860-08fd-4293-8c2d-9aa04e4348fa-workload-certs" pod "6c75a860-08fd-4293-8c2d-9aa04e4348fa" (UID: "6c75a860-08fd-4293-8c2d-9aa04e4348fa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:20:33.019311 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:33.019086    5002 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c75a860-08fd-4293-8c2d-9aa04e4348fa-workload-socket\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:20:33.019311 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:33.019093    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c75a860-08fd-4293-8c2d-9aa04e4348fa-istiod-ca-cert" pod "6c75a860-08fd-4293-8c2d-9aa04e4348fa" (UID: "6c75a860-08fd-4293-8c2d-9aa04e4348fa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:20:33.019311 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:33.019105    5002 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c75a860-08fd-4293-8c2d-9aa04e4348fa-workload-certs\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:20:33.019311 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:33.019282    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c75a860-08fd-4293-8c2d-9aa04e4348fa-istio-data" pod "6c75a860-08fd-4293-8c2d-9aa04e4348fa" (UID: "6c75a860-08fd-4293-8c2d-9aa04e4348fa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:20:33.019631 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:33.019542    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c75a860-08fd-4293-8c2d-9aa04e4348fa-credential-socket" pod "6c75a860-08fd-4293-8c2d-9aa04e4348fa" (UID: "6c75a860-08fd-4293-8c2d-9aa04e4348fa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:20:33.021874 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:33.021840    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6c75a860-08fd-4293-8c2d-9aa04e4348fa-istio-podinfo" pod "6c75a860-08fd-4293-8c2d-9aa04e4348fa" (UID: "6c75a860-08fd-4293-8c2d-9aa04e4348fa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:20:33.021995 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:33.021910    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c75a860-08fd-4293-8c2d-9aa04e4348fa-istio-envoy" pod "6c75a860-08fd-4293-8c2d-9aa04e4348fa" (UID: "6c75a860-08fd-4293-8c2d-9aa04e4348fa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:20:33.021995 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:33.021940    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c75a860-08fd-4293-8c2d-9aa04e4348fa-kube-api-access-lt4jg" pod "6c75a860-08fd-4293-8c2d-9aa04e4348fa" (UID: "6c75a860-08fd-4293-8c2d-9aa04e4348fa"). InnerVolumeSpecName "kube-api-access-lt4jg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:20:33.022273 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:33.022255    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c75a860-08fd-4293-8c2d-9aa04e4348fa-istio-token" pod "6c75a860-08fd-4293-8c2d-9aa04e4348fa" (UID: "6c75a860-08fd-4293-8c2d-9aa04e4348fa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:20:33.040039 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:33.037947    5002 generic.go:356] "Generic (PLEG): container finished" podID="6c75a860-08fd-4293-8c2d-9aa04e4348fa" containerID="7e12842879ac1d33d95b87234c9f4a05a54894efc661d754b0f43ad0562306a8" exitCode=0
Oct 30 01:20:33.040039 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:33.038282    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-a7ca1f-openshift-default-77b7bf58df-l8qlx"
Oct 30 01:20:33.040039 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:33.039111    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a7ca1f-openshift-default-77b7bf58df-l8qlx" event={"ID":"6c75a860-08fd-4293-8c2d-9aa04e4348fa","Type":"ContainerDied","Data":"7e12842879ac1d33d95b87234c9f4a05a54894efc661d754b0f43ad0562306a8"}
Oct 30 01:20:33.040039 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:33.039171    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a7ca1f-openshift-default-77b7bf58df-l8qlx" event={"ID":"6c75a860-08fd-4293-8c2d-9aa04e4348fa","Type":"ContainerDied","Data":"c24b9f1466f945b4fc655d99f4e2ae77ee4d6c8cb51c2e31a96a410d030fa1da"}
Oct 30 01:20:33.040039 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:33.039200    5002 scope.go:122] "RemoveContainer" containerID="7e12842879ac1d33d95b87234c9f4a05a54894efc661d754b0f43ad0562306a8"
Oct 30 01:20:33.042315 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:33.041451    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-cf4567-42nwr" event={"ID":"f9a6030a-1f44-41f4-87a7-760d1c5ac463","Type":"ContainerDied","Data":"443e2622e765a221f1fffa1a3951693d680acb30a9a75968de0f0bed3bf4a4fc"}
Oct 30 01:20:33.042315 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:33.041501    5002 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="443e2622e765a221f1fffa1a3951693d680acb30a9a75968de0f0bed3bf4a4fc"
Oct 30 01:20:33.042315 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:33.041564    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-cf4567-42nwr"
Oct 30 01:20:33.061919 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:33.061570    5002 scope.go:122] "RemoveContainer" containerID="7e12842879ac1d33d95b87234c9f4a05a54894efc661d754b0f43ad0562306a8"
Oct 30 01:20:33.062046 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:20:33.061928    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e12842879ac1d33d95b87234c9f4a05a54894efc661d754b0f43ad0562306a8\": container with ID starting with 7e12842879ac1d33d95b87234c9f4a05a54894efc661d754b0f43ad0562306a8 not found: ID does not exist" containerID="7e12842879ac1d33d95b87234c9f4a05a54894efc661d754b0f43ad0562306a8"
Oct 30 01:20:33.062046 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:33.061986    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e12842879ac1d33d95b87234c9f4a05a54894efc661d754b0f43ad0562306a8"} err="failed to get container status \"7e12842879ac1d33d95b87234c9f4a05a54894efc661d754b0f43ad0562306a8\": rpc error: code = NotFound desc = could not find container \"7e12842879ac1d33d95b87234c9f4a05a54894efc661d754b0f43ad0562306a8\": container with ID starting with 7e12842879ac1d33d95b87234c9f4a05a54894efc661d754b0f43ad0562306a8 not found: ID does not exist"
Oct 30 01:20:33.105418 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:33.105348    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-a7ca1f-openshift-default-77b7bf58df-l8qlx"]
Oct 30 01:20:33.109217 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:33.109187    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-a7ca1f-openshift-default-77b7bf58df-l8qlx"]
Oct 30 01:20:33.120656 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:33.120621    5002 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c75a860-08fd-4293-8c2d-9aa04e4348fa-istiod-ca-cert\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:20:33.120656 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:33.120649    5002 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c75a860-08fd-4293-8c2d-9aa04e4348fa-credential-socket\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:20:33.120656 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:33.120659    5002 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c75a860-08fd-4293-8c2d-9aa04e4348fa-istio-envoy\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:20:33.120863 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:33.120669    5002 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c75a860-08fd-4293-8c2d-9aa04e4348fa-istio-data\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:20:33.120863 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:33.120677    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lt4jg\" (UniqueName: \"kubernetes.io/projected/6c75a860-08fd-4293-8c2d-9aa04e4348fa-kube-api-access-lt4jg\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:20:33.120863 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:33.120687    5002 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c75a860-08fd-4293-8c2d-9aa04e4348fa-istio-token\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:20:33.120863 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:33.120697    5002 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c75a860-08fd-4293-8c2d-9aa04e4348fa-istio-podinfo\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:20:34.529677 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:34.529628    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c75a860-08fd-4293-8c2d-9aa04e4348fa" path="/var/lib/kubelet/pods/6c75a860-08fd-4293-8c2d-9aa04e4348fa/volumes"
Oct 30 01:20:37.083187 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:37.083148    5002 generic.go:356] "Generic (PLEG): container finished" podID="e09703cf-6ee3-45e9-893d-5a626ca636c9" containerID="4ffab9d9a6fd83e8b2639aaf218cbf94607909f3ff720ba0d2aa6083b0d37bcd" exitCode=0
Oct 30 01:20:37.083835 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:37.083200    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-1e8c0e-openshift-default-6fb676f64f-ld62q" event={"ID":"e09703cf-6ee3-45e9-893d-5a626ca636c9","Type":"ContainerDied","Data":"4ffab9d9a6fd83e8b2639aaf218cbf94607909f3ff720ba0d2aa6083b0d37bcd"}
Oct 30 01:20:37.083835 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:37.083765    5002 scope.go:122] "RemoveContainer" containerID="4ffab9d9a6fd83e8b2639aaf218cbf94607909f3ff720ba0d2aa6083b0d37bcd"
Oct 30 01:20:37.085992 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:37.085954    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:20:37.086126 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:37.086052    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:20:37.086126 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:37.086104    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:20:38.092932 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:38.092891    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-1e8c0e-openshift-default-6fb676f64f-ld62q" event={"ID":"e09703cf-6ee3-45e9-893d-5a626ca636c9","Type":"ContainerStarted","Data":"7efdd2fa7f3ce097256c5cf2e582e81107eea9a013d1a6e703608ecd06c063b6"}
Oct 30 01:20:38.116570 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:38.116537    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-1e8c0e-openshift-default-6fb676f64f-ld62q"
Oct 30 01:20:38.116829 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:38.116805    5002 patch_prober.go:29] interesting pod/e2e-ipp-b-1e8c0e-openshift-default-6fb676f64f-ld62q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" start-of-body=
Oct 30 01:20:38.116927 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:38.116861    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-1e8c0e-openshift-default-6fb676f64f-ld62q" podUID="e09703cf-6ee3-45e9-893d-5a626ca636c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused"
Oct 30 01:20:39.117087 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:39.117048    5002 patch_prober.go:29] interesting pod/e2e-ipp-b-1e8c0e-openshift-default-6fb676f64f-ld62q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" start-of-body=
Oct 30 01:20:39.117534 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:39.117111    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-1e8c0e-openshift-default-6fb676f64f-ld62q" podUID="e09703cf-6ee3-45e9-893d-5a626ca636c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused"
Oct 30 01:20:40.116205 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:40.116165    5002 patch_prober.go:29] interesting pod/e2e-ipp-b-1e8c0e-openshift-default-6fb676f64f-ld62q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" start-of-body=
Oct 30 01:20:40.116425 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:40.116226    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-1e8c0e-openshift-default-6fb676f64f-ld62q" podUID="e09703cf-6ee3-45e9-893d-5a626ca636c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused"
Oct 30 01:20:41.116059 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:41.116020    5002 patch_prober.go:29] interesting pod/e2e-ipp-b-1e8c0e-openshift-default-6fb676f64f-ld62q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" start-of-body=
Oct 30 01:20:41.116497 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:41.116075    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-1e8c0e-openshift-default-6fb676f64f-ld62q" podUID="e09703cf-6ee3-45e9-893d-5a626ca636c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused"
Oct 30 01:20:42.121942 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:42.121893    5002 patch_prober.go:29] interesting pod/e2e-ipp-b-1e8c0e-openshift-default-6fb676f64f-ld62q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" start-of-body=
Oct 30 01:20:42.122624 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:42.122581    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-1e8c0e-openshift-default-6fb676f64f-ld62q" podUID="e09703cf-6ee3-45e9-893d-5a626ca636c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused"
Oct 30 01:20:42.886892 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:20:42.886847    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbjdq" containerName="extractor"
Oct 30 01:20:43.117034 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:43.116992    5002 patch_prober.go:29] interesting pod/e2e-ipp-b-1e8c0e-openshift-default-6fb676f64f-ld62q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" start-of-body=
Oct 30 01:20:43.117242 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:43.117058    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-1e8c0e-openshift-default-6fb676f64f-ld62q" podUID="e09703cf-6ee3-45e9-893d-5a626ca636c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused"
Oct 30 01:20:44.121894 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:44.121859    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-1e8c0e-openshift-default-6fb676f64f-ld62q"
Oct 30 01:20:44.122433 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:44.122247    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-1e8c0e-openshift-default-6fb676f64f-ld62q"
Oct 30 01:20:44.123060 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:44.123037    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-1e8c0e-openshift-default-6fb676f64f-ld62q"
Oct 30 01:20:49.346305 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:49.346269    5002 scope.go:122] "RemoveContainer" containerID="5b6f7ddf81803ecf23ca5464b4be2d0086aed004b40663c0f8cb3a03d41cb044"
Oct 30 01:20:49.358092 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:49.358066    5002 scope.go:122] "RemoveContainer" containerID="b083d3a73415f8eeb1babf12286851cdf1b73235d006997fd24cc85d1aa70108"
Oct 30 01:20:49.370822 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:49.370792    5002 scope.go:122] "RemoveContainer" containerID="89d1fba3da05502b8b86752b306d87bf56e3e9a31b826b611ff0a10fc08848b2"
Oct 30 01:20:49.380782 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:49.380759    5002 scope.go:122] "RemoveContainer" containerID="bd67c8dfee363796e43f186d9a92e2678340c2074e57cf02667b49a4098641b2"
Oct 30 01:20:49.391270 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:49.391250    5002 scope.go:122] "RemoveContainer" containerID="214aab239baab3265c67e5caf3e6ff8f8957e72b6d9139c2dbfc0c2ee91a0ec7"
Oct 30 01:20:49.401446 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:49.401427    5002 scope.go:122] "RemoveContainer" containerID="1b86f6a042ede8c72ad2e2102fed1823cc856adb9a5bde84160e03eea02ed319"
Oct 30 01:20:49.411186 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:49.411160    5002 scope.go:122] "RemoveContainer" containerID="a7d528c6408b9ba5eabd8cd05c298d1ddbf23c67f108c84f3be409838ab78c36"
Oct 30 01:20:50.009317 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:50.009279    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-1e8c0e-857cdc4678-hxkmw"]
Oct 30 01:20:53.525245 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:53.525204    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-93079c-66kbl"]
Oct 30 01:20:53.539203 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:53.539158    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-93079c-66kbl"]
Oct 30 01:20:54.496663 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:54.496618    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87a79eb3-d09c-4519-9c1a-fa8c7fdef2e6" path="/var/lib/kubelet/pods/87a79eb3-d09c-4519-9c1a-fa8c7fdef2e6/volumes"
Oct 30 01:20:55.010516 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:55.010444    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-1e8c0e-857cdc4678-hxkmw" podUID="7178b0c4-ce99-4aa0-8c35-bddc5936b53f" containerName="payload-processing" containerID="cri-o://e768837311e3c1b3e946fa09062d99042cf865c3da507190f2c95bd2387e657e" gracePeriod=25
Oct 30 01:20:55.264948 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:55.264872    5002 generic.go:356] "Generic (PLEG): container finished" podID="7178b0c4-ce99-4aa0-8c35-bddc5936b53f" containerID="e768837311e3c1b3e946fa09062d99042cf865c3da507190f2c95bd2387e657e" exitCode=0
Oct 30 01:20:55.265100 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:55.264951    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-1e8c0e-857cdc4678-hxkmw" event={"ID":"7178b0c4-ce99-4aa0-8c35-bddc5936b53f","Type":"ContainerDied","Data":"e768837311e3c1b3e946fa09062d99042cf865c3da507190f2c95bd2387e657e"}
Oct 30 01:20:55.300541 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:55.300516    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-1e8c0e-857cdc4678-hxkmw"
Oct 30 01:20:55.404168 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:55.404128    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7178b0c4-ce99-4aa0-8c35-bddc5936b53f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7178b0c4-ce99-4aa0-8c35-bddc5936b53f-plugins-config-volume\") pod \"7178b0c4-ce99-4aa0-8c35-bddc5936b53f\" (UID: \"7178b0c4-ce99-4aa0-8c35-bddc5936b53f\") "
Oct 30 01:20:55.404409 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:55.404238    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7178b0c4-ce99-4aa0-8c35-bddc5936b53f-kube-api-access-jrfvg\" (UniqueName: \"kubernetes.io/projected/7178b0c4-ce99-4aa0-8c35-bddc5936b53f-kube-api-access-jrfvg\") pod \"7178b0c4-ce99-4aa0-8c35-bddc5936b53f\" (UID: \"7178b0c4-ce99-4aa0-8c35-bddc5936b53f\") "
Oct 30 01:20:55.404606 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:55.404579    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7178b0c4-ce99-4aa0-8c35-bddc5936b53f-plugins-config-volume" pod "7178b0c4-ce99-4aa0-8c35-bddc5936b53f" (UID: "7178b0c4-ce99-4aa0-8c35-bddc5936b53f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:20:55.405039 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:55.405017    5002 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7178b0c4-ce99-4aa0-8c35-bddc5936b53f-plugins-config-volume\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:20:55.406801 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:55.406773    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7178b0c4-ce99-4aa0-8c35-bddc5936b53f-kube-api-access-jrfvg" pod "7178b0c4-ce99-4aa0-8c35-bddc5936b53f" (UID: "7178b0c4-ce99-4aa0-8c35-bddc5936b53f"). InnerVolumeSpecName "kube-api-access-jrfvg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:20:55.506031 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:55.505994    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jrfvg\" (UniqueName: \"kubernetes.io/projected/7178b0c4-ce99-4aa0-8c35-bddc5936b53f-kube-api-access-jrfvg\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:20:56.274085 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:56.274051    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-1e8c0e-857cdc4678-hxkmw"
Oct 30 01:20:56.274085 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:56.274073    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-1e8c0e-857cdc4678-hxkmw" event={"ID":"7178b0c4-ce99-4aa0-8c35-bddc5936b53f","Type":"ContainerDied","Data":"15bea8aa7dc6f8b0a69973aaf07918ea8af2a3a76d5ab76bb7fade0b8e7dda81"}
Oct 30 01:20:56.274651 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:56.274140    5002 scope.go:122] "RemoveContainer" containerID="e768837311e3c1b3e946fa09062d99042cf865c3da507190f2c95bd2387e657e"
Oct 30 01:20:56.331962 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:56.331926    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-1e8c0e-857cdc4678-hxkmw"]
Oct 30 01:20:56.346186 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:56.346149    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-1e8c0e-857cdc4678-hxkmw"]
Oct 30 01:20:56.502534 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:20:56.502496    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7178b0c4-ce99-4aa0-8c35-bddc5936b53f" path="/var/lib/kubelet/pods/7178b0c4-ce99-4aa0-8c35-bddc5936b53f/volumes"
Oct 30 01:21:01.482525 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:01.482480    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-a5e9ab-m2f7j"]
Oct 30 01:21:01.483087 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:01.483072    5002 cpu_manager.go:414] "containerMap: removing container" podUID="7178b0c4-ce99-4aa0-8c35-bddc5936b53f" containerName="payload-processing"
Oct 30 01:21:01.483087 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:01.483087    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7178b0c4-ce99-4aa0-8c35-bddc5936b53f" containerName="payload-processing"
Oct 30 01:21:01.483172 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:01.483118    5002 cpu_manager.go:414] "containerMap: removing container" podUID="6c75a860-08fd-4293-8c2d-9aa04e4348fa" containerName="istio-proxy"
Oct 30 01:21:01.483172 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:01.483124    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c75a860-08fd-4293-8c2d-9aa04e4348fa" containerName="istio-proxy"
Oct 30 01:21:01.483172 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:01.483136    5002 cpu_manager.go:414] "containerMap: removing container" podUID="f9a6030a-1f44-41f4-87a7-760d1c5ac463" containerName="revoke-keys"
Oct 30 01:21:01.483172 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:01.483141    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9a6030a-1f44-41f4-87a7-760d1c5ac463" containerName="revoke-keys"
Oct 30 01:21:01.483319 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:01.483221    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c75a860-08fd-4293-8c2d-9aa04e4348fa" containerName="istio-proxy"
Oct 30 01:21:01.483319 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:01.483231    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9a6030a-1f44-41f4-87a7-760d1c5ac463" containerName="revoke-keys"
Oct 30 01:21:01.483319 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:01.483241    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="7178b0c4-ce99-4aa0-8c35-bddc5936b53f" containerName="payload-processing"
Oct 30 01:21:01.488323 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:01.488294    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-a5e9ab-m2f7j"
Oct 30 01:21:01.504840 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:01.504815    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:21:01.506958 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:01.506931    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-a5e9ab-m2f7j"]
Oct 30 01:21:01.515822 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:01.515789    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:21:01.519658 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:01.519634    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5phsh\""
Oct 30 01:21:01.519658 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:01.519634    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:21:01.623159 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:01.623125    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nz9s\" (UniqueName: \"kubernetes.io/projected/f556c088-edfc-4b0e-bf11-8568691d46a4-kube-api-access-9nz9s\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-a5e9ab-m2f7j\" (UID: \"f556c088-edfc-4b0e-bf11-8568691d46a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-a5e9ab-m2f7j"
Oct 30 01:21:01.623393 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:01.623283    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f556c088-edfc-4b0e-bf11-8568691d46a4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-a5e9ab-m2f7j\" (UID: \"f556c088-edfc-4b0e-bf11-8568691d46a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-a5e9ab-m2f7j"
Oct 30 01:21:01.623393 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:01.623319    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f556c088-edfc-4b0e-bf11-8568691d46a4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-a5e9ab-m2f7j\" (UID: \"f556c088-edfc-4b0e-bf11-8568691d46a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-a5e9ab-m2f7j"
Oct 30 01:21:01.724531 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:01.724491    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nz9s\" (UniqueName: \"kubernetes.io/projected/f556c088-edfc-4b0e-bf11-8568691d46a4-kube-api-access-9nz9s\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-a5e9ab-m2f7j\" (UID: \"f556c088-edfc-4b0e-bf11-8568691d46a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-a5e9ab-m2f7j"
Oct 30 01:21:01.724723 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:01.724612    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f556c088-edfc-4b0e-bf11-8568691d46a4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-a5e9ab-m2f7j\" (UID: \"f556c088-edfc-4b0e-bf11-8568691d46a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-a5e9ab-m2f7j"
Oct 30 01:21:01.724723 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:01.724634    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f556c088-edfc-4b0e-bf11-8568691d46a4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-a5e9ab-m2f7j\" (UID: \"f556c088-edfc-4b0e-bf11-8568691d46a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-a5e9ab-m2f7j"
Oct 30 01:21:01.725448 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:01.725421    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f556c088-edfc-4b0e-bf11-8568691d46a4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-a5e9ab-m2f7j\" (UID: \"f556c088-edfc-4b0e-bf11-8568691d46a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-a5e9ab-m2f7j"
Oct 30 01:21:01.727067 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:01.727047    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f556c088-edfc-4b0e-bf11-8568691d46a4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-a5e9ab-m2f7j\" (UID: \"f556c088-edfc-4b0e-bf11-8568691d46a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-a5e9ab-m2f7j"
Oct 30 01:21:01.749656 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:01.749582    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nz9s\" (UniqueName: \"kubernetes.io/projected/f556c088-edfc-4b0e-bf11-8568691d46a4-kube-api-access-9nz9s\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-a5e9ab-m2f7j\" (UID: \"f556c088-edfc-4b0e-bf11-8568691d46a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-a5e9ab-m2f7j"
Oct 30 01:21:01.802653 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:01.802617    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-a5e9ab-m2f7j"
Oct 30 01:21:02.110417 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:02.110366    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-a5e9ab-m2f7j"]
Oct 30 01:21:02.110729 ip-10-0-128-7 kubenswrapper[5002]: W1030 01:21:02.110690    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf556c088_edfc_4b0e_bf11_8568691d46a4.slice/crio-0f54621a37e5bcfcd7ae06ff3f39b4685bd8d0f9c72d45a2aeb7fc05c7481e2d WatchSource:0}: Error finding container 0f54621a37e5bcfcd7ae06ff3f39b4685bd8d0f9c72d45a2aeb7fc05c7481e2d: Status 404 returned error can't find the container with id 0f54621a37e5bcfcd7ae06ff3f39b4685bd8d0f9c72d45a2aeb7fc05c7481e2d
Oct 30 01:21:02.323085 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:02.323034    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-a5e9ab-m2f7j" event={"ID":"f556c088-edfc-4b0e-bf11-8568691d46a4","Type":"ContainerStarted","Data":"0f54621a37e5bcfcd7ae06ff3f39b4685bd8d0f9c72d45a2aeb7fc05c7481e2d"}
Oct 30 01:21:02.907913 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:02.907810    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f6d5b8-554bh"]
Oct 30 01:21:02.910827 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:02.910803    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f6d5b8-554bh"]
Oct 30 01:21:03.331475 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:03.331436    5002 generic.go:356] "Generic (PLEG): container finished" podID="f556c088-edfc-4b0e-bf11-8568691d46a4" containerID="d427548acd3258707e85ee7c7ba1a0c29897dfd2c94184abf2c0fc9dc3cc9bc6" exitCode=0
Oct 30 01:21:03.331645 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:03.331527    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-a5e9ab-m2f7j" event={"ID":"f556c088-edfc-4b0e-bf11-8568691d46a4","Type":"ContainerDied","Data":"d427548acd3258707e85ee7c7ba1a0c29897dfd2c94184abf2c0fc9dc3cc9bc6"}
Oct 30 01:21:04.499046 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:04.499014    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd051518-6e55-4dd2-91a4-052386ed2328" path="/var/lib/kubelet/pods/cd051518-6e55-4dd2-91a4-052386ed2328/volumes"
Oct 30 01:21:04.500568 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:04.500551    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-a5e9ab-m2f7j"
Oct 30 01:21:04.578165 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:04.578126    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f556c088-edfc-4b0e-bf11-8568691d46a4-kube-api-access-9nz9s\" (UniqueName: \"kubernetes.io/projected/f556c088-edfc-4b0e-bf11-8568691d46a4-kube-api-access-9nz9s\") pod \"f556c088-edfc-4b0e-bf11-8568691d46a4\" (UID: \"f556c088-edfc-4b0e-bf11-8568691d46a4\") "
Oct 30 01:21:04.578869 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:04.578838    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f556c088-edfc-4b0e-bf11-8568691d46a4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f556c088-edfc-4b0e-bf11-8568691d46a4-maas-api-xks-ca\") pod \"f556c088-edfc-4b0e-bf11-8568691d46a4\" (UID: \"f556c088-edfc-4b0e-bf11-8568691d46a4\") "
Oct 30 01:21:04.579029 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:04.578893    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f556c088-edfc-4b0e-bf11-8568691d46a4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f556c088-edfc-4b0e-bf11-8568691d46a4-maas-api-openshift-ca\") pod \"f556c088-edfc-4b0e-bf11-8568691d46a4\" (UID: \"f556c088-edfc-4b0e-bf11-8568691d46a4\") "
Oct 30 01:21:04.579855 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:04.579812    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f556c088-edfc-4b0e-bf11-8568691d46a4-maas-api-openshift-ca" pod "f556c088-edfc-4b0e-bf11-8568691d46a4" (UID: "f556c088-edfc-4b0e-bf11-8568691d46a4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:21:04.582546 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:04.582523    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f556c088-edfc-4b0e-bf11-8568691d46a4-kube-api-access-9nz9s" pod "f556c088-edfc-4b0e-bf11-8568691d46a4" (UID: "f556c088-edfc-4b0e-bf11-8568691d46a4"). InnerVolumeSpecName "kube-api-access-9nz9s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:21:04.582998 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:04.582969    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f556c088-edfc-4b0e-bf11-8568691d46a4-maas-api-xks-ca" pod "f556c088-edfc-4b0e-bf11-8568691d46a4" (UID: "f556c088-edfc-4b0e-bf11-8568691d46a4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:21:04.681122 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:04.681082    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9nz9s\" (UniqueName: \"kubernetes.io/projected/f556c088-edfc-4b0e-bf11-8568691d46a4-kube-api-access-9nz9s\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:21:04.681122 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:04.681119    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f556c088-edfc-4b0e-bf11-8568691d46a4-maas-api-xks-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:21:04.681122 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:04.681129    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f556c088-edfc-4b0e-bf11-8568691d46a4-maas-api-openshift-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:21:05.347966 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:05.347929    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-a5e9ab-m2f7j" event={"ID":"f556c088-edfc-4b0e-bf11-8568691d46a4","Type":"ContainerDied","Data":"0f54621a37e5bcfcd7ae06ff3f39b4685bd8d0f9c72d45a2aeb7fc05c7481e2d"}
Oct 30 01:21:05.348152 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:05.347937    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-a5e9ab-m2f7j"
Oct 30 01:21:05.348152 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:05.348020    5002 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0f54621a37e5bcfcd7ae06ff3f39b4685bd8d0f9c72d45a2aeb7fc05c7481e2d"
Oct 30 01:21:09.563691 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:09.563656    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-f6d5b8-openshift-default-68ddb87d77-n4kjt"]
Oct 30 01:21:09.564192 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:09.563963    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-f6d5b8-openshift-default-68ddb87d77-n4kjt" podUID="45d94cd6-d704-48c7-b1fb-e97c89e7d74b" containerName="istio-proxy" containerID="cri-o://46a8d9dd43666738dcd4774c805c1e18802a96df9f50015c0add7defb66c9b6f" gracePeriod=30
Oct 30 01:21:10.030223 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:10.030181    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f2b534c-9xvkw"]
Oct 30 01:21:10.034687 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:10.034652    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f2b534c-9xvkw"]
Oct 30 01:21:10.112257 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:10.112216    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-16c30f-tw72l"]
Oct 30 01:21:10.113128 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:10.113106    5002 cpu_manager.go:414] "containerMap: removing container" podUID="f556c088-edfc-4b0e-bf11-8568691d46a4" containerName="revoke-keys"
Oct 30 01:21:10.113189 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:10.113132    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f556c088-edfc-4b0e-bf11-8568691d46a4" containerName="revoke-keys"
Oct 30 01:21:10.113280 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:10.113267    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="f556c088-edfc-4b0e-bf11-8568691d46a4" containerName="revoke-keys"
Oct 30 01:21:10.118741 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:10.118715    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-16c30f-tw72l"
Oct 30 01:21:10.136029 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:10.135516    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:21:10.136029 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:10.135599    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:21:10.136029 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:10.135677    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:21:10.136029 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:10.135752    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5phsh\""
Oct 30 01:21:10.138190 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:10.138169    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-16c30f-tw72l"]
Oct 30 01:21:10.271846 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:10.271808    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9ecf218-bae9-4ec0-bc28-ebff788c01fe-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-16c30f-tw72l\" (UID: \"f9ecf218-bae9-4ec0-bc28-ebff788c01fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-16c30f-tw72l"
Oct 30 01:21:10.272046 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:10.271949    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9ecf218-bae9-4ec0-bc28-ebff788c01fe-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-16c30f-tw72l\" (UID: \"f9ecf218-bae9-4ec0-bc28-ebff788c01fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-16c30f-tw72l"
Oct 30 01:21:10.272046 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:10.271991    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6b4hn\" (UniqueName: \"kubernetes.io/projected/f9ecf218-bae9-4ec0-bc28-ebff788c01fe-kube-api-access-6b4hn\") pod \"maas-api-revoke-keys-e2e-ipp-b-16c30f-tw72l\" (UID: \"f9ecf218-bae9-4ec0-bc28-ebff788c01fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-16c30f-tw72l"
Oct 30 01:21:10.353462 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:10.352427    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7d8z4"]
Oct 30 01:21:10.376393 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:10.376340    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9ecf218-bae9-4ec0-bc28-ebff788c01fe-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-16c30f-tw72l\" (UID: \"f9ecf218-bae9-4ec0-bc28-ebff788c01fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-16c30f-tw72l"
Oct 30 01:21:10.377153 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:10.377129    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6b4hn\" (UniqueName: \"kubernetes.io/projected/f9ecf218-bae9-4ec0-bc28-ebff788c01fe-kube-api-access-6b4hn\") pod \"maas-api-revoke-keys-e2e-ipp-b-16c30f-tw72l\" (UID: \"f9ecf218-bae9-4ec0-bc28-ebff788c01fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-16c30f-tw72l"
Oct 30 01:21:10.377530 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:10.377512    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9ecf218-bae9-4ec0-bc28-ebff788c01fe-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-16c30f-tw72l\" (UID: \"f9ecf218-bae9-4ec0-bc28-ebff788c01fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-16c30f-tw72l"
Oct 30 01:21:10.378817 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:10.378401    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9ecf218-bae9-4ec0-bc28-ebff788c01fe-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-16c30f-tw72l\" (UID: \"f9ecf218-bae9-4ec0-bc28-ebff788c01fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-16c30f-tw72l"
Oct 30 01:21:10.399843 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:10.399811    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9ecf218-bae9-4ec0-bc28-ebff788c01fe-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-16c30f-tw72l\" (UID: \"f9ecf218-bae9-4ec0-bc28-ebff788c01fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-16c30f-tw72l"
Oct 30 01:21:10.412434 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:10.409018    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6b4hn\" (UniqueName: \"kubernetes.io/projected/f9ecf218-bae9-4ec0-bc28-ebff788c01fe-kube-api-access-6b4hn\") pod \"maas-api-revoke-keys-e2e-ipp-b-16c30f-tw72l\" (UID: \"f9ecf218-bae9-4ec0-bc28-ebff788c01fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-16c30f-tw72l"
Oct 30 01:21:10.432834 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:10.432801    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-16c30f-tw72l"
Oct 30 01:21:10.498038 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:10.497998    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5525c91f-ca9d-41b2-b2eb-4cf53262541c" path="/var/lib/kubelet/pods/5525c91f-ca9d-41b2-b2eb-4cf53262541c/volumes"
Oct 30 01:21:10.756036 ip-10-0-128-7 kubenswrapper[5002]: W1030 01:21:10.756004    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9ecf218_bae9_4ec0_bc28_ebff788c01fe.slice/crio-4fdcba47c748f11a35264d414a6f83bf0b0e01989c27966a6e174a59b3c5d854 WatchSource:0}: Error finding container 4fdcba47c748f11a35264d414a6f83bf0b0e01989c27966a6e174a59b3c5d854: Status 404 returned error can't find the container with id 4fdcba47c748f11a35264d414a6f83bf0b0e01989c27966a6e174a59b3c5d854
Oct 30 01:21:10.757777 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:10.757747    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-16c30f-tw72l"]
Oct 30 01:21:11.431725 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:11.431625    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-16c30f-tw72l" event={"ID":"f9ecf218-bae9-4ec0-bc28-ebff788c01fe","Type":"ContainerStarted","Data":"39d626c438fe1b3f09a2ee53154e67f4b28bba3e310f9d4b3f39090a03f39399"}
Oct 30 01:21:11.431725 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:11.431699    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-16c30f-tw72l" event={"ID":"f9ecf218-bae9-4ec0-bc28-ebff788c01fe","Type":"ContainerStarted","Data":"4fdcba47c748f11a35264d414a6f83bf0b0e01989c27966a6e174a59b3c5d854"}
Oct 30 01:21:11.510403 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:11.493707    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-16c30f-tw72l" podStartSLOduration=1.493684329 podStartE2EDuration="1.493684329s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:21:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:21:11.483939621 +0000 UTC m=+2967.100372502" watchObservedRunningTime="2026-10-30 01:21:11.493684329 +0000 UTC m=+2967.110117210"
Oct 30 01:21:11.530403 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:11.520554    5002 patch_prober.go:29] interesting pod/e2e-ipp-a-f6d5b8-openshift-default-68ddb87d77-n4kjt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 01:21:11.530403 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:11.520621    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-f6d5b8-openshift-default-68ddb87d77-n4kjt" podUID="45d94cd6-d704-48c7-b1fb-e97c89e7d74b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 01:21:12.440896 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:12.440863    5002 generic.go:356] "Generic (PLEG): container finished" podID="f9ecf218-bae9-4ec0-bc28-ebff788c01fe" containerID="39d626c438fe1b3f09a2ee53154e67f4b28bba3e310f9d4b3f39090a03f39399" exitCode=0
Oct 30 01:21:12.441346 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:12.440951    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-16c30f-tw72l" event={"ID":"f9ecf218-bae9-4ec0-bc28-ebff788c01fe","Type":"ContainerDied","Data":"39d626c438fe1b3f09a2ee53154e67f4b28bba3e310f9d4b3f39090a03f39399"}
Oct 30 01:21:12.895279 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:21:12.895180    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbjdq" containerName="extractor"
Oct 30 01:21:13.683203 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:13.683165    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-16c30f-tw72l"
Oct 30 01:21:13.749055 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:13.748973    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f9ecf218-bae9-4ec0-bc28-ebff788c01fe-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9ecf218-bae9-4ec0-bc28-ebff788c01fe-maas-api-xks-ca\") pod \"f9ecf218-bae9-4ec0-bc28-ebff788c01fe\" (UID: \"f9ecf218-bae9-4ec0-bc28-ebff788c01fe\") "
Oct 30 01:21:13.749219 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:13.749177    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9ecf218-bae9-4ec0-bc28-ebff788c01fe-kube-api-access-6b4hn\" (UniqueName: \"kubernetes.io/projected/f9ecf218-bae9-4ec0-bc28-ebff788c01fe-kube-api-access-6b4hn\") pod \"f9ecf218-bae9-4ec0-bc28-ebff788c01fe\" (UID: \"f9ecf218-bae9-4ec0-bc28-ebff788c01fe\") "
Oct 30 01:21:13.749308 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:13.749280    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9ecf218-bae9-4ec0-bc28-ebff788c01fe-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9ecf218-bae9-4ec0-bc28-ebff788c01fe-maas-api-openshift-ca\") pod \"f9ecf218-bae9-4ec0-bc28-ebff788c01fe\" (UID: \"f9ecf218-bae9-4ec0-bc28-ebff788c01fe\") "
Oct 30 01:21:13.773403 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:13.770745    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9ecf218-bae9-4ec0-bc28-ebff788c01fe-maas-api-openshift-ca" pod "f9ecf218-bae9-4ec0-bc28-ebff788c01fe" (UID: "f9ecf218-bae9-4ec0-bc28-ebff788c01fe"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:21:13.788053 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:13.788016    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9ecf218-bae9-4ec0-bc28-ebff788c01fe-kube-api-access-6b4hn" pod "f9ecf218-bae9-4ec0-bc28-ebff788c01fe" (UID: "f9ecf218-bae9-4ec0-bc28-ebff788c01fe"). InnerVolumeSpecName "kube-api-access-6b4hn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:21:13.789484 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:13.789452    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9ecf218-bae9-4ec0-bc28-ebff788c01fe-maas-api-xks-ca" pod "f9ecf218-bae9-4ec0-bc28-ebff788c01fe" (UID: "f9ecf218-bae9-4ec0-bc28-ebff788c01fe"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:21:13.897094 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:13.858364    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6b4hn\" (UniqueName: \"kubernetes.io/projected/f9ecf218-bae9-4ec0-bc28-ebff788c01fe-kube-api-access-6b4hn\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:21:13.897360 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:13.897336    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9ecf218-bae9-4ec0-bc28-ebff788c01fe-maas-api-openshift-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:21:13.897532 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:13.897515    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9ecf218-bae9-4ec0-bc28-ebff788c01fe-maas-api-xks-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:21:14.484249 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:14.484198    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-16c30f-tw72l" event={"ID":"f9ecf218-bae9-4ec0-bc28-ebff788c01fe","Type":"ContainerDied","Data":"4fdcba47c748f11a35264d414a6f83bf0b0e01989c27966a6e174a59b3c5d854"}
Oct 30 01:21:14.484482 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:14.484220    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-16c30f-tw72l"
Oct 30 01:21:14.484989 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:14.484275    5002 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4fdcba47c748f11a35264d414a6f83bf0b0e01989c27966a6e174a59b3c5d854"
Oct 30 01:21:14.936854 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:14.936826    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f6d5b8-openshift-default-68ddb87d77-n4kjt"
Oct 30 01:21:15.111948 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:15.111859    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/45d94cd6-d704-48c7-b1fb-e97c89e7d74b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45d94cd6-d704-48c7-b1fb-e97c89e7d74b-istio-podinfo\") pod \"45d94cd6-d704-48c7-b1fb-e97c89e7d74b\" (UID: \"45d94cd6-d704-48c7-b1fb-e97c89e7d74b\") "
Oct 30 01:21:15.111948 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:15.111910    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45d94cd6-d704-48c7-b1fb-e97c89e7d74b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45d94cd6-d704-48c7-b1fb-e97c89e7d74b-istio-data\") pod \"45d94cd6-d704-48c7-b1fb-e97c89e7d74b\" (UID: \"45d94cd6-d704-48c7-b1fb-e97c89e7d74b\") "
Oct 30 01:21:15.112209 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:15.111953    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45d94cd6-d704-48c7-b1fb-e97c89e7d74b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45d94cd6-d704-48c7-b1fb-e97c89e7d74b-istiod-ca-cert\") pod \"45d94cd6-d704-48c7-b1fb-e97c89e7d74b\" (UID: \"45d94cd6-d704-48c7-b1fb-e97c89e7d74b\") "
Oct 30 01:21:15.112209 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:15.112007    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45d94cd6-d704-48c7-b1fb-e97c89e7d74b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45d94cd6-d704-48c7-b1fb-e97c89e7d74b-workload-certs\") pod \"45d94cd6-d704-48c7-b1fb-e97c89e7d74b\" (UID: \"45d94cd6-d704-48c7-b1fb-e97c89e7d74b\") "
Oct 30 01:21:15.112209 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:15.112072    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45d94cd6-d704-48c7-b1fb-e97c89e7d74b-istio-token\" (UniqueName: \"kubernetes.io/projected/45d94cd6-d704-48c7-b1fb-e97c89e7d74b-istio-token\") pod \"45d94cd6-d704-48c7-b1fb-e97c89e7d74b\" (UID: \"45d94cd6-d704-48c7-b1fb-e97c89e7d74b\") "
Oct 30 01:21:15.112209 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:15.112147    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45d94cd6-d704-48c7-b1fb-e97c89e7d74b-kube-api-access-9nkm2\" (UniqueName: \"kubernetes.io/projected/45d94cd6-d704-48c7-b1fb-e97c89e7d74b-kube-api-access-9nkm2\") pod \"45d94cd6-d704-48c7-b1fb-e97c89e7d74b\" (UID: \"45d94cd6-d704-48c7-b1fb-e97c89e7d74b\") "
Oct 30 01:21:15.112483 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:15.112230    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45d94cd6-d704-48c7-b1fb-e97c89e7d74b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45d94cd6-d704-48c7-b1fb-e97c89e7d74b-istio-envoy\") pod \"45d94cd6-d704-48c7-b1fb-e97c89e7d74b\" (UID: \"45d94cd6-d704-48c7-b1fb-e97c89e7d74b\") "
Oct 30 01:21:15.112483 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:15.112275    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45d94cd6-d704-48c7-b1fb-e97c89e7d74b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45d94cd6-d704-48c7-b1fb-e97c89e7d74b-credential-socket\") pod \"45d94cd6-d704-48c7-b1fb-e97c89e7d74b\" (UID: \"45d94cd6-d704-48c7-b1fb-e97c89e7d74b\") "
Oct 30 01:21:15.112483 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:15.112361    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45d94cd6-d704-48c7-b1fb-e97c89e7d74b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45d94cd6-d704-48c7-b1fb-e97c89e7d74b-workload-socket\") pod \"45d94cd6-d704-48c7-b1fb-e97c89e7d74b\" (UID: \"45d94cd6-d704-48c7-b1fb-e97c89e7d74b\") "
Oct 30 01:21:15.112483 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:15.112439    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45d94cd6-d704-48c7-b1fb-e97c89e7d74b-istiod-ca-cert" pod "45d94cd6-d704-48c7-b1fb-e97c89e7d74b" (UID: "45d94cd6-d704-48c7-b1fb-e97c89e7d74b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:21:15.112713 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:15.112583    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45d94cd6-d704-48c7-b1fb-e97c89e7d74b-workload-certs" pod "45d94cd6-d704-48c7-b1fb-e97c89e7d74b" (UID: "45d94cd6-d704-48c7-b1fb-e97c89e7d74b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:21:15.112822 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:15.112796    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45d94cd6-d704-48c7-b1fb-e97c89e7d74b-credential-socket" pod "45d94cd6-d704-48c7-b1fb-e97c89e7d74b" (UID: "45d94cd6-d704-48c7-b1fb-e97c89e7d74b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:21:15.112965 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:15.112858    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45d94cd6-d704-48c7-b1fb-e97c89e7d74b-istio-data" pod "45d94cd6-d704-48c7-b1fb-e97c89e7d74b" (UID: "45d94cd6-d704-48c7-b1fb-e97c89e7d74b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:21:15.113041 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:15.112981    5002 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45d94cd6-d704-48c7-b1fb-e97c89e7d74b-workload-certs\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:21:15.113041 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:15.113001    5002 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45d94cd6-d704-48c7-b1fb-e97c89e7d74b-credential-socket\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:21:15.113169 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:15.113050    5002 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45d94cd6-d704-48c7-b1fb-e97c89e7d74b-istio-data\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:21:15.113169 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:15.113065    5002 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45d94cd6-d704-48c7-b1fb-e97c89e7d74b-istiod-ca-cert\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:21:15.113169 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:15.113097    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45d94cd6-d704-48c7-b1fb-e97c89e7d74b-workload-socket" pod "45d94cd6-d704-48c7-b1fb-e97c89e7d74b" (UID: "45d94cd6-d704-48c7-b1fb-e97c89e7d74b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:21:15.114525 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:15.114499    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/45d94cd6-d704-48c7-b1fb-e97c89e7d74b-istio-podinfo" pod "45d94cd6-d704-48c7-b1fb-e97c89e7d74b" (UID: "45d94cd6-d704-48c7-b1fb-e97c89e7d74b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:21:15.114626 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:15.114558    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45d94cd6-d704-48c7-b1fb-e97c89e7d74b-kube-api-access-9nkm2" pod "45d94cd6-d704-48c7-b1fb-e97c89e7d74b" (UID: "45d94cd6-d704-48c7-b1fb-e97c89e7d74b"). InnerVolumeSpecName "kube-api-access-9nkm2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:21:15.114670 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:15.114636    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45d94cd6-d704-48c7-b1fb-e97c89e7d74b-istio-envoy" pod "45d94cd6-d704-48c7-b1fb-e97c89e7d74b" (UID: "45d94cd6-d704-48c7-b1fb-e97c89e7d74b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:21:15.114840 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:15.114820    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45d94cd6-d704-48c7-b1fb-e97c89e7d74b-istio-token" pod "45d94cd6-d704-48c7-b1fb-e97c89e7d74b" (UID: "45d94cd6-d704-48c7-b1fb-e97c89e7d74b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:21:15.215296 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:15.215255    5002 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45d94cd6-d704-48c7-b1fb-e97c89e7d74b-istio-token\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:21:15.215296 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:15.215299    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9nkm2\" (UniqueName: \"kubernetes.io/projected/45d94cd6-d704-48c7-b1fb-e97c89e7d74b-kube-api-access-9nkm2\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:21:15.215547 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:15.215314    5002 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45d94cd6-d704-48c7-b1fb-e97c89e7d74b-istio-envoy\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:21:15.215547 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:15.215329    5002 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45d94cd6-d704-48c7-b1fb-e97c89e7d74b-workload-socket\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:21:15.215547 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:15.215345    5002 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45d94cd6-d704-48c7-b1fb-e97c89e7d74b-istio-podinfo\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:21:15.540137 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:15.535535    5002 generic.go:356] "Generic (PLEG): container finished" podID="45d94cd6-d704-48c7-b1fb-e97c89e7d74b" containerID="46a8d9dd43666738dcd4774c805c1e18802a96df9f50015c0add7defb66c9b6f" exitCode=0
Oct 30 01:21:15.540137 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:15.535590    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f6d5b8-openshift-default-68ddb87d77-n4kjt"
Oct 30 01:21:15.540137 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:15.535703    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f6d5b8-openshift-default-68ddb87d77-n4kjt" event={"ID":"45d94cd6-d704-48c7-b1fb-e97c89e7d74b","Type":"ContainerDied","Data":"46a8d9dd43666738dcd4774c805c1e18802a96df9f50015c0add7defb66c9b6f"}
Oct 30 01:21:15.540137 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:15.535746    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f6d5b8-openshift-default-68ddb87d77-n4kjt" event={"ID":"45d94cd6-d704-48c7-b1fb-e97c89e7d74b","Type":"ContainerDied","Data":"ae77c938d542d367725032c1e41dd2e2ceda2595ad40ca36c1e148d9149d55d3"}
Oct 30 01:21:15.540137 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:15.535773    5002 scope.go:122] "RemoveContainer" containerID="46a8d9dd43666738dcd4774c805c1e18802a96df9f50015c0add7defb66c9b6f"
Oct 30 01:21:15.605580 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:15.605545    5002 scope.go:122] "RemoveContainer" containerID="3ffbaae21284766ab63e7484c47642a5f0e7df30c05526a2204493a9825c63c1"
Oct 30 01:21:15.622063 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:15.622038    5002 scope.go:122] "RemoveContainer" containerID="46a8d9dd43666738dcd4774c805c1e18802a96df9f50015c0add7defb66c9b6f"
Oct 30 01:21:15.622472 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:21:15.622441    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46a8d9dd43666738dcd4774c805c1e18802a96df9f50015c0add7defb66c9b6f\": container with ID starting with 46a8d9dd43666738dcd4774c805c1e18802a96df9f50015c0add7defb66c9b6f not found: ID does not exist" containerID="46a8d9dd43666738dcd4774c805c1e18802a96df9f50015c0add7defb66c9b6f"
Oct 30 01:21:15.622576 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:15.622482    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46a8d9dd43666738dcd4774c805c1e18802a96df9f50015c0add7defb66c9b6f"} err="failed to get container status \"46a8d9dd43666738dcd4774c805c1e18802a96df9f50015c0add7defb66c9b6f\": rpc error: code = NotFound desc = could not find container \"46a8d9dd43666738dcd4774c805c1e18802a96df9f50015c0add7defb66c9b6f\": container with ID starting with 46a8d9dd43666738dcd4774c805c1e18802a96df9f50015c0add7defb66c9b6f not found: ID does not exist"
Oct 30 01:21:15.622576 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:15.622521    5002 scope.go:122] "RemoveContainer" containerID="3ffbaae21284766ab63e7484c47642a5f0e7df30c05526a2204493a9825c63c1"
Oct 30 01:21:15.623007 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:21:15.622980    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ffbaae21284766ab63e7484c47642a5f0e7df30c05526a2204493a9825c63c1\": container with ID starting with 3ffbaae21284766ab63e7484c47642a5f0e7df30c05526a2204493a9825c63c1 not found: ID does not exist" containerID="3ffbaae21284766ab63e7484c47642a5f0e7df30c05526a2204493a9825c63c1"
Oct 30 01:21:15.623179 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:15.623153    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ffbaae21284766ab63e7484c47642a5f0e7df30c05526a2204493a9825c63c1"} err="failed to get container status \"3ffbaae21284766ab63e7484c47642a5f0e7df30c05526a2204493a9825c63c1\": rpc error: code = NotFound desc = could not find container \"3ffbaae21284766ab63e7484c47642a5f0e7df30c05526a2204493a9825c63c1\": container with ID starting with 3ffbaae21284766ab63e7484c47642a5f0e7df30c05526a2204493a9825c63c1 not found: ID does not exist"
Oct 30 01:21:15.629069 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:15.629045    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-f6d5b8-openshift-default-68ddb87d77-n4kjt"]
Oct 30 01:21:15.634170 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:15.634142    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-f6d5b8-openshift-default-68ddb87d77-n4kjt"]
Oct 30 01:21:16.217947 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:16.217901    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-cf4567-42nwr"]
Oct 30 01:21:16.238415 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:16.234565    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-cf4567-42nwr"]
Oct 30 01:21:16.496373 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:16.496285    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45d94cd6-d704-48c7-b1fb-e97c89e7d74b" path="/var/lib/kubelet/pods/45d94cd6-d704-48c7-b1fb-e97c89e7d74b/volumes"
Oct 30 01:21:16.496823 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:16.496807    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9a6030a-1f44-41f4-87a7-760d1c5ac463" path="/var/lib/kubelet/pods/f9a6030a-1f44-41f4-87a7-760d1c5ac463/volumes"
Oct 30 01:21:20.037163 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:20.037115    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6e506fd-kv6tc"]
Oct 30 01:21:20.047164 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:20.047129    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6e506fd-kv6tc"]
Oct 30 01:21:20.500194 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:20.500150    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44d5bcd2-3029-4d1d-b163-f90ee946d53b" path="/var/lib/kubelet/pods/44d5bcd2-3029-4d1d-b163-f90ee946d53b/volumes"
Oct 30 01:21:21.590395 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:21.590351    5002 generic.go:356] "Generic (PLEG): container finished" podID="7077b8b8-1f35-49c4-9e58-762297c046cf" containerID="3f0d5174659d934cb002b76b41d644fd6133deca3d9b55042a24a21b138cf830" exitCode=0
Oct 30 01:21:21.590953 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:21.590455    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-658c99-openshift-default-54d78694df-cfhwj" event={"ID":"7077b8b8-1f35-49c4-9e58-762297c046cf","Type":"ContainerDied","Data":"3f0d5174659d934cb002b76b41d644fd6133deca3d9b55042a24a21b138cf830"}
Oct 30 01:21:21.591025 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:21.590962    5002 scope.go:122] "RemoveContainer" containerID="3f0d5174659d934cb002b76b41d644fd6133deca3d9b55042a24a21b138cf830"
Oct 30 01:21:21.592790 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:21.592758    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:21:21.592917 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:21.592830    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:21:21.592917 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:21.592876    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:21:22.154863 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:22.154821    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-16c30f-tw72l"]
Oct 30 01:21:22.158315 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:22.158279    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-16c30f-tw72l"]
Oct 30 01:21:22.497077 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:22.497038    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9ecf218-bae9-4ec0-bc28-ebff788c01fe" path="/var/lib/kubelet/pods/f9ecf218-bae9-4ec0-bc28-ebff788c01fe/volumes"
Oct 30 01:21:22.640405 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:22.637226    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-658c99-openshift-default-54d78694df-cfhwj" event={"ID":"7077b8b8-1f35-49c4-9e58-762297c046cf","Type":"ContainerStarted","Data":"230d62681ee1a91e15bbf16f693d6386e160dc893faefaeadff7312dc9807422"}
Oct 30 01:21:22.665363 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:22.665318    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-658c99-openshift-default-54d78694df-cfhwj"
Oct 30 01:21:22.871490 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:22.870444    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-658c99-openshift-default-54d78694df-cfhwj"
Oct 30 01:21:22.871679 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:22.871523    5002 patch_prober.go:29] interesting pod/e2e-ipp-b-658c99-openshift-default-54d78694df-cfhwj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.14:15021/healthz/ready\": dial tcp 10.132.1.14:15021: connect: connection refused" start-of-body=
Oct 30 01:21:22.871679 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:22.871584    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-658c99-openshift-default-54d78694df-cfhwj" podUID="7077b8b8-1f35-49c4-9e58-762297c046cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.14:15021/healthz/ready\": dial tcp 10.132.1.14:15021: connect: connection refused"
Oct 30 01:21:23.866057 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:23.863866    5002 patch_prober.go:29] interesting pod/e2e-ipp-b-658c99-openshift-default-54d78694df-cfhwj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.14:15021/healthz/ready\": dial tcp 10.132.1.14:15021: connect: connection refused" start-of-body=
Oct 30 01:21:23.866057 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:23.863932    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-658c99-openshift-default-54d78694df-cfhwj" podUID="7077b8b8-1f35-49c4-9e58-762297c046cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.14:15021/healthz/ready\": dial tcp 10.132.1.14:15021: connect: connection refused"
Oct 30 01:21:24.875518 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:24.864838    5002 patch_prober.go:29] interesting pod/e2e-ipp-b-658c99-openshift-default-54d78694df-cfhwj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.14:15021/healthz/ready\": dial tcp 10.132.1.14:15021: connect: connection refused" start-of-body=
Oct 30 01:21:24.875518 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:24.864905    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-658c99-openshift-default-54d78694df-cfhwj" podUID="7077b8b8-1f35-49c4-9e58-762297c046cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.14:15021/healthz/ready\": dial tcp 10.132.1.14:15021: connect: connection refused"
Oct 30 01:21:25.863869 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:25.863827    5002 patch_prober.go:29] interesting pod/e2e-ipp-b-658c99-openshift-default-54d78694df-cfhwj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.14:15021/healthz/ready\": dial tcp 10.132.1.14:15021: connect: connection refused" start-of-body=
Oct 30 01:21:25.864306 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:25.864270    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-658c99-openshift-default-54d78694df-cfhwj" podUID="7077b8b8-1f35-49c4-9e58-762297c046cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.14:15021/healthz/ready\": dial tcp 10.132.1.14:15021: connect: connection refused"
Oct 30 01:21:26.864405 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:26.864150    5002 patch_prober.go:29] interesting pod/e2e-ipp-b-658c99-openshift-default-54d78694df-cfhwj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.14:15021/healthz/ready\": dial tcp 10.132.1.14:15021: connect: connection refused" start-of-body=
Oct 30 01:21:26.864405 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:26.864218    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-658c99-openshift-default-54d78694df-cfhwj" podUID="7077b8b8-1f35-49c4-9e58-762297c046cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.14:15021/healthz/ready\": dial tcp 10.132.1.14:15021: connect: connection refused"
Oct 30 01:21:27.867404 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:27.863858    5002 patch_prober.go:29] interesting pod/e2e-ipp-b-658c99-openshift-default-54d78694df-cfhwj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.14:15021/healthz/ready\": dial tcp 10.132.1.14:15021: connect: connection refused" start-of-body=
Oct 30 01:21:27.867404 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:27.863934    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-658c99-openshift-default-54d78694df-cfhwj" podUID="7077b8b8-1f35-49c4-9e58-762297c046cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.14:15021/healthz/ready\": dial tcp 10.132.1.14:15021: connect: connection refused"
Oct 30 01:21:28.906437 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:28.904012    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-658c99-openshift-default-54d78694df-cfhwj"
Oct 30 01:21:28.908502 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:28.908476    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-658c99-openshift-default-54d78694df-cfhwj"
Oct 30 01:21:29.392131 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:29.392096    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-16c30f-openshift-default-dffb9ddc9-6jkdj"]
Oct 30 01:21:29.392362 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:29.392332    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-16c30f-openshift-default-dffb9ddc9-6jkdj" podUID="d1c17dcc-9b86-41c5-bea2-e284d0ec960c" containerName="istio-proxy" containerID="cri-o://fc33b85c11faab6a91031b034174bee3ac3d14507e1bdec2e5015614cecef3f6" gracePeriod=30
Oct 30 01:21:34.704990 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:34.704959    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-16c30f-openshift-default-dffb9ddc9-6jkdj"
Oct 30 01:21:34.801835 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:34.801793    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1c17dcc-9b86-41c5-bea2-e284d0ec960c-kube-api-access-lmp5m\" (UniqueName: \"kubernetes.io/projected/d1c17dcc-9b86-41c5-bea2-e284d0ec960c-kube-api-access-lmp5m\") pod \"d1c17dcc-9b86-41c5-bea2-e284d0ec960c\" (UID: \"d1c17dcc-9b86-41c5-bea2-e284d0ec960c\") "
Oct 30 01:21:34.802035 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:34.801875    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1c17dcc-9b86-41c5-bea2-e284d0ec960c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1c17dcc-9b86-41c5-bea2-e284d0ec960c-credential-socket\") pod \"d1c17dcc-9b86-41c5-bea2-e284d0ec960c\" (UID: \"d1c17dcc-9b86-41c5-bea2-e284d0ec960c\") "
Oct 30 01:21:34.802035 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:34.801914    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1c17dcc-9b86-41c5-bea2-e284d0ec960c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1c17dcc-9b86-41c5-bea2-e284d0ec960c-workload-socket\") pod \"d1c17dcc-9b86-41c5-bea2-e284d0ec960c\" (UID: \"d1c17dcc-9b86-41c5-bea2-e284d0ec960c\") "
Oct 30 01:21:34.802035 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:34.802014    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1c17dcc-9b86-41c5-bea2-e284d0ec960c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1c17dcc-9b86-41c5-bea2-e284d0ec960c-istio-data\") pod \"d1c17dcc-9b86-41c5-bea2-e284d0ec960c\" (UID: \"d1c17dcc-9b86-41c5-bea2-e284d0ec960c\") "
Oct 30 01:21:34.802202 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:34.802085    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1c17dcc-9b86-41c5-bea2-e284d0ec960c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1c17dcc-9b86-41c5-bea2-e284d0ec960c-istiod-ca-cert\") pod \"d1c17dcc-9b86-41c5-bea2-e284d0ec960c\" (UID: \"d1c17dcc-9b86-41c5-bea2-e284d0ec960c\") "
Oct 30 01:21:34.802202 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:34.802188    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1c17dcc-9b86-41c5-bea2-e284d0ec960c-istio-token\" (UniqueName: \"kubernetes.io/projected/d1c17dcc-9b86-41c5-bea2-e284d0ec960c-istio-token\") pod \"d1c17dcc-9b86-41c5-bea2-e284d0ec960c\" (UID: \"d1c17dcc-9b86-41c5-bea2-e284d0ec960c\") "
Oct 30 01:21:34.802307 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:34.802235    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1c17dcc-9b86-41c5-bea2-e284d0ec960c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1c17dcc-9b86-41c5-bea2-e284d0ec960c-workload-certs\") pod \"d1c17dcc-9b86-41c5-bea2-e284d0ec960c\" (UID: \"d1c17dcc-9b86-41c5-bea2-e284d0ec960c\") "
Oct 30 01:21:34.802361 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:34.802329    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1c17dcc-9b86-41c5-bea2-e284d0ec960c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1c17dcc-9b86-41c5-bea2-e284d0ec960c-istio-envoy\") pod \"d1c17dcc-9b86-41c5-bea2-e284d0ec960c\" (UID: \"d1c17dcc-9b86-41c5-bea2-e284d0ec960c\") "
Oct 30 01:21:34.802479 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:34.802362    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d1c17dcc-9b86-41c5-bea2-e284d0ec960c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1c17dcc-9b86-41c5-bea2-e284d0ec960c-istio-podinfo\") pod \"d1c17dcc-9b86-41c5-bea2-e284d0ec960c\" (UID: \"d1c17dcc-9b86-41c5-bea2-e284d0ec960c\") "
Oct 30 01:21:34.802543 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:34.802506    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1c17dcc-9b86-41c5-bea2-e284d0ec960c-istio-data" pod "d1c17dcc-9b86-41c5-bea2-e284d0ec960c" (UID: "d1c17dcc-9b86-41c5-bea2-e284d0ec960c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:21:34.802600 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:34.802515    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1c17dcc-9b86-41c5-bea2-e284d0ec960c-workload-socket" pod "d1c17dcc-9b86-41c5-bea2-e284d0ec960c" (UID: "d1c17dcc-9b86-41c5-bea2-e284d0ec960c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:21:34.802813 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:34.802779    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1c17dcc-9b86-41c5-bea2-e284d0ec960c-credential-socket" pod "d1c17dcc-9b86-41c5-bea2-e284d0ec960c" (UID: "d1c17dcc-9b86-41c5-bea2-e284d0ec960c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:21:34.802909 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:34.802855    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1c17dcc-9b86-41c5-bea2-e284d0ec960c-istiod-ca-cert" pod "d1c17dcc-9b86-41c5-bea2-e284d0ec960c" (UID: "d1c17dcc-9b86-41c5-bea2-e284d0ec960c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:21:34.803002 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:34.802987    5002 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1c17dcc-9b86-41c5-bea2-e284d0ec960c-credential-socket\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:21:34.803062 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:34.803003    5002 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1c17dcc-9b86-41c5-bea2-e284d0ec960c-workload-socket\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:21:34.803062 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:34.803016    5002 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1c17dcc-9b86-41c5-bea2-e284d0ec960c-istio-data\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:21:34.803062 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:34.803034    5002 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1c17dcc-9b86-41c5-bea2-e284d0ec960c-istiod-ca-cert\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:21:34.803240 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:34.803097    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1c17dcc-9b86-41c5-bea2-e284d0ec960c-workload-certs" pod "d1c17dcc-9b86-41c5-bea2-e284d0ec960c" (UID: "d1c17dcc-9b86-41c5-bea2-e284d0ec960c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:21:34.805265 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:34.805240    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1c17dcc-9b86-41c5-bea2-e284d0ec960c-istio-envoy" pod "d1c17dcc-9b86-41c5-bea2-e284d0ec960c" (UID: "d1c17dcc-9b86-41c5-bea2-e284d0ec960c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:21:34.805401 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:34.805322    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1c17dcc-9b86-41c5-bea2-e284d0ec960c-istio-token" pod "d1c17dcc-9b86-41c5-bea2-e284d0ec960c" (UID: "d1c17dcc-9b86-41c5-bea2-e284d0ec960c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:21:34.805647 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:34.805616    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1c17dcc-9b86-41c5-bea2-e284d0ec960c-kube-api-access-lmp5m" pod "d1c17dcc-9b86-41c5-bea2-e284d0ec960c" (UID: "d1c17dcc-9b86-41c5-bea2-e284d0ec960c"). InnerVolumeSpecName "kube-api-access-lmp5m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:21:34.805880 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:34.805852    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d1c17dcc-9b86-41c5-bea2-e284d0ec960c-istio-podinfo" pod "d1c17dcc-9b86-41c5-bea2-e284d0ec960c" (UID: "d1c17dcc-9b86-41c5-bea2-e284d0ec960c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:21:34.909511 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:34.909326    5002 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1c17dcc-9b86-41c5-bea2-e284d0ec960c-istio-token\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:21:34.909511 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:34.909369    5002 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1c17dcc-9b86-41c5-bea2-e284d0ec960c-workload-certs\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:21:34.909511 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:34.909399    5002 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1c17dcc-9b86-41c5-bea2-e284d0ec960c-istio-envoy\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:21:34.909511 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:34.909412    5002 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1c17dcc-9b86-41c5-bea2-e284d0ec960c-istio-podinfo\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:21:34.909511 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:34.909426    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lmp5m\" (UniqueName: \"kubernetes.io/projected/d1c17dcc-9b86-41c5-bea2-e284d0ec960c-kube-api-access-lmp5m\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:21:34.977405 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:34.976543    5002 generic.go:356] "Generic (PLEG): container finished" podID="d1c17dcc-9b86-41c5-bea2-e284d0ec960c" containerID="fc33b85c11faab6a91031b034174bee3ac3d14507e1bdec2e5015614cecef3f6" exitCode=0
Oct 30 01:21:34.977405 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:34.976752    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-16c30f-openshift-default-dffb9ddc9-6jkdj" event={"ID":"d1c17dcc-9b86-41c5-bea2-e284d0ec960c","Type":"ContainerDied","Data":"fc33b85c11faab6a91031b034174bee3ac3d14507e1bdec2e5015614cecef3f6"}
Oct 30 01:21:34.977405 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:34.976806    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-16c30f-openshift-default-dffb9ddc9-6jkdj" event={"ID":"d1c17dcc-9b86-41c5-bea2-e284d0ec960c","Type":"ContainerDied","Data":"d866a76fc1176643be4606907737a42d7b87e60b28263c628be534013376b6c8"}
Oct 30 01:21:34.977405 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:34.976836    5002 scope.go:122] "RemoveContainer" containerID="fc33b85c11faab6a91031b034174bee3ac3d14507e1bdec2e5015614cecef3f6"
Oct 30 01:21:34.977405 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:34.977017    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-16c30f-openshift-default-dffb9ddc9-6jkdj"
Oct 30 01:21:35.051250 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:35.051222    5002 scope.go:122] "RemoveContainer" containerID="69cd1b815a3940aefd4ac4b378500a4e582337393a76bd80c0d630e223325db5"
Oct 30 01:21:35.068836 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:35.068808    5002 scope.go:122] "RemoveContainer" containerID="fc33b85c11faab6a91031b034174bee3ac3d14507e1bdec2e5015614cecef3f6"
Oct 30 01:21:35.069188 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:21:35.069162    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc33b85c11faab6a91031b034174bee3ac3d14507e1bdec2e5015614cecef3f6\": container with ID starting with fc33b85c11faab6a91031b034174bee3ac3d14507e1bdec2e5015614cecef3f6 not found: ID does not exist" containerID="fc33b85c11faab6a91031b034174bee3ac3d14507e1bdec2e5015614cecef3f6"
Oct 30 01:21:35.069266 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:35.069203    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc33b85c11faab6a91031b034174bee3ac3d14507e1bdec2e5015614cecef3f6"} err="failed to get container status \"fc33b85c11faab6a91031b034174bee3ac3d14507e1bdec2e5015614cecef3f6\": rpc error: code = NotFound desc = could not find container \"fc33b85c11faab6a91031b034174bee3ac3d14507e1bdec2e5015614cecef3f6\": container with ID starting with fc33b85c11faab6a91031b034174bee3ac3d14507e1bdec2e5015614cecef3f6 not found: ID does not exist"
Oct 30 01:21:35.069266 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:35.069231    5002 scope.go:122] "RemoveContainer" containerID="69cd1b815a3940aefd4ac4b378500a4e582337393a76bd80c0d630e223325db5"
Oct 30 01:21:35.069582 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:21:35.069550    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69cd1b815a3940aefd4ac4b378500a4e582337393a76bd80c0d630e223325db5\": container with ID starting with 69cd1b815a3940aefd4ac4b378500a4e582337393a76bd80c0d630e223325db5 not found: ID does not exist" containerID="69cd1b815a3940aefd4ac4b378500a4e582337393a76bd80c0d630e223325db5"
Oct 30 01:21:35.069666 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:35.069591    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69cd1b815a3940aefd4ac4b378500a4e582337393a76bd80c0d630e223325db5"} err="failed to get container status \"69cd1b815a3940aefd4ac4b378500a4e582337393a76bd80c0d630e223325db5\": rpc error: code = NotFound desc = could not find container \"69cd1b815a3940aefd4ac4b378500a4e582337393a76bd80c0d630e223325db5\": container with ID starting with 69cd1b815a3940aefd4ac4b378500a4e582337393a76bd80c0d630e223325db5 not found: ID does not exist"
Oct 30 01:21:35.097356 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:35.097314    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-16c30f-openshift-default-dffb9ddc9-6jkdj"]
Oct 30 01:21:35.102780 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:35.102752    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-16c30f-openshift-default-dffb9ddc9-6jkdj"]
Oct 30 01:21:36.496544 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:36.496507    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1c17dcc-9b86-41c5-bea2-e284d0ec960c" path="/var/lib/kubelet/pods/d1c17dcc-9b86-41c5-bea2-e284d0ec960c/volumes"
Oct 30 01:21:39.918675 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:39.918636    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-a5e9ab-m2f7j"]
Oct 30 01:21:39.932184 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:39.932150    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-a5e9ab-m2f7j"]
Oct 30 01:21:40.498318 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:40.498278    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f556c088-edfc-4b0e-bf11-8568691d46a4" path="/var/lib/kubelet/pods/f556c088-edfc-4b0e-bf11-8568691d46a4/volumes"
Oct 30 01:21:42.868447 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:21:42.868375    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbjdq" containerName="extractor"
Oct 30 01:21:45.249288 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:45.249256    5002 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-g7c2w_5dc56e7e-f3c1-43eb-a21b-8be6bb4706cf/console-operator/1.log"
Oct 30 01:21:45.278066 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:45.278036    5002 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-g7c2w_5dc56e7e-f3c1-43eb-a21b-8be6bb4706cf/console-operator/1.log"
Oct 30 01:21:46.996978 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:46.996935    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-a5e9ab-openshift-default-8498d5b8b9-vr4jt"]
Oct 30 01:21:46.997561 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:46.997205    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w5-a5e9ab-openshift-default-8498d5b8b9-vr4jt" podUID="a61107c1-4b26-4cb7-85d3-eb83cad66da6" containerName="istio-proxy" containerID="cri-o://bade7345ee202ad59779741a0d89b828943e1db291e90b257a79d3e8a207cfae" gracePeriod=30
Oct 30 01:21:47.033233 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:47.033199    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa210800-6sttw"]
Oct 30 01:21:47.040110 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:47.040077    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9cff2b63-94bdm"]
Oct 30 01:21:47.043209 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:47.043187    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa210800-6sttw"]
Oct 30 01:21:47.064249 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:47.064211    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9cff2b63-94bdm"]
Oct 30 01:21:48.495617 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:48.495582    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d8a608f-20c9-4568-9135-4d6e9ef23c97" path="/var/lib/kubelet/pods/0d8a608f-20c9-4568-9135-4d6e9ef23c97/volumes"
Oct 30 01:21:48.496102 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:48.496087    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22a7542d-4dcf-4059-b4b6-84f6b33c4b6a" path="/var/lib/kubelet/pods/22a7542d-4dcf-4059-b4b6-84f6b33c4b6a/volumes"
Oct 30 01:21:49.496228 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:49.496177    5002 scope.go:122] "RemoveContainer" containerID="5ae3e9b5c539a28285ac7448bf74ad6b8fd23c1461d36884d436a32b87fe5f58"
Oct 30 01:21:51.457145 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:51.457109    5002 patch_prober.go:29] interesting pod/e2e-shared-b-w5-a5e9ab-openshift-default-8498d5b8b9-vr4jt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 01:21:51.457571 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:51.457177    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w5-a5e9ab-openshift-default-8498d5b8b9-vr4jt" podUID="a61107c1-4b26-4cb7-85d3-eb83cad66da6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 01:21:52.202538 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:52.202492    5002 generic.go:356] "Generic (PLEG): container finished" podID="a61107c1-4b26-4cb7-85d3-eb83cad66da6" containerID="bade7345ee202ad59779741a0d89b828943e1db291e90b257a79d3e8a207cfae" exitCode=0
Oct 30 01:21:52.202741 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:52.202634    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-a5e9ab-openshift-default-8498d5b8b9-vr4jt" event={"ID":"a61107c1-4b26-4cb7-85d3-eb83cad66da6","Type":"ContainerDied","Data":"bade7345ee202ad59779741a0d89b828943e1db291e90b257a79d3e8a207cfae"}
Oct 30 01:21:52.477302 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:52.477264    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-a5e9ab-openshift-default-8498d5b8b9-vr4jt"
Oct 30 01:21:52.556189 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:52.556145    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-6f4fba/simulator-w5-6f4fba-kserve-5bf8dfddd5-wcwzh"]
Oct 30 01:21:52.564614 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:52.562758    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a61107c1-4b26-4cb7-85d3-eb83cad66da6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a61107c1-4b26-4cb7-85d3-eb83cad66da6-workload-certs\") pod \"a61107c1-4b26-4cb7-85d3-eb83cad66da6\" (UID: \"a61107c1-4b26-4cb7-85d3-eb83cad66da6\") "
Oct 30 01:21:52.564614 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:52.562817    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a61107c1-4b26-4cb7-85d3-eb83cad66da6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a61107c1-4b26-4cb7-85d3-eb83cad66da6-istiod-ca-cert\") pod \"a61107c1-4b26-4cb7-85d3-eb83cad66da6\" (UID: \"a61107c1-4b26-4cb7-85d3-eb83cad66da6\") "
Oct 30 01:21:52.564614 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:52.562853    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a61107c1-4b26-4cb7-85d3-eb83cad66da6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a61107c1-4b26-4cb7-85d3-eb83cad66da6-istio-envoy\") pod \"a61107c1-4b26-4cb7-85d3-eb83cad66da6\" (UID: \"a61107c1-4b26-4cb7-85d3-eb83cad66da6\") "
Oct 30 01:21:52.564614 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:52.562915    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a61107c1-4b26-4cb7-85d3-eb83cad66da6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a61107c1-4b26-4cb7-85d3-eb83cad66da6-workload-socket\") pod \"a61107c1-4b26-4cb7-85d3-eb83cad66da6\" (UID: \"a61107c1-4b26-4cb7-85d3-eb83cad66da6\") "
Oct 30 01:21:52.564614 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:52.562980    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a61107c1-4b26-4cb7-85d3-eb83cad66da6-istio-token\" (UniqueName: \"kubernetes.io/projected/a61107c1-4b26-4cb7-85d3-eb83cad66da6-istio-token\") pod \"a61107c1-4b26-4cb7-85d3-eb83cad66da6\" (UID: \"a61107c1-4b26-4cb7-85d3-eb83cad66da6\") "
Oct 30 01:21:52.564614 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:52.563410    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a61107c1-4b26-4cb7-85d3-eb83cad66da6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a61107c1-4b26-4cb7-85d3-eb83cad66da6-credential-socket\") pod \"a61107c1-4b26-4cb7-85d3-eb83cad66da6\" (UID: \"a61107c1-4b26-4cb7-85d3-eb83cad66da6\") "
Oct 30 01:21:52.564614 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:52.563487    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a61107c1-4b26-4cb7-85d3-eb83cad66da6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a61107c1-4b26-4cb7-85d3-eb83cad66da6-istio-podinfo\") pod \"a61107c1-4b26-4cb7-85d3-eb83cad66da6\" (UID: \"a61107c1-4b26-4cb7-85d3-eb83cad66da6\") "
Oct 30 01:21:52.564614 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:52.563558    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a61107c1-4b26-4cb7-85d3-eb83cad66da6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a61107c1-4b26-4cb7-85d3-eb83cad66da6-istio-data\") pod \"a61107c1-4b26-4cb7-85d3-eb83cad66da6\" (UID: \"a61107c1-4b26-4cb7-85d3-eb83cad66da6\") "
Oct 30 01:21:52.564614 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:52.563677    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a61107c1-4b26-4cb7-85d3-eb83cad66da6-kube-api-access-m4dq9\" (UniqueName: \"kubernetes.io/projected/a61107c1-4b26-4cb7-85d3-eb83cad66da6-kube-api-access-m4dq9\") pod \"a61107c1-4b26-4cb7-85d3-eb83cad66da6\" (UID: \"a61107c1-4b26-4cb7-85d3-eb83cad66da6\") "
Oct 30 01:21:52.564614 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:52.563771    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a61107c1-4b26-4cb7-85d3-eb83cad66da6-workload-certs" pod "a61107c1-4b26-4cb7-85d3-eb83cad66da6" (UID: "a61107c1-4b26-4cb7-85d3-eb83cad66da6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:21:52.564614 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:52.564197    5002 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a61107c1-4b26-4cb7-85d3-eb83cad66da6-workload-certs\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:21:52.565421 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:52.564766    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a61107c1-4b26-4cb7-85d3-eb83cad66da6-workload-socket" pod "a61107c1-4b26-4cb7-85d3-eb83cad66da6" (UID: "a61107c1-4b26-4cb7-85d3-eb83cad66da6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:21:52.565421 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:52.565049    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a61107c1-4b26-4cb7-85d3-eb83cad66da6-istiod-ca-cert" pod "a61107c1-4b26-4cb7-85d3-eb83cad66da6" (UID: "a61107c1-4b26-4cb7-85d3-eb83cad66da6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:21:52.565421 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:52.565330    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a61107c1-4b26-4cb7-85d3-eb83cad66da6-credential-socket" pod "a61107c1-4b26-4cb7-85d3-eb83cad66da6" (UID: "a61107c1-4b26-4cb7-85d3-eb83cad66da6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:21:52.565613 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:52.565553    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a61107c1-4b26-4cb7-85d3-eb83cad66da6-istio-data" pod "a61107c1-4b26-4cb7-85d3-eb83cad66da6" (UID: "a61107c1-4b26-4cb7-85d3-eb83cad66da6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:21:52.567280 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:52.567249    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a61107c1-4b26-4cb7-85d3-eb83cad66da6-istio-podinfo" pod "a61107c1-4b26-4cb7-85d3-eb83cad66da6" (UID: "a61107c1-4b26-4cb7-85d3-eb83cad66da6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:21:52.572950 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:52.572880    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a61107c1-4b26-4cb7-85d3-eb83cad66da6-istio-envoy" pod "a61107c1-4b26-4cb7-85d3-eb83cad66da6" (UID: "a61107c1-4b26-4cb7-85d3-eb83cad66da6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:21:52.572950 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:52.572906    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a61107c1-4b26-4cb7-85d3-eb83cad66da6-istio-token" pod "a61107c1-4b26-4cb7-85d3-eb83cad66da6" (UID: "a61107c1-4b26-4cb7-85d3-eb83cad66da6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:21:52.574257 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:52.574228    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a61107c1-4b26-4cb7-85d3-eb83cad66da6-kube-api-access-m4dq9" pod "a61107c1-4b26-4cb7-85d3-eb83cad66da6" (UID: "a61107c1-4b26-4cb7-85d3-eb83cad66da6"). InnerVolumeSpecName "kube-api-access-m4dq9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:21:52.665285 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:52.665246    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4dq9\" (UniqueName: \"kubernetes.io/projected/a61107c1-4b26-4cb7-85d3-eb83cad66da6-kube-api-access-m4dq9\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:21:52.665285 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:52.665284    5002 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a61107c1-4b26-4cb7-85d3-eb83cad66da6-istiod-ca-cert\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:21:52.665285 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:52.665294    5002 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a61107c1-4b26-4cb7-85d3-eb83cad66da6-istio-envoy\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:21:52.665579 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:52.665303    5002 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a61107c1-4b26-4cb7-85d3-eb83cad66da6-workload-socket\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:21:52.665579 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:52.665311    5002 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a61107c1-4b26-4cb7-85d3-eb83cad66da6-istio-token\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:21:52.665579 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:52.665319    5002 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a61107c1-4b26-4cb7-85d3-eb83cad66da6-credential-socket\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:21:52.665579 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:52.665327    5002 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a61107c1-4b26-4cb7-85d3-eb83cad66da6-istio-podinfo\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:21:52.665579 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:52.665335    5002 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a61107c1-4b26-4cb7-85d3-eb83cad66da6-istio-data\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:21:53.211617 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:53.211581    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-a5e9ab-openshift-default-8498d5b8b9-vr4jt" event={"ID":"a61107c1-4b26-4cb7-85d3-eb83cad66da6","Type":"ContainerDied","Data":"56cc6c5936988295e798157d244ac83ae4a6b72b7fe41ae9ba4ecf5e81ef3216"}
Oct 30 01:21:53.211835 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:53.211651    5002 scope.go:122] "RemoveContainer" containerID="bade7345ee202ad59779741a0d89b828943e1db291e90b257a79d3e8a207cfae"
Oct 30 01:21:53.211835 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:53.211655    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-a5e9ab-openshift-default-8498d5b8b9-vr4jt"
Oct 30 01:21:53.285795 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:53.285720    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-a5e9ab-openshift-default-8498d5b8b9-vr4jt"]
Oct 30 01:21:53.299323 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:53.299292    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-a5e9ab-openshift-default-8498d5b8b9-vr4jt"]
Oct 30 01:21:54.496521 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:54.496484    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a61107c1-4b26-4cb7-85d3-eb83cad66da6" path="/var/lib/kubelet/pods/a61107c1-4b26-4cb7-85d3-eb83cad66da6/volumes"
Oct 30 01:21:59.028547 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:59.028466    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f47f1548-n9tw2"]
Oct 30 01:21:59.042171 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:21:59.042138    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f47f1548-n9tw2"]
Oct 30 01:22:00.498429 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:00.498372    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3b976fd-1e7c-4bb1-b514-abc54f0a188f" path="/var/lib/kubelet/pods/b3b976fd-1e7c-4bb1-b514-abc54f0a188f/volumes"
Oct 30 01:22:02.329288 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:02.329235    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7d8z4"]
Oct 30 01:22:07.029765 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:07.029725    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-34f855e6-84xqh"]
Oct 30 01:22:07.043218 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:07.043181    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-34f855e6-84xqh"]
Oct 30 01:22:07.589913 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:07.589852    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-6f4fba/simulator-w5-6f4fba-kserve-5bf8dfddd5-wcwzh" podUID="3a2c2f8f-649c-4ab6-af98-5d87df0e511f" containerName="main" containerID="cri-o://2f55590b96efc9d7861b27804d263d18b107d5ab17f5fb4bfd04c2d7db0f2477" gracePeriod=45
Oct 30 01:22:07.866597 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:07.866568    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-6f4fba/simulator-w5-6f4fba-kserve-5bf8dfddd5-wcwzh"
Oct 30 01:22:07.930969 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:07.930931    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a2c2f8f-649c-4ab6-af98-5d87df0e511f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a2c2f8f-649c-4ab6-af98-5d87df0e511f-model-cache\") pod \"3a2c2f8f-649c-4ab6-af98-5d87df0e511f\" (UID: \"3a2c2f8f-649c-4ab6-af98-5d87df0e511f\") "
Oct 30 01:22:07.931170 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:07.931081    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a2c2f8f-649c-4ab6-af98-5d87df0e511f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a2c2f8f-649c-4ab6-af98-5d87df0e511f-dshm\") pod \"3a2c2f8f-649c-4ab6-af98-5d87df0e511f\" (UID: \"3a2c2f8f-649c-4ab6-af98-5d87df0e511f\") "
Oct 30 01:22:07.931170 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:07.931127    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a2c2f8f-649c-4ab6-af98-5d87df0e511f-kube-api-access-cj9tn\" (UniqueName: \"kubernetes.io/projected/3a2c2f8f-649c-4ab6-af98-5d87df0e511f-kube-api-access-cj9tn\") pod \"3a2c2f8f-649c-4ab6-af98-5d87df0e511f\" (UID: \"3a2c2f8f-649c-4ab6-af98-5d87df0e511f\") "
Oct 30 01:22:07.931313 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:07.931179    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3a2c2f8f-649c-4ab6-af98-5d87df0e511f-tls-certs\" (UniqueName: \"kubernetes.io/secret/3a2c2f8f-649c-4ab6-af98-5d87df0e511f-tls-certs\") pod \"3a2c2f8f-649c-4ab6-af98-5d87df0e511f\" (UID: \"3a2c2f8f-649c-4ab6-af98-5d87df0e511f\") "
Oct 30 01:22:07.931313 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:07.931199    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a2c2f8f-649c-4ab6-af98-5d87df0e511f-model-cache" pod "3a2c2f8f-649c-4ab6-af98-5d87df0e511f" (UID: "3a2c2f8f-649c-4ab6-af98-5d87df0e511f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:22:07.931313 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:07.931275    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a2c2f8f-649c-4ab6-af98-5d87df0e511f-home\" (UniqueName: \"kubernetes.io/empty-dir/3a2c2f8f-649c-4ab6-af98-5d87df0e511f-home\") pod \"3a2c2f8f-649c-4ab6-af98-5d87df0e511f\" (UID: \"3a2c2f8f-649c-4ab6-af98-5d87df0e511f\") "
Oct 30 01:22:07.931524 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:07.931311    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a2c2f8f-649c-4ab6-af98-5d87df0e511f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a2c2f8f-649c-4ab6-af98-5d87df0e511f-tmp-dir\") pod \"3a2c2f8f-649c-4ab6-af98-5d87df0e511f\" (UID: \"3a2c2f8f-649c-4ab6-af98-5d87df0e511f\") "
Oct 30 01:22:07.932045 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:07.931796    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a2c2f8f-649c-4ab6-af98-5d87df0e511f-home" pod "3a2c2f8f-649c-4ab6-af98-5d87df0e511f" (UID: "3a2c2f8f-649c-4ab6-af98-5d87df0e511f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:22:07.932045 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:07.932020    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a2c2f8f-649c-4ab6-af98-5d87df0e511f-tmp-dir" pod "3a2c2f8f-649c-4ab6-af98-5d87df0e511f" (UID: "3a2c2f8f-649c-4ab6-af98-5d87df0e511f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:22:07.932045 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:07.932026    5002 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a2c2f8f-649c-4ab6-af98-5d87df0e511f-home\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:22:07.932391 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:07.932073    5002 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a2c2f8f-649c-4ab6-af98-5d87df0e511f-model-cache\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:22:07.933880 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:07.933853    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a2c2f8f-649c-4ab6-af98-5d87df0e511f-tls-certs" pod "3a2c2f8f-649c-4ab6-af98-5d87df0e511f" (UID: "3a2c2f8f-649c-4ab6-af98-5d87df0e511f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:22:07.934149 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:07.934130    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a2c2f8f-649c-4ab6-af98-5d87df0e511f-kube-api-access-cj9tn" pod "3a2c2f8f-649c-4ab6-af98-5d87df0e511f" (UID: "3a2c2f8f-649c-4ab6-af98-5d87df0e511f"). InnerVolumeSpecName "kube-api-access-cj9tn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:22:07.934616 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:07.934592    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a2c2f8f-649c-4ab6-af98-5d87df0e511f-dshm" pod "3a2c2f8f-649c-4ab6-af98-5d87df0e511f" (UID: "3a2c2f8f-649c-4ab6-af98-5d87df0e511f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:22:08.033860 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:08.033823    5002 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a2c2f8f-649c-4ab6-af98-5d87df0e511f-tmp-dir\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:22:08.033860 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:08.033856    5002 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a2c2f8f-649c-4ab6-af98-5d87df0e511f-dshm\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:22:08.034304 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:08.033869    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cj9tn\" (UniqueName: \"kubernetes.io/projected/3a2c2f8f-649c-4ab6-af98-5d87df0e511f-kube-api-access-cj9tn\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:22:08.034304 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:08.033883    5002 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a2c2f8f-649c-4ab6-af98-5d87df0e511f-tls-certs\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:22:08.355515 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:08.355466    5002 generic.go:356] "Generic (PLEG): container finished" podID="3a2c2f8f-649c-4ab6-af98-5d87df0e511f" containerID="2f55590b96efc9d7861b27804d263d18b107d5ab17f5fb4bfd04c2d7db0f2477" exitCode=0
Oct 30 01:22:08.355719 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:08.355536    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-6f4fba/simulator-w5-6f4fba-kserve-5bf8dfddd5-wcwzh"
Oct 30 01:22:08.355719 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:08.355569    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-6f4fba/simulator-w5-6f4fba-kserve-5bf8dfddd5-wcwzh" event={"ID":"3a2c2f8f-649c-4ab6-af98-5d87df0e511f","Type":"ContainerDied","Data":"2f55590b96efc9d7861b27804d263d18b107d5ab17f5fb4bfd04c2d7db0f2477"}
Oct 30 01:22:08.355719 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:08.355626    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-6f4fba/simulator-w5-6f4fba-kserve-5bf8dfddd5-wcwzh" event={"ID":"3a2c2f8f-649c-4ab6-af98-5d87df0e511f","Type":"ContainerDied","Data":"d74f16da72402e61bf2badd1275a05d2b7ac28f968d73c9d2de86ca15aff92ff"}
Oct 30 01:22:08.355719 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:08.355651    5002 scope.go:122] "RemoveContainer" containerID="2f55590b96efc9d7861b27804d263d18b107d5ab17f5fb4bfd04c2d7db0f2477"
Oct 30 01:22:08.367510 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:08.367486    5002 scope.go:122] "RemoveContainer" containerID="2f55590b96efc9d7861b27804d263d18b107d5ab17f5fb4bfd04c2d7db0f2477"
Oct 30 01:22:08.367848 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:22:08.367828    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f55590b96efc9d7861b27804d263d18b107d5ab17f5fb4bfd04c2d7db0f2477\": container with ID starting with 2f55590b96efc9d7861b27804d263d18b107d5ab17f5fb4bfd04c2d7db0f2477 not found: ID does not exist" containerID="2f55590b96efc9d7861b27804d263d18b107d5ab17f5fb4bfd04c2d7db0f2477"
Oct 30 01:22:08.367910 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:08.367857    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f55590b96efc9d7861b27804d263d18b107d5ab17f5fb4bfd04c2d7db0f2477"} err="failed to get container status \"2f55590b96efc9d7861b27804d263d18b107d5ab17f5fb4bfd04c2d7db0f2477\": rpc error: code = NotFound desc = could not find container \"2f55590b96efc9d7861b27804d263d18b107d5ab17f5fb4bfd04c2d7db0f2477\": container with ID starting with 2f55590b96efc9d7861b27804d263d18b107d5ab17f5fb4bfd04c2d7db0f2477 not found: ID does not exist"
Oct 30 01:22:08.411445 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:08.411407    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-6f4fba/simulator-w5-6f4fba-kserve-5bf8dfddd5-wcwzh"]
Oct 30 01:22:08.425483 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:08.425436    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-6f4fba/simulator-w5-6f4fba-kserve-5bf8dfddd5-wcwzh"]
Oct 30 01:22:08.499698 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:08.499652    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a2c2f8f-649c-4ab6-af98-5d87df0e511f" path="/var/lib/kubelet/pods/3a2c2f8f-649c-4ab6-af98-5d87df0e511f/volumes"
Oct 30 01:22:08.500449 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:08.500424    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8933c664-e833-4091-98c0-2e39fc21e321" path="/var/lib/kubelet/pods/8933c664-e833-4091-98c0-2e39fc21e321/volumes"
Oct 30 01:22:12.862053 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:22:12.862008    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbjdq" containerName="extractor"
Oct 30 01:22:19.114504 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:19.114464    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c3c6878-mwshd"]
Oct 30 01:22:19.115036 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:19.115016    5002 cpu_manager.go:414] "containerMap: removing container" podUID="a61107c1-4b26-4cb7-85d3-eb83cad66da6" containerName="istio-proxy"
Oct 30 01:22:19.115116 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:19.115031    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a61107c1-4b26-4cb7-85d3-eb83cad66da6" containerName="istio-proxy"
Oct 30 01:22:19.115116 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:19.115069    5002 cpu_manager.go:414] "containerMap: removing container" podUID="45d94cd6-d704-48c7-b1fb-e97c89e7d74b" containerName="istio-proxy"
Oct 30 01:22:19.115116 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:19.115076    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45d94cd6-d704-48c7-b1fb-e97c89e7d74b" containerName="istio-proxy"
Oct 30 01:22:19.115116 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:19.115096    5002 cpu_manager.go:414] "containerMap: removing container" podUID="d1c17dcc-9b86-41c5-bea2-e284d0ec960c" containerName="istio-proxy"
Oct 30 01:22:19.115116 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:19.115105    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1c17dcc-9b86-41c5-bea2-e284d0ec960c" containerName="istio-proxy"
Oct 30 01:22:19.115116 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:19.115115    5002 cpu_manager.go:414] "containerMap: removing container" podUID="3a2c2f8f-649c-4ab6-af98-5d87df0e511f" containerName="main"
Oct 30 01:22:19.115116 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:19.115119    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a2c2f8f-649c-4ab6-af98-5d87df0e511f" containerName="main"
Oct 30 01:22:19.115549 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:19.115133    5002 cpu_manager.go:414] "containerMap: removing container" podUID="d1c17dcc-9b86-41c5-bea2-e284d0ec960c" containerName="istio-proxy"
Oct 30 01:22:19.115549 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:19.115138    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1c17dcc-9b86-41c5-bea2-e284d0ec960c" containerName="istio-proxy"
Oct 30 01:22:19.115549 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:19.115154    5002 cpu_manager.go:414] "containerMap: removing container" podUID="f9ecf218-bae9-4ec0-bc28-ebff788c01fe" containerName="revoke-keys"
Oct 30 01:22:19.115549 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:19.115159    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9ecf218-bae9-4ec0-bc28-ebff788c01fe" containerName="revoke-keys"
Oct 30 01:22:19.115549 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:19.115242    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="45d94cd6-d704-48c7-b1fb-e97c89e7d74b" containerName="istio-proxy"
Oct 30 01:22:19.115549 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:19.115255    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a2c2f8f-649c-4ab6-af98-5d87df0e511f" containerName="main"
Oct 30 01:22:19.115549 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:19.115264    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1c17dcc-9b86-41c5-bea2-e284d0ec960c" containerName="istio-proxy"
Oct 30 01:22:19.115549 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:19.115273    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1c17dcc-9b86-41c5-bea2-e284d0ec960c" containerName="istio-proxy"
Oct 30 01:22:19.115549 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:19.115281    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9ecf218-bae9-4ec0-bc28-ebff788c01fe" containerName="revoke-keys"
Oct 30 01:22:19.115549 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:19.115287    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="a61107c1-4b26-4cb7-85d3-eb83cad66da6" containerName="istio-proxy"
Oct 30 01:22:19.118668 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:19.118648    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c3c6878-mwshd"
Oct 30 01:22:19.139461 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:19.139423    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c3c6878-mwshd"]
Oct 30 01:22:19.148495 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:19.148463    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:22:19.148726 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:19.148529    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:22:19.148861 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:19.148629    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5phsh\""
Oct 30 01:22:19.154608 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:19.154582    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:22:19.209585 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:19.209549    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b878fdb-4d45-4e91-910e-a10aec5326e6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4c3c6878-mwshd\" (UID: \"9b878fdb-4d45-4e91-910e-a10aec5326e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c3c6878-mwshd"
Oct 30 01:22:19.209759 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:19.209615    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swn49\" (UniqueName: \"kubernetes.io/projected/9b878fdb-4d45-4e91-910e-a10aec5326e6-kube-api-access-swn49\") pod \"maas-api-revoke-sub-4c3c6878-mwshd\" (UID: \"9b878fdb-4d45-4e91-910e-a10aec5326e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c3c6878-mwshd"
Oct 30 01:22:19.209805 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:19.209756    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b878fdb-4d45-4e91-910e-a10aec5326e6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4c3c6878-mwshd\" (UID: \"9b878fdb-4d45-4e91-910e-a10aec5326e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c3c6878-mwshd"
Oct 30 01:22:19.311784 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:19.311750    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b878fdb-4d45-4e91-910e-a10aec5326e6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4c3c6878-mwshd\" (UID: \"9b878fdb-4d45-4e91-910e-a10aec5326e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c3c6878-mwshd"
Oct 30 01:22:19.312937 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:19.312912    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swn49\" (UniqueName: \"kubernetes.io/projected/9b878fdb-4d45-4e91-910e-a10aec5326e6-kube-api-access-swn49\") pod \"maas-api-revoke-sub-4c3c6878-mwshd\" (UID: \"9b878fdb-4d45-4e91-910e-a10aec5326e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c3c6878-mwshd"
Oct 30 01:22:19.313582 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:19.313549    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b878fdb-4d45-4e91-910e-a10aec5326e6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4c3c6878-mwshd\" (UID: \"9b878fdb-4d45-4e91-910e-a10aec5326e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c3c6878-mwshd"
Oct 30 01:22:19.314179 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:19.312794    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b878fdb-4d45-4e91-910e-a10aec5326e6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4c3c6878-mwshd\" (UID: \"9b878fdb-4d45-4e91-910e-a10aec5326e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c3c6878-mwshd"
Oct 30 01:22:19.321665 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:19.321641    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b878fdb-4d45-4e91-910e-a10aec5326e6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4c3c6878-mwshd\" (UID: \"9b878fdb-4d45-4e91-910e-a10aec5326e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c3c6878-mwshd"
Oct 30 01:22:19.338198 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:19.338170    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-swn49\" (UniqueName: \"kubernetes.io/projected/9b878fdb-4d45-4e91-910e-a10aec5326e6-kube-api-access-swn49\") pod \"maas-api-revoke-sub-4c3c6878-mwshd\" (UID: \"9b878fdb-4d45-4e91-910e-a10aec5326e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c3c6878-mwshd"
Oct 30 01:22:19.450853 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:19.449620    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c3c6878-mwshd"
Oct 30 01:22:19.873752 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:19.873719    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c3c6878-mwshd"]
Oct 30 01:22:19.873891 ip-10-0-128-7 kubenswrapper[5002]: W1030 01:22:19.873721    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b878fdb_4d45_4e91_910e_a10aec5326e6.slice/crio-492a9a452abf721152edf6d4d8f9d61b8024fd3137b11c7cb25acf4f4863d622 WatchSource:0}: Error finding container 492a9a452abf721152edf6d4d8f9d61b8024fd3137b11c7cb25acf4f4863d622: Status 404 returned error can't find the container with id 492a9a452abf721152edf6d4d8f9d61b8024fd3137b11c7cb25acf4f4863d622
Oct 30 01:22:19.876708 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:19.876684    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-73435a-zh592"]
Oct 30 01:22:19.877307 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:19.877288    5002 cpu_manager.go:414] "containerMap: removing container" podUID="45d94cd6-d704-48c7-b1fb-e97c89e7d74b" containerName="istio-proxy"
Oct 30 01:22:19.877438 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:19.877307    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45d94cd6-d704-48c7-b1fb-e97c89e7d74b" containerName="istio-proxy"
Oct 30 01:22:19.877515 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:19.877453    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="45d94cd6-d704-48c7-b1fb-e97c89e7d74b" containerName="istio-proxy"
Oct 30 01:22:19.881870 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:19.881850    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-73435a-zh592"
Oct 30 01:22:19.902755 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:19.902729    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-73435a-zh592"]
Oct 30 01:22:20.028035 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:20.027992    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvx8c\" (UniqueName: \"kubernetes.io/projected/58f2f201-d6fa-488b-8723-0655a7039e21-kube-api-access-hvx8c\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-73435a-zh592\" (UID: \"58f2f201-d6fa-488b-8723-0655a7039e21\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-73435a-zh592"
Oct 30 01:22:20.028225 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:20.028071    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58f2f201-d6fa-488b-8723-0655a7039e21-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-73435a-zh592\" (UID: \"58f2f201-d6fa-488b-8723-0655a7039e21\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-73435a-zh592"
Oct 30 01:22:20.028225 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:20.028118    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58f2f201-d6fa-488b-8723-0655a7039e21-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-73435a-zh592\" (UID: \"58f2f201-d6fa-488b-8723-0655a7039e21\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-73435a-zh592"
Oct 30 01:22:20.129458 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:20.129332    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvx8c\" (UniqueName: \"kubernetes.io/projected/58f2f201-d6fa-488b-8723-0655a7039e21-kube-api-access-hvx8c\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-73435a-zh592\" (UID: \"58f2f201-d6fa-488b-8723-0655a7039e21\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-73435a-zh592"
Oct 30 01:22:20.129458 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:20.129428    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58f2f201-d6fa-488b-8723-0655a7039e21-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-73435a-zh592\" (UID: \"58f2f201-d6fa-488b-8723-0655a7039e21\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-73435a-zh592"
Oct 30 01:22:20.129458 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:20.129456    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58f2f201-d6fa-488b-8723-0655a7039e21-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-73435a-zh592\" (UID: \"58f2f201-d6fa-488b-8723-0655a7039e21\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-73435a-zh592"
Oct 30 01:22:20.130260 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:20.130236    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58f2f201-d6fa-488b-8723-0655a7039e21-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-73435a-zh592\" (UID: \"58f2f201-d6fa-488b-8723-0655a7039e21\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-73435a-zh592"
Oct 30 01:22:20.132429 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:20.132403    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58f2f201-d6fa-488b-8723-0655a7039e21-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-73435a-zh592\" (UID: \"58f2f201-d6fa-488b-8723-0655a7039e21\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-73435a-zh592"
Oct 30 01:22:20.154755 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:20.154728    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvx8c\" (UniqueName: \"kubernetes.io/projected/58f2f201-d6fa-488b-8723-0655a7039e21-kube-api-access-hvx8c\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-73435a-zh592\" (UID: \"58f2f201-d6fa-488b-8723-0655a7039e21\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-73435a-zh592"
Oct 30 01:22:20.204726 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:20.204688    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-73435a-zh592"
Oct 30 01:22:20.506818 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:20.506774    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c3c6878-mwshd" event={"ID":"9b878fdb-4d45-4e91-910e-a10aec5326e6","Type":"ContainerStarted","Data":"642555ab9c1737905a24a3577f3fd39390af54d2d5aebde47da78edcb98a1a6d"}
Oct 30 01:22:20.507050 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:20.506843    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c3c6878-mwshd" event={"ID":"9b878fdb-4d45-4e91-910e-a10aec5326e6","Type":"ContainerStarted","Data":"492a9a452abf721152edf6d4d8f9d61b8024fd3137b11c7cb25acf4f4863d622"}
Oct 30 01:22:20.514925 ip-10-0-128-7 kubenswrapper[5002]: W1030 01:22:20.514897    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58f2f201_d6fa_488b_8723_0655a7039e21.slice/crio-02c2ea99f4f1d88c6c76a86dafc2fcf64d9f5138bdd32d12ce26649a5001c689 WatchSource:0}: Error finding container 02c2ea99f4f1d88c6c76a86dafc2fcf64d9f5138bdd32d12ce26649a5001c689: Status 404 returned error can't find the container with id 02c2ea99f4f1d88c6c76a86dafc2fcf64d9f5138bdd32d12ce26649a5001c689
Oct 30 01:22:20.515156 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:20.515128    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-73435a-zh592"]
Oct 30 01:22:20.552466 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:20.552415    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c3c6878-mwshd" podStartSLOduration=1.552365854 podStartE2EDuration="1.552365854s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:22:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:22:20.552203454 +0000 UTC m=+3036.168636360" watchObservedRunningTime="2026-10-30 01:22:20.552365854 +0000 UTC m=+3036.168798732"
Oct 30 01:22:21.540311 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:21.540271    5002 generic.go:356] "Generic (PLEG): container finished" podID="58f2f201-d6fa-488b-8723-0655a7039e21" containerID="87e2f1653f106f3b7486b227c24256dc236733d15945bcbec2ca7cefcf4e7215" exitCode=0
Oct 30 01:22:21.540753 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:21.540346    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-73435a-zh592" event={"ID":"58f2f201-d6fa-488b-8723-0655a7039e21","Type":"ContainerDied","Data":"87e2f1653f106f3b7486b227c24256dc236733d15945bcbec2ca7cefcf4e7215"}
Oct 30 01:22:21.540753 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:21.540414    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-73435a-zh592" event={"ID":"58f2f201-d6fa-488b-8723-0655a7039e21","Type":"ContainerStarted","Data":"02c2ea99f4f1d88c6c76a86dafc2fcf64d9f5138bdd32d12ce26649a5001c689"}
Oct 30 01:22:21.541964 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:21.541941    5002 generic.go:356] "Generic (PLEG): container finished" podID="9b878fdb-4d45-4e91-910e-a10aec5326e6" containerID="642555ab9c1737905a24a3577f3fd39390af54d2d5aebde47da78edcb98a1a6d" exitCode=0
Oct 30 01:22:21.542086 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:21.542020    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c3c6878-mwshd" event={"ID":"9b878fdb-4d45-4e91-910e-a10aec5326e6","Type":"ContainerDied","Data":"642555ab9c1737905a24a3577f3fd39390af54d2d5aebde47da78edcb98a1a6d"}
Oct 30 01:22:22.722268 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:22.722240    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c3c6878-mwshd"
Oct 30 01:22:22.781853 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:22.781825    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-73435a-zh592"
Oct 30 01:22:22.862624 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:22.861257    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/58f2f201-d6fa-488b-8723-0655a7039e21-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58f2f201-d6fa-488b-8723-0655a7039e21-maas-api-xks-ca\") pod \"58f2f201-d6fa-488b-8723-0655a7039e21\" (UID: \"58f2f201-d6fa-488b-8723-0655a7039e21\") "
Oct 30 01:22:22.862624 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:22.861402    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b878fdb-4d45-4e91-910e-a10aec5326e6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b878fdb-4d45-4e91-910e-a10aec5326e6-maas-api-openshift-ca\") pod \"9b878fdb-4d45-4e91-910e-a10aec5326e6\" (UID: \"9b878fdb-4d45-4e91-910e-a10aec5326e6\") "
Oct 30 01:22:22.862624 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:22.861442    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9b878fdb-4d45-4e91-910e-a10aec5326e6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b878fdb-4d45-4e91-910e-a10aec5326e6-maas-api-xks-ca\") pod \"9b878fdb-4d45-4e91-910e-a10aec5326e6\" (UID: \"9b878fdb-4d45-4e91-910e-a10aec5326e6\") "
Oct 30 01:22:22.862624 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:22.861501    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b878fdb-4d45-4e91-910e-a10aec5326e6-kube-api-access-swn49\" (UniqueName: \"kubernetes.io/projected/9b878fdb-4d45-4e91-910e-a10aec5326e6-kube-api-access-swn49\") pod \"9b878fdb-4d45-4e91-910e-a10aec5326e6\" (UID: \"9b878fdb-4d45-4e91-910e-a10aec5326e6\") "
Oct 30 01:22:22.862624 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:22.861533    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58f2f201-d6fa-488b-8723-0655a7039e21-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58f2f201-d6fa-488b-8723-0655a7039e21-maas-api-openshift-ca\") pod \"58f2f201-d6fa-488b-8723-0655a7039e21\" (UID: \"58f2f201-d6fa-488b-8723-0655a7039e21\") "
Oct 30 01:22:22.862624 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:22.861648    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58f2f201-d6fa-488b-8723-0655a7039e21-kube-api-access-hvx8c\" (UniqueName: \"kubernetes.io/projected/58f2f201-d6fa-488b-8723-0655a7039e21-kube-api-access-hvx8c\") pod \"58f2f201-d6fa-488b-8723-0655a7039e21\" (UID: \"58f2f201-d6fa-488b-8723-0655a7039e21\") "
Oct 30 01:22:22.864475 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:22.864239    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b878fdb-4d45-4e91-910e-a10aec5326e6-maas-api-openshift-ca" pod "9b878fdb-4d45-4e91-910e-a10aec5326e6" (UID: "9b878fdb-4d45-4e91-910e-a10aec5326e6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:22:22.864475 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:22.864440    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58f2f201-d6fa-488b-8723-0655a7039e21-maas-api-openshift-ca" pod "58f2f201-d6fa-488b-8723-0655a7039e21" (UID: "58f2f201-d6fa-488b-8723-0655a7039e21"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:22:22.864902 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:22.864868    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58f2f201-d6fa-488b-8723-0655a7039e21-kube-api-access-hvx8c" pod "58f2f201-d6fa-488b-8723-0655a7039e21" (UID: "58f2f201-d6fa-488b-8723-0655a7039e21"). InnerVolumeSpecName "kube-api-access-hvx8c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:22:22.867635 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:22.867600    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b878fdb-4d45-4e91-910e-a10aec5326e6-kube-api-access-swn49" pod "9b878fdb-4d45-4e91-910e-a10aec5326e6" (UID: "9b878fdb-4d45-4e91-910e-a10aec5326e6"). InnerVolumeSpecName "kube-api-access-swn49". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:22:22.868069 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:22.868036    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58f2f201-d6fa-488b-8723-0655a7039e21-maas-api-xks-ca" pod "58f2f201-d6fa-488b-8723-0655a7039e21" (UID: "58f2f201-d6fa-488b-8723-0655a7039e21"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:22:22.869090 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:22.869056    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b878fdb-4d45-4e91-910e-a10aec5326e6-maas-api-xks-ca" pod "9b878fdb-4d45-4e91-910e-a10aec5326e6" (UID: "9b878fdb-4d45-4e91-910e-a10aec5326e6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:22:22.963757 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:22.963714    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58f2f201-d6fa-488b-8723-0655a7039e21-maas-api-xks-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:22:22.963757 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:22.963756    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b878fdb-4d45-4e91-910e-a10aec5326e6-maas-api-openshift-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:22:22.964020 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:22.963771    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b878fdb-4d45-4e91-910e-a10aec5326e6-maas-api-xks-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:22:22.964020 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:22.963786    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-swn49\" (UniqueName: \"kubernetes.io/projected/9b878fdb-4d45-4e91-910e-a10aec5326e6-kube-api-access-swn49\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:22:22.964020 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:22.963801    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58f2f201-d6fa-488b-8723-0655a7039e21-maas-api-openshift-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:22:22.964020 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:22.963817    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hvx8c\" (UniqueName: \"kubernetes.io/projected/58f2f201-d6fa-488b-8723-0655a7039e21-kube-api-access-hvx8c\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:22:23.558519 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:23.558486    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-73435a-zh592"
Oct 30 01:22:23.558865 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:23.558831    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-73435a-zh592" event={"ID":"58f2f201-d6fa-488b-8723-0655a7039e21","Type":"ContainerDied","Data":"02c2ea99f4f1d88c6c76a86dafc2fcf64d9f5138bdd32d12ce26649a5001c689"}
Oct 30 01:22:23.558954 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:23.558896    5002 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="02c2ea99f4f1d88c6c76a86dafc2fcf64d9f5138bdd32d12ce26649a5001c689"
Oct 30 01:22:23.560448 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:23.560414    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c3c6878-mwshd" event={"ID":"9b878fdb-4d45-4e91-910e-a10aec5326e6","Type":"ContainerDied","Data":"492a9a452abf721152edf6d4d8f9d61b8024fd3137b11c7cb25acf4f4863d622"}
Oct 30 01:22:23.560608 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:23.560448    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c3c6878-mwshd"
Oct 30 01:22:23.560608 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:23.560464    5002 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="492a9a452abf721152edf6d4d8f9d61b8024fd3137b11c7cb25acf4f4863d622"
Oct 30 01:22:24.141720 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:24.141560    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1e8c0e-rzkzc"]
Oct 30 01:22:24.142163 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:24.142144    5002 cpu_manager.go:414] "containerMap: removing container" podUID="9b878fdb-4d45-4e91-910e-a10aec5326e6" containerName="revoke-keys"
Oct 30 01:22:24.142209 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:24.142164    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b878fdb-4d45-4e91-910e-a10aec5326e6" containerName="revoke-keys"
Oct 30 01:22:24.142209 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:24.142195    5002 cpu_manager.go:414] "containerMap: removing container" podUID="58f2f201-d6fa-488b-8723-0655a7039e21" containerName="revoke-keys"
Oct 30 01:22:24.142209 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:24.142200    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58f2f201-d6fa-488b-8723-0655a7039e21" containerName="revoke-keys"
Oct 30 01:22:24.142316 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:24.142305    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b878fdb-4d45-4e91-910e-a10aec5326e6" containerName="revoke-keys"
Oct 30 01:22:24.142355 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:24.142322    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="58f2f201-d6fa-488b-8723-0655a7039e21" containerName="revoke-keys"
Oct 30 01:22:24.146736 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:24.146716    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1e8c0e-rzkzc"
Oct 30 01:22:24.167108 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:24.167073    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1e8c0e-rzkzc"]
Oct 30 01:22:24.175426 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:24.175365    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:22:24.175805 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:24.175787    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:22:24.178974 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:24.178950    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:22:24.179106 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:24.179045    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5phsh\""
Oct 30 01:22:24.296875 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:24.296829    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3af29b6-2cf9-451e-bf6b-7009c5f20f26-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-1e8c0e-rzkzc\" (UID: \"d3af29b6-2cf9-451e-bf6b-7009c5f20f26\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1e8c0e-rzkzc"
Oct 30 01:22:24.297059 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:24.296947    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3af29b6-2cf9-451e-bf6b-7009c5f20f26-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-1e8c0e-rzkzc\" (UID: \"d3af29b6-2cf9-451e-bf6b-7009c5f20f26\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1e8c0e-rzkzc"
Oct 30 01:22:24.297059 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:24.297031    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snw2r\" (UniqueName: \"kubernetes.io/projected/d3af29b6-2cf9-451e-bf6b-7009c5f20f26-kube-api-access-snw2r\") pod \"maas-api-revoke-keys-e2e-ipp-b-1e8c0e-rzkzc\" (UID: \"d3af29b6-2cf9-451e-bf6b-7009c5f20f26\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1e8c0e-rzkzc"
Oct 30 01:22:24.398087 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:24.397991    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3af29b6-2cf9-451e-bf6b-7009c5f20f26-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-1e8c0e-rzkzc\" (UID: \"d3af29b6-2cf9-451e-bf6b-7009c5f20f26\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1e8c0e-rzkzc"
Oct 30 01:22:24.398087 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:24.398059    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3af29b6-2cf9-451e-bf6b-7009c5f20f26-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-1e8c0e-rzkzc\" (UID: \"d3af29b6-2cf9-451e-bf6b-7009c5f20f26\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1e8c0e-rzkzc"
Oct 30 01:22:24.398324 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:24.398148    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snw2r\" (UniqueName: \"kubernetes.io/projected/d3af29b6-2cf9-451e-bf6b-7009c5f20f26-kube-api-access-snw2r\") pod \"maas-api-revoke-keys-e2e-ipp-b-1e8c0e-rzkzc\" (UID: \"d3af29b6-2cf9-451e-bf6b-7009c5f20f26\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1e8c0e-rzkzc"
Oct 30 01:22:24.399143 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:24.399113    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3af29b6-2cf9-451e-bf6b-7009c5f20f26-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-1e8c0e-rzkzc\" (UID: \"d3af29b6-2cf9-451e-bf6b-7009c5f20f26\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1e8c0e-rzkzc"
Oct 30 01:22:24.401847 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:24.401824    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3af29b6-2cf9-451e-bf6b-7009c5f20f26-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-1e8c0e-rzkzc\" (UID: \"d3af29b6-2cf9-451e-bf6b-7009c5f20f26\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1e8c0e-rzkzc"
Oct 30 01:22:24.422988 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:24.422958    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-snw2r\" (UniqueName: \"kubernetes.io/projected/d3af29b6-2cf9-451e-bf6b-7009c5f20f26-kube-api-access-snw2r\") pod \"maas-api-revoke-keys-e2e-ipp-b-1e8c0e-rzkzc\" (UID: \"d3af29b6-2cf9-451e-bf6b-7009c5f20f26\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1e8c0e-rzkzc"
Oct 30 01:22:24.464278 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:24.464244    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1e8c0e-rzkzc"
Oct 30 01:22:24.770141 ip-10-0-128-7 kubenswrapper[5002]: W1030 01:22:24.770105    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3af29b6_2cf9_451e_bf6b_7009c5f20f26.slice/crio-95531d13119a7d0eee3007112bea7a049e57dcaf93a45e2e3a96280499005a17 WatchSource:0}: Error finding container 95531d13119a7d0eee3007112bea7a049e57dcaf93a45e2e3a96280499005a17: Status 404 returned error can't find the container with id 95531d13119a7d0eee3007112bea7a049e57dcaf93a45e2e3a96280499005a17
Oct 30 01:22:24.770310 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:24.770190    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1e8c0e-rzkzc"]
Oct 30 01:22:25.580078 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:25.580040    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1e8c0e-rzkzc" event={"ID":"d3af29b6-2cf9-451e-bf6b-7009c5f20f26","Type":"ContainerStarted","Data":"b254f80b2e38979762ddab4f2d84ee77460cfa08d1d56bcc8e55e04cdc01e5df"}
Oct 30 01:22:25.580618 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:25.580098    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1e8c0e-rzkzc" event={"ID":"d3af29b6-2cf9-451e-bf6b-7009c5f20f26","Type":"ContainerStarted","Data":"95531d13119a7d0eee3007112bea7a049e57dcaf93a45e2e3a96280499005a17"}
Oct 30 01:22:25.623861 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:25.623805    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1e8c0e-rzkzc" podStartSLOduration=1.62379192 podStartE2EDuration="1.62379192s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:22:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:22:25.622762904 +0000 UTC m=+3041.239195784" watchObservedRunningTime="2026-10-30 01:22:25.62379192 +0000 UTC m=+3041.240224799"
Oct 30 01:22:25.731017 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:25.730984    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7d8z4"]
Oct 30 01:22:26.589936 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:26.589900    5002 generic.go:356] "Generic (PLEG): container finished" podID="d3af29b6-2cf9-451e-bf6b-7009c5f20f26" containerID="b254f80b2e38979762ddab4f2d84ee77460cfa08d1d56bcc8e55e04cdc01e5df" exitCode=0
Oct 30 01:22:26.590363 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:26.589984    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1e8c0e-rzkzc" event={"ID":"d3af29b6-2cf9-451e-bf6b-7009c5f20f26","Type":"ContainerDied","Data":"b254f80b2e38979762ddab4f2d84ee77460cfa08d1d56bcc8e55e04cdc01e5df"}
Oct 30 01:22:27.750900 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:27.750870    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1e8c0e-rzkzc"
Oct 30 01:22:27.940860 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:27.940821    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3af29b6-2cf9-451e-bf6b-7009c5f20f26-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3af29b6-2cf9-451e-bf6b-7009c5f20f26-maas-api-openshift-ca\") pod \"d3af29b6-2cf9-451e-bf6b-7009c5f20f26\" (UID: \"d3af29b6-2cf9-451e-bf6b-7009c5f20f26\") "
Oct 30 01:22:27.941065 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:27.940940    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d3af29b6-2cf9-451e-bf6b-7009c5f20f26-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3af29b6-2cf9-451e-bf6b-7009c5f20f26-maas-api-xks-ca\") pod \"d3af29b6-2cf9-451e-bf6b-7009c5f20f26\" (UID: \"d3af29b6-2cf9-451e-bf6b-7009c5f20f26\") "
Oct 30 01:22:27.941135 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:27.941066    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3af29b6-2cf9-451e-bf6b-7009c5f20f26-kube-api-access-snw2r\" (UniqueName: \"kubernetes.io/projected/d3af29b6-2cf9-451e-bf6b-7009c5f20f26-kube-api-access-snw2r\") pod \"d3af29b6-2cf9-451e-bf6b-7009c5f20f26\" (UID: \"d3af29b6-2cf9-451e-bf6b-7009c5f20f26\") "
Oct 30 01:22:27.941270 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:27.941240    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3af29b6-2cf9-451e-bf6b-7009c5f20f26-maas-api-openshift-ca" pod "d3af29b6-2cf9-451e-bf6b-7009c5f20f26" (UID: "d3af29b6-2cf9-451e-bf6b-7009c5f20f26"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:22:27.941619 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:27.941593    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3af29b6-2cf9-451e-bf6b-7009c5f20f26-maas-api-openshift-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:22:27.943233 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:27.943206    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3af29b6-2cf9-451e-bf6b-7009c5f20f26-maas-api-xks-ca" pod "d3af29b6-2cf9-451e-bf6b-7009c5f20f26" (UID: "d3af29b6-2cf9-451e-bf6b-7009c5f20f26"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:22:27.943332 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:27.943238    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3af29b6-2cf9-451e-bf6b-7009c5f20f26-kube-api-access-snw2r" pod "d3af29b6-2cf9-451e-bf6b-7009c5f20f26" (UID: "d3af29b6-2cf9-451e-bf6b-7009c5f20f26"). InnerVolumeSpecName "kube-api-access-snw2r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:22:28.043601 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:28.043563    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-snw2r\" (UniqueName: \"kubernetes.io/projected/d3af29b6-2cf9-451e-bf6b-7009c5f20f26-kube-api-access-snw2r\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:22:28.043601 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:28.043604    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3af29b6-2cf9-451e-bf6b-7009c5f20f26-maas-api-xks-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:22:28.606701 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:28.606659    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1e8c0e-rzkzc" event={"ID":"d3af29b6-2cf9-451e-bf6b-7009c5f20f26","Type":"ContainerDied","Data":"95531d13119a7d0eee3007112bea7a049e57dcaf93a45e2e3a96280499005a17"}
Oct 30 01:22:28.606701 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:28.606691    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1e8c0e-rzkzc"
Oct 30 01:22:28.606926 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:28.606715    5002 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="95531d13119a7d0eee3007112bea7a049e57dcaf93a45e2e3a96280499005a17"
Oct 30 01:22:30.588158 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:30.588123    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-73435a-7645c9bc64-wcm9t"]
Oct 30 01:22:30.622008 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:30.621973    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-73435a-766b94d975-d8ztv"]
Oct 30 01:22:35.589197 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:35.589132    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-73435a-7645c9bc64-wcm9t" podUID="60bf7933-a5cb-447a-96cc-e28389894a20" containerName="payload-processing" containerID="cri-o://3f04f15f187a63d291d20282ba8ba29ee3840ad9cf05ebe52dec8927a08f0e60" gracePeriod=25
Oct 30 01:22:35.622334 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:35.622286    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-73435a-766b94d975-d8ztv" podUID="a02e27db-2cc5-404f-8881-6efd27b6f48a" containerName="payload-pre-processing" containerID="cri-o://ae3ab6cc2a1adc5d05878cd80fb856333b98280bfd735d3b21b250cb1185a37a" gracePeriod=25
Oct 30 01:22:37.790209 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:37.790164    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-73435a-766b94d975-d8ztv" podUID="a02e27db-2cc5-404f-8881-6efd27b6f48a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:22:37.790669 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:37.790164    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-73435a-7645c9bc64-wcm9t" podUID="60bf7933-a5cb-447a-96cc-e28389894a20" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:22:41.923904 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:41.923864    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-73435a-zh592"]
Oct 30 01:22:41.937157 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:41.937120    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-73435a-zh592"]
Oct 30 01:22:42.501573 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:42.501520    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58f2f201-d6fa-488b-8723-0655a7039e21" path="/var/lib/kubelet/pods/58f2f201-d6fa-488b-8723-0655a7039e21/volumes"
Oct 30 01:22:42.862807 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:22:42.862715    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbjdq" containerName="extractor"
Oct 30 01:22:47.561636 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:47.561596    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7d8z4"]
Oct 30 01:22:47.790354 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:47.790310    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-73435a-7645c9bc64-wcm9t" podUID="60bf7933-a5cb-447a-96cc-e28389894a20" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:22:47.790577 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:47.790428    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-73435a-766b94d975-d8ztv" podUID="a02e27db-2cc5-404f-8881-6efd27b6f48a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:22:49.150881 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:49.150846    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1e8c0e-rzkzc"]
Oct 30 01:22:49.165506 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:49.165473    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1e8c0e-rzkzc"]
Oct 30 01:22:49.539035 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:49.538993    5002 scope.go:122] "RemoveContainer" containerID="bad96d37e14523ca508eef3afdc62587d6fd3fdd5cc51a9a50da1094d2903720"
Oct 30 01:22:49.549856 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:49.549830    5002 scope.go:122] "RemoveContainer" containerID="385fc6acc6328c609ed4691b8e1bddc3de7155e3040e27f06aa79d9089e9bed8"
Oct 30 01:22:49.560744 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:49.560721    5002 scope.go:122] "RemoveContainer" containerID="ed25f2124508554d400dfa6c85427fdf4aad342c3c1e9c7318f002e95fbf2408"
Oct 30 01:22:49.570970 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:49.570946    5002 scope.go:122] "RemoveContainer" containerID="341ca5dd074ba6de591068dce98d9ffe848e8250a9c42b3f56211db52426d38e"
Oct 30 01:22:49.582185 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:49.582164    5002 scope.go:122] "RemoveContainer" containerID="bcf9afd51c11fb718dd6a70d15ada064c566d06c34fc8a6790a9a7562ac7f613"
Oct 30 01:22:49.593184 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:49.593164    5002 scope.go:122] "RemoveContainer" containerID="ba5ceaa053f661f3ae62cc6771ba6eca78089c96823f62396a13bd50c999cc95"
Oct 30 01:22:49.603498 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:49.602628    5002 scope.go:122] "RemoveContainer" containerID="1d07332934936c02743528807ce7ca35a0c42fe2afac4e47ccb2628f6bd16524"
Oct 30 01:22:50.531416 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:50.528036    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3af29b6-2cf9-451e-bf6b-7009c5f20f26" path="/var/lib/kubelet/pods/d3af29b6-2cf9-451e-bf6b-7009c5f20f26/volumes"
Oct 30 01:22:50.622008 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:50.621963    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-73435a-openshift-default-fc454477f-x6g2b"]
Oct 30 01:22:50.622258 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:50.622226    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-73435a-openshift-default-fc454477f-x6g2b" podUID="b38dbda1-9a06-461d-8d0a-a9413b76d333" containerName="istio-proxy" containerID="cri-o://83add8a04a5e9f8469035acafb8674f0fbc3bf3e48da05e04540c95c8ba8038d" gracePeriod=30
Oct 30 01:22:50.806454 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:50.801077    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-b73232-openshift-default-59c5464ccc-2kpc7"]
Oct 30 01:22:50.806454 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:50.801720    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-b73232-openshift-default-59c5464ccc-2kpc7" podUID="8c2a594b-3bbf-4c94-8a9f-7b79c663f135" containerName="istio-proxy" containerID="cri-o://24749d74a59c84ce731b6945d64b5d7397191f25a67a3b0836dfac7a2c54ff95" gracePeriod=30
Oct 30 01:22:51.432236 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:51.432200    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-cbs5z"]
Oct 30 01:22:51.432519 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:51.432488    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-cbs5z" podUID="f86c006d-5087-4443-b907-c2a30815b206" containerName="istio-proxy" containerID="cri-o://3f464cc547370328ffeb0384981e8e49111bd27743b52f2b77406696aa09d5c5" gracePeriod=30
Oct 30 01:22:52.395855 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:52.395821    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-cefb27/simulator-w3-cefb27-kserve-6b669c9d9f-zx2vm"]
Oct 30 01:22:55.939795 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:55.939603    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-73435a-7645c9bc64-wcm9t"
Oct 30 01:22:55.972816 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:55.972586    5002 generic.go:356] "Generic (PLEG): container finished" podID="60bf7933-a5cb-447a-96cc-e28389894a20" containerID="3f04f15f187a63d291d20282ba8ba29ee3840ad9cf05ebe52dec8927a08f0e60" exitCode=0
Oct 30 01:22:55.972816 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:55.972639    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-73435a-7645c9bc64-wcm9t"
Oct 30 01:22:55.972816 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:55.972693    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-73435a-7645c9bc64-wcm9t" event={"ID":"60bf7933-a5cb-447a-96cc-e28389894a20","Type":"ContainerDied","Data":"3f04f15f187a63d291d20282ba8ba29ee3840ad9cf05ebe52dec8927a08f0e60"}
Oct 30 01:22:55.973125 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:55.973106    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-73435a-7645c9bc64-wcm9t" event={"ID":"60bf7933-a5cb-447a-96cc-e28389894a20","Type":"ContainerDied","Data":"2bf22295b61b1631141841905a8dbc599177db0855ba6397d02287bbbd958e47"}
Oct 30 01:22:55.973180 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:55.973150    5002 scope.go:122] "RemoveContainer" containerID="3f04f15f187a63d291d20282ba8ba29ee3840ad9cf05ebe52dec8927a08f0e60"
Oct 30 01:22:55.975056 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:55.975005    5002 generic.go:356] "Generic (PLEG): container finished" podID="a02e27db-2cc5-404f-8881-6efd27b6f48a" containerID="ae3ab6cc2a1adc5d05878cd80fb856333b98280bfd735d3b21b250cb1185a37a" exitCode=0
Oct 30 01:22:55.975197 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:55.975165    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-73435a-766b94d975-d8ztv" event={"ID":"a02e27db-2cc5-404f-8881-6efd27b6f48a","Type":"ContainerDied","Data":"ae3ab6cc2a1adc5d05878cd80fb856333b98280bfd735d3b21b250cb1185a37a"}
Oct 30 01:22:55.983188 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:55.983161    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60bf7933-a5cb-447a-96cc-e28389894a20-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60bf7933-a5cb-447a-96cc-e28389894a20-plugins-config-volume\") pod \"60bf7933-a5cb-447a-96cc-e28389894a20\" (UID: \"60bf7933-a5cb-447a-96cc-e28389894a20\") "
Oct 30 01:22:55.983400 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:55.983346    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60bf7933-a5cb-447a-96cc-e28389894a20-kube-api-access-vbg9g\" (UniqueName: \"kubernetes.io/projected/60bf7933-a5cb-447a-96cc-e28389894a20-kube-api-access-vbg9g\") pod \"60bf7933-a5cb-447a-96cc-e28389894a20\" (UID: \"60bf7933-a5cb-447a-96cc-e28389894a20\") "
Oct 30 01:22:55.989956 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:55.988012    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60bf7933-a5cb-447a-96cc-e28389894a20-plugins-config-volume" pod "60bf7933-a5cb-447a-96cc-e28389894a20" (UID: "60bf7933-a5cb-447a-96cc-e28389894a20"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:22:55.990433 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:55.990368    5002 generic.go:356] "Generic (PLEG): container finished" podID="b38dbda1-9a06-461d-8d0a-a9413b76d333" containerID="83add8a04a5e9f8469035acafb8674f0fbc3bf3e48da05e04540c95c8ba8038d" exitCode=0
Oct 30 01:22:55.990433 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:55.990412    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-73435a-openshift-default-fc454477f-x6g2b" event={"ID":"b38dbda1-9a06-461d-8d0a-a9413b76d333","Type":"ContainerDied","Data":"83add8a04a5e9f8469035acafb8674f0fbc3bf3e48da05e04540c95c8ba8038d"}
Oct 30 01:22:55.992945 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:55.992912    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60bf7933-a5cb-447a-96cc-e28389894a20-kube-api-access-vbg9g" pod "60bf7933-a5cb-447a-96cc-e28389894a20" (UID: "60bf7933-a5cb-447a-96cc-e28389894a20"). InnerVolumeSpecName "kube-api-access-vbg9g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:22:56.002560 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:56.002544    5002 scope.go:122] "RemoveContainer" containerID="3f04f15f187a63d291d20282ba8ba29ee3840ad9cf05ebe52dec8927a08f0e60"
Oct 30 01:22:56.002848 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:22:56.002828    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f04f15f187a63d291d20282ba8ba29ee3840ad9cf05ebe52dec8927a08f0e60\": container with ID starting with 3f04f15f187a63d291d20282ba8ba29ee3840ad9cf05ebe52dec8927a08f0e60 not found: ID does not exist" containerID="3f04f15f187a63d291d20282ba8ba29ee3840ad9cf05ebe52dec8927a08f0e60"
Oct 30 01:22:56.002908 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:56.002861    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f04f15f187a63d291d20282ba8ba29ee3840ad9cf05ebe52dec8927a08f0e60"} err="failed to get container status \"3f04f15f187a63d291d20282ba8ba29ee3840ad9cf05ebe52dec8927a08f0e60\": rpc error: code = NotFound desc = could not find container \"3f04f15f187a63d291d20282ba8ba29ee3840ad9cf05ebe52dec8927a08f0e60\": container with ID starting with 3f04f15f187a63d291d20282ba8ba29ee3840ad9cf05ebe52dec8927a08f0e60 not found: ID does not exist"
Oct 30 01:22:56.002908 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:56.002879    5002 scope.go:122] "RemoveContainer" containerID="b245cf8d44445c2ed5ef47ce7669b2c108f4555f746c312e4322c4795e538938"
Oct 30 01:22:56.021889 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:56.021861    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-73435a-766b94d975-d8ztv"
Oct 30 01:22:56.030765 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:56.030732    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-73435a-openshift-default-fc454477f-x6g2b"
Oct 30 01:22:56.084952 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:56.084919    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b38dbda1-9a06-461d-8d0a-a9413b76d333-kube-api-access-vt5w8\" (UniqueName: \"kubernetes.io/projected/b38dbda1-9a06-461d-8d0a-a9413b76d333-kube-api-access-vt5w8\") pod \"b38dbda1-9a06-461d-8d0a-a9413b76d333\" (UID: \"b38dbda1-9a06-461d-8d0a-a9413b76d333\") "
Oct 30 01:22:56.085149 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:56.084963    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a02e27db-2cc5-404f-8881-6efd27b6f48a-kube-api-access-jrj6d\" (UniqueName: \"kubernetes.io/projected/a02e27db-2cc5-404f-8881-6efd27b6f48a-kube-api-access-jrj6d\") pod \"a02e27db-2cc5-404f-8881-6efd27b6f48a\" (UID: \"a02e27db-2cc5-404f-8881-6efd27b6f48a\") "
Oct 30 01:22:56.085149 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:56.085014    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b38dbda1-9a06-461d-8d0a-a9413b76d333-istio-token\" (UniqueName: \"kubernetes.io/projected/b38dbda1-9a06-461d-8d0a-a9413b76d333-istio-token\") pod \"b38dbda1-9a06-461d-8d0a-a9413b76d333\" (UID: \"b38dbda1-9a06-461d-8d0a-a9413b76d333\") "
Oct 30 01:22:56.085149 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:56.085052    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b38dbda1-9a06-461d-8d0a-a9413b76d333-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b38dbda1-9a06-461d-8d0a-a9413b76d333-workload-socket\") pod \"b38dbda1-9a06-461d-8d0a-a9413b76d333\" (UID: \"b38dbda1-9a06-461d-8d0a-a9413b76d333\") "
Oct 30 01:22:56.085149 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:56.085124    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b38dbda1-9a06-461d-8d0a-a9413b76d333-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b38dbda1-9a06-461d-8d0a-a9413b76d333-istiod-ca-cert\") pod \"b38dbda1-9a06-461d-8d0a-a9413b76d333\" (UID: \"b38dbda1-9a06-461d-8d0a-a9413b76d333\") "
Oct 30 01:22:56.085414 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:56.085167    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b38dbda1-9a06-461d-8d0a-a9413b76d333-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b38dbda1-9a06-461d-8d0a-a9413b76d333-credential-socket\") pod \"b38dbda1-9a06-461d-8d0a-a9413b76d333\" (UID: \"b38dbda1-9a06-461d-8d0a-a9413b76d333\") "
Oct 30 01:22:56.085414 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:56.085207    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b38dbda1-9a06-461d-8d0a-a9413b76d333-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b38dbda1-9a06-461d-8d0a-a9413b76d333-istio-data\") pod \"b38dbda1-9a06-461d-8d0a-a9413b76d333\" (UID: \"b38dbda1-9a06-461d-8d0a-a9413b76d333\") "
Oct 30 01:22:56.085414 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:56.085314    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a02e27db-2cc5-404f-8881-6efd27b6f48a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a02e27db-2cc5-404f-8881-6efd27b6f48a-plugins-config-volume\") pod \"a02e27db-2cc5-404f-8881-6efd27b6f48a\" (UID: \"a02e27db-2cc5-404f-8881-6efd27b6f48a\") "
Oct 30 01:22:56.085414 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:56.085344    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b38dbda1-9a06-461d-8d0a-a9413b76d333-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b38dbda1-9a06-461d-8d0a-a9413b76d333-istio-envoy\") pod \"b38dbda1-9a06-461d-8d0a-a9413b76d333\" (UID: \"b38dbda1-9a06-461d-8d0a-a9413b76d333\") "
Oct 30 01:22:56.085656 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:56.085453    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b38dbda1-9a06-461d-8d0a-a9413b76d333-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b38dbda1-9a06-461d-8d0a-a9413b76d333-istio-podinfo\") pod \"b38dbda1-9a06-461d-8d0a-a9413b76d333\" (UID: \"b38dbda1-9a06-461d-8d0a-a9413b76d333\") "
Oct 30 01:22:56.085656 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:56.085540    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b38dbda1-9a06-461d-8d0a-a9413b76d333-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b38dbda1-9a06-461d-8d0a-a9413b76d333-workload-certs\") pod \"b38dbda1-9a06-461d-8d0a-a9413b76d333\" (UID: \"b38dbda1-9a06-461d-8d0a-a9413b76d333\") "
Oct 30 01:22:56.085901 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:56.085845    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b38dbda1-9a06-461d-8d0a-a9413b76d333-workload-socket" pod "b38dbda1-9a06-461d-8d0a-a9413b76d333" (UID: "b38dbda1-9a06-461d-8d0a-a9413b76d333"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:22:56.086055 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:56.086026    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b38dbda1-9a06-461d-8d0a-a9413b76d333-istio-data" pod "b38dbda1-9a06-461d-8d0a-a9413b76d333" (UID: "b38dbda1-9a06-461d-8d0a-a9413b76d333"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:22:56.086153 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:56.086133    5002 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60bf7933-a5cb-447a-96cc-e28389894a20-plugins-config-volume\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:22:56.086219 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:56.086160    5002 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b38dbda1-9a06-461d-8d0a-a9413b76d333-istio-data\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:22:56.086219 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:56.086173    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vbg9g\" (UniqueName: \"kubernetes.io/projected/60bf7933-a5cb-447a-96cc-e28389894a20-kube-api-access-vbg9g\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:22:56.086219 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:56.086186    5002 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b38dbda1-9a06-461d-8d0a-a9413b76d333-workload-socket\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:22:56.088182 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:56.086715    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b38dbda1-9a06-461d-8d0a-a9413b76d333-istiod-ca-cert" pod "b38dbda1-9a06-461d-8d0a-a9413b76d333" (UID: "b38dbda1-9a06-461d-8d0a-a9413b76d333"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:22:56.088182 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:56.087102    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b38dbda1-9a06-461d-8d0a-a9413b76d333-credential-socket" pod "b38dbda1-9a06-461d-8d0a-a9413b76d333" (UID: "b38dbda1-9a06-461d-8d0a-a9413b76d333"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:22:56.088182 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:56.088147    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a02e27db-2cc5-404f-8881-6efd27b6f48a-plugins-config-volume" pod "a02e27db-2cc5-404f-8881-6efd27b6f48a" (UID: "a02e27db-2cc5-404f-8881-6efd27b6f48a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:22:56.088550 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:56.088517    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b38dbda1-9a06-461d-8d0a-a9413b76d333-workload-certs" pod "b38dbda1-9a06-461d-8d0a-a9413b76d333" (UID: "b38dbda1-9a06-461d-8d0a-a9413b76d333"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:22:56.090166 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:56.090146    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b38dbda1-9a06-461d-8d0a-a9413b76d333-istio-podinfo" pod "b38dbda1-9a06-461d-8d0a-a9413b76d333" (UID: "b38dbda1-9a06-461d-8d0a-a9413b76d333"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:22:56.090344 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:56.090307    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a02e27db-2cc5-404f-8881-6efd27b6f48a-kube-api-access-jrj6d" pod "a02e27db-2cc5-404f-8881-6efd27b6f48a" (UID: "a02e27db-2cc5-404f-8881-6efd27b6f48a"). InnerVolumeSpecName "kube-api-access-jrj6d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:22:56.090638 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:56.090598    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b38dbda1-9a06-461d-8d0a-a9413b76d333-kube-api-access-vt5w8" pod "b38dbda1-9a06-461d-8d0a-a9413b76d333" (UID: "b38dbda1-9a06-461d-8d0a-a9413b76d333"). InnerVolumeSpecName "kube-api-access-vt5w8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:22:56.091397 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:56.091360    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b38dbda1-9a06-461d-8d0a-a9413b76d333-istio-token" pod "b38dbda1-9a06-461d-8d0a-a9413b76d333" (UID: "b38dbda1-9a06-461d-8d0a-a9413b76d333"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:22:56.099651 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:56.099239    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-ca5363/distinct-2-w4-ca5363-kserve-576576f9b5-xkkqc"]
Oct 30 01:22:56.099651 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:56.092715    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b38dbda1-9a06-461d-8d0a-a9413b76d333-istio-envoy" pod "b38dbda1-9a06-461d-8d0a-a9413b76d333" (UID: "b38dbda1-9a06-461d-8d0a-a9413b76d333"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:22:56.187458 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:56.187413    5002 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b38dbda1-9a06-461d-8d0a-a9413b76d333-istio-token\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:22:56.187671 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:56.187481    5002 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b38dbda1-9a06-461d-8d0a-a9413b76d333-istiod-ca-cert\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:22:56.187671 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:56.187499    5002 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b38dbda1-9a06-461d-8d0a-a9413b76d333-credential-socket\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:22:56.187671 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:56.187519    5002 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a02e27db-2cc5-404f-8881-6efd27b6f48a-plugins-config-volume\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:22:56.187671 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:56.187534    5002 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b38dbda1-9a06-461d-8d0a-a9413b76d333-istio-envoy\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:22:56.187671 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:56.187559    5002 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b38dbda1-9a06-461d-8d0a-a9413b76d333-istio-podinfo\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:22:56.187671 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:56.187572    5002 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b38dbda1-9a06-461d-8d0a-a9413b76d333-workload-certs\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:22:56.187671 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:56.187585    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vt5w8\" (UniqueName: \"kubernetes.io/projected/b38dbda1-9a06-461d-8d0a-a9413b76d333-kube-api-access-vt5w8\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:22:56.187671 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:56.187598    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jrj6d\" (UniqueName: \"kubernetes.io/projected/a02e27db-2cc5-404f-8881-6efd27b6f48a-kube-api-access-jrj6d\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:22:56.218995 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:56.218867    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-1e8c0e-openshift-default-6fb676f64f-ld62q"]
Oct 30 01:22:56.219196 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:56.219145    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-1e8c0e-openshift-default-6fb676f64f-ld62q" podUID="e09703cf-6ee3-45e9-893d-5a626ca636c9" containerName="istio-proxy" containerID="cri-o://7efdd2fa7f3ce097256c5cf2e582e81107eea9a013d1a6e703608ecd06c063b6" gracePeriod=30
Oct 30 01:22:56.240982 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:56.240951    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-b73232-openshift-default-59c5464ccc-2kpc7"
Oct 30 01:22:56.288756 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:56.288724    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c2a594b-3bbf-4c94-8a9f-7b79c663f135-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c2a594b-3bbf-4c94-8a9f-7b79c663f135-istio-data\") pod \"8c2a594b-3bbf-4c94-8a9f-7b79c663f135\" (UID: \"8c2a594b-3bbf-4c94-8a9f-7b79c663f135\") "
Oct 30 01:22:56.288982 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:56.288777    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8c2a594b-3bbf-4c94-8a9f-7b79c663f135-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c2a594b-3bbf-4c94-8a9f-7b79c663f135-istio-podinfo\") pod \"8c2a594b-3bbf-4c94-8a9f-7b79c663f135\" (UID: \"8c2a594b-3bbf-4c94-8a9f-7b79c663f135\") "
Oct 30 01:22:56.288982 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:56.288809    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c2a594b-3bbf-4c94-8a9f-7b79c663f135-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c2a594b-3bbf-4c94-8a9f-7b79c663f135-workload-socket\") pod \"8c2a594b-3bbf-4c94-8a9f-7b79c663f135\" (UID: \"8c2a594b-3bbf-4c94-8a9f-7b79c663f135\") "
Oct 30 01:22:56.288982 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:56.288840    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c2a594b-3bbf-4c94-8a9f-7b79c663f135-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c2a594b-3bbf-4c94-8a9f-7b79c663f135-istio-envoy\") pod \"8c2a594b-3bbf-4c94-8a9f-7b79c663f135\" (UID: \"8c2a594b-3bbf-4c94-8a9f-7b79c663f135\") "
Oct 30 01:22:56.288982 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:56.288869    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c2a594b-3bbf-4c94-8a9f-7b79c663f135-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c2a594b-3bbf-4c94-8a9f-7b79c663f135-credential-socket\") pod \"8c2a594b-3bbf-4c94-8a9f-7b79c663f135\" (UID: \"8c2a594b-3bbf-4c94-8a9f-7b79c663f135\") "
Oct 30 01:22:56.288982 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:56.288963    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c2a594b-3bbf-4c94-8a9f-7b79c663f135-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c2a594b-3bbf-4c94-8a9f-7b79c663f135-workload-certs\") pod \"8c2a594b-3bbf-4c94-8a9f-7b79c663f135\" (UID: \"8c2a594b-3bbf-4c94-8a9f-7b79c663f135\") "
Oct 30 01:22:56.289268 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:56.288993    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c2a594b-3bbf-4c94-8a9f-7b79c663f135-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c2a594b-3bbf-4c94-8a9f-7b79c663f135-istiod-ca-cert\") pod \"8c2a594b-3bbf-4c94-8a9f-7b79c663f135\" (UID: \"8c2a594b-3bbf-4c94-8a9f-7b79c663f135\") "
Oct 30 01:22:56.289268 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:56.289027    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c2a594b-3bbf-4c94-8a9f-7b79c663f135-istio-token\" (UniqueName: \"kubernetes.io/projected/8c2a594b-3bbf-4c94-8a9f-7b79c663f135-istio-token\") pod \"8c2a594b-3bbf-4c94-8a9f-7b79c663f135\" (UID: \"8c2a594b-3bbf-4c94-8a9f-7b79c663f135\") "
Oct 30 01:22:56.289268 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:56.289022    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c2a594b-3bbf-4c94-8a9f-7b79c663f135-istio-data" pod "8c2a594b-3bbf-4c94-8a9f-7b79c663f135" (UID: "8c2a594b-3bbf-4c94-8a9f-7b79c663f135"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:22:56.289268 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:56.289098    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c2a594b-3bbf-4c94-8a9f-7b79c663f135-workload-socket" pod "8c2a594b-3bbf-4c94-8a9f-7b79c663f135" (UID: "8c2a594b-3bbf-4c94-8a9f-7b79c663f135"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:22:56.289268 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:56.289154    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c2a594b-3bbf-4c94-8a9f-7b79c663f135-kube-api-access-jj6kb\" (UniqueName: \"kubernetes.io/projected/8c2a594b-3bbf-4c94-8a9f-7b79c663f135-kube-api-access-jj6kb\") pod \"8c2a594b-3bbf-4c94-8a9f-7b79c663f135\" (UID: \"8c2a594b-3bbf-4c94-8a9f-7b79c663f135\") "
Oct 30 01:22:56.289540 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:56.289263    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c2a594b-3bbf-4c94-8a9f-7b79c663f135-workload-certs" pod "8c2a594b-3bbf-4c94-8a9f-7b79c663f135" (UID: "8c2a594b-3bbf-4c94-8a9f-7b79c663f135"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:22:56.289540 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:56.289302    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c2a594b-3bbf-4c94-8a9f-7b79c663f135-credential-socket" pod "8c2a594b-3bbf-4c94-8a9f-7b79c663f135" (UID: "8c2a594b-3bbf-4c94-8a9f-7b79c663f135"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:22:56.289643 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:56.289616    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c2a594b-3bbf-4c94-8a9f-7b79c663f135-istiod-ca-cert" pod "8c2a594b-3bbf-4c94-8a9f-7b79c663f135" (UID: "8c2a594b-3bbf-4c94-8a9f-7b79c663f135"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:22:56.291817 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:56.290011    5002 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c2a594b-3bbf-4c94-8a9f-7b79c663f135-workload-socket\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:22:56.291952 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:56.291830    5002 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c2a594b-3bbf-4c94-8a9f-7b79c663f135-credential-socket\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:22:56.291952 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:56.291846    5002 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c2a594b-3bbf-4c94-8a9f-7b79c663f135-workload-certs\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:22:56.291952 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:56.291860    5002 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c2a594b-3bbf-4c94-8a9f-7b79c663f135-istiod-ca-cert\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:22:56.291952 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:56.291874    5002 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c2a594b-3bbf-4c94-8a9f-7b79c663f135-istio-data\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:22:56.292223 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:56.292183    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8c2a594b-3bbf-4c94-8a9f-7b79c663f135-istio-podinfo" pod "8c2a594b-3bbf-4c94-8a9f-7b79c663f135" (UID: "8c2a594b-3bbf-4c94-8a9f-7b79c663f135"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:22:56.292410 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:56.292357    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c2a594b-3bbf-4c94-8a9f-7b79c663f135-kube-api-access-jj6kb" pod "8c2a594b-3bbf-4c94-8a9f-7b79c663f135" (UID: "8c2a594b-3bbf-4c94-8a9f-7b79c663f135"). InnerVolumeSpecName "kube-api-access-jj6kb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:22:56.292510 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:56.292495    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c2a594b-3bbf-4c94-8a9f-7b79c663f135-istio-envoy" pod "8c2a594b-3bbf-4c94-8a9f-7b79c663f135" (UID: "8c2a594b-3bbf-4c94-8a9f-7b79c663f135"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:22:56.292623 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:56.292606    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c2a594b-3bbf-4c94-8a9f-7b79c663f135-istio-token" pod "8c2a594b-3bbf-4c94-8a9f-7b79c663f135" (UID: "8c2a594b-3bbf-4c94-8a9f-7b79c663f135"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:22:56.392173 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:56.392140    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-73435a-7645c9bc64-wcm9t"]
Oct 30 01:22:56.394668 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:56.394635    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jj6kb\" (UniqueName: \"kubernetes.io/projected/8c2a594b-3bbf-4c94-8a9f-7b79c663f135-kube-api-access-jj6kb\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:22:56.394871 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:56.394850    5002 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c2a594b-3bbf-4c94-8a9f-7b79c663f135-istio-podinfo\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:22:56.395005 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:56.394990    5002 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c2a594b-3bbf-4c94-8a9f-7b79c663f135-istio-envoy\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:22:56.395117 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:56.395104    5002 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c2a594b-3bbf-4c94-8a9f-7b79c663f135-istio-token\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:22:56.398317 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:56.398292    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-73435a-7645c9bc64-wcm9t"]
Oct 30 01:22:56.497207 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:56.497173    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60bf7933-a5cb-447a-96cc-e28389894a20" path="/var/lib/kubelet/pods/60bf7933-a5cb-447a-96cc-e28389894a20/volumes"
Oct 30 01:22:56.732404 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:56.732361    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-cbs5z"
Oct 30 01:22:56.800307 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:56.800263    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f86c006d-5087-4443-b907-c2a30815b206-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f86c006d-5087-4443-b907-c2a30815b206-workload-socket\") pod \"f86c006d-5087-4443-b907-c2a30815b206\" (UID: \"f86c006d-5087-4443-b907-c2a30815b206\") "
Oct 30 01:22:56.800532 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:56.800366    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f86c006d-5087-4443-b907-c2a30815b206-kube-api-access-8lmhz\" (UniqueName: \"kubernetes.io/projected/f86c006d-5087-4443-b907-c2a30815b206-kube-api-access-8lmhz\") pod \"f86c006d-5087-4443-b907-c2a30815b206\" (UID: \"f86c006d-5087-4443-b907-c2a30815b206\") "
Oct 30 01:22:56.800532 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:56.800460    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f86c006d-5087-4443-b907-c2a30815b206-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f86c006d-5087-4443-b907-c2a30815b206-istio-envoy\") pod \"f86c006d-5087-4443-b907-c2a30815b206\" (UID: \"f86c006d-5087-4443-b907-c2a30815b206\") "
Oct 30 01:22:56.800532 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:56.800493    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f86c006d-5087-4443-b907-c2a30815b206-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f86c006d-5087-4443-b907-c2a30815b206-istio-podinfo\") pod \"f86c006d-5087-4443-b907-c2a30815b206\" (UID: \"f86c006d-5087-4443-b907-c2a30815b206\") "
Oct 30 01:22:56.800704 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:56.800535    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f86c006d-5087-4443-b907-c2a30815b206-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f86c006d-5087-4443-b907-c2a30815b206-credential-socket\") pod \"f86c006d-5087-4443-b907-c2a30815b206\" (UID: \"f86c006d-5087-4443-b907-c2a30815b206\") "
Oct 30 01:22:56.800704 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:56.800586    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f86c006d-5087-4443-b907-c2a30815b206-istio-token\" (UniqueName: \"kubernetes.io/projected/f86c006d-5087-4443-b907-c2a30815b206-istio-token\") pod \"f86c006d-5087-4443-b907-c2a30815b206\" (UID: \"f86c006d-5087-4443-b907-c2a30815b206\") "
Oct 30 01:22:56.800704 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:56.800654    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f86c006d-5087-4443-b907-c2a30815b206-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f86c006d-5087-4443-b907-c2a30815b206-workload-certs\") pod \"f86c006d-5087-4443-b907-c2a30815b206\" (UID: \"f86c006d-5087-4443-b907-c2a30815b206\") "
Oct 30 01:22:56.800704 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:56.800687    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f86c006d-5087-4443-b907-c2a30815b206-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f86c006d-5087-4443-b907-c2a30815b206-istio-data\") pod \"f86c006d-5087-4443-b907-c2a30815b206\" (UID: \"f86c006d-5087-4443-b907-c2a30815b206\") "
Oct 30 01:22:56.800920 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:56.800746    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f86c006d-5087-4443-b907-c2a30815b206-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f86c006d-5087-4443-b907-c2a30815b206-istiod-ca-cert\") pod \"f86c006d-5087-4443-b907-c2a30815b206\" (UID: \"f86c006d-5087-4443-b907-c2a30815b206\") "
Oct 30 01:22:56.801638 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:56.801605    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f86c006d-5087-4443-b907-c2a30815b206-istiod-ca-cert" pod "f86c006d-5087-4443-b907-c2a30815b206" (UID: "f86c006d-5087-4443-b907-c2a30815b206"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:22:56.801943 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:56.801918    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f86c006d-5087-4443-b907-c2a30815b206-workload-socket" pod "f86c006d-5087-4443-b907-c2a30815b206" (UID: "f86c006d-5087-4443-b907-c2a30815b206"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:22:56.802986 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:56.802783    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f86c006d-5087-4443-b907-c2a30815b206-credential-socket" pod "f86c006d-5087-4443-b907-c2a30815b206" (UID: "f86c006d-5087-4443-b907-c2a30815b206"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:22:56.804011 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:56.803966    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f86c006d-5087-4443-b907-c2a30815b206-istio-data" pod "f86c006d-5087-4443-b907-c2a30815b206" (UID: "f86c006d-5087-4443-b907-c2a30815b206"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:22:56.804351 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:56.804331    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f86c006d-5087-4443-b907-c2a30815b206-workload-certs" pod "f86c006d-5087-4443-b907-c2a30815b206" (UID: "f86c006d-5087-4443-b907-c2a30815b206"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:22:56.808777 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:56.808727    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f86c006d-5087-4443-b907-c2a30815b206-istio-podinfo" pod "f86c006d-5087-4443-b907-c2a30815b206" (UID: "f86c006d-5087-4443-b907-c2a30815b206"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:22:56.808777 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:56.808734    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f86c006d-5087-4443-b907-c2a30815b206-istio-envoy" pod "f86c006d-5087-4443-b907-c2a30815b206" (UID: "f86c006d-5087-4443-b907-c2a30815b206"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:22:56.809086 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:56.809061    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f86c006d-5087-4443-b907-c2a30815b206-kube-api-access-8lmhz" pod "f86c006d-5087-4443-b907-c2a30815b206" (UID: "f86c006d-5087-4443-b907-c2a30815b206"). InnerVolumeSpecName "kube-api-access-8lmhz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:22:56.810915 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:56.810889    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f86c006d-5087-4443-b907-c2a30815b206-istio-token" pod "f86c006d-5087-4443-b907-c2a30815b206" (UID: "f86c006d-5087-4443-b907-c2a30815b206"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:22:56.902719 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:56.902649    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8lmhz\" (UniqueName: \"kubernetes.io/projected/f86c006d-5087-4443-b907-c2a30815b206-kube-api-access-8lmhz\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:22:56.902719 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:56.902723    5002 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f86c006d-5087-4443-b907-c2a30815b206-istio-envoy\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:22:56.902973 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:56.902739    5002 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f86c006d-5087-4443-b907-c2a30815b206-istio-podinfo\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:22:56.902973 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:56.902767    5002 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f86c006d-5087-4443-b907-c2a30815b206-credential-socket\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:22:56.902973 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:56.902782    5002 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f86c006d-5087-4443-b907-c2a30815b206-istio-token\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:22:56.902973 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:56.902794    5002 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f86c006d-5087-4443-b907-c2a30815b206-workload-certs\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:22:56.902973 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:56.902830    5002 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f86c006d-5087-4443-b907-c2a30815b206-istio-data\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:22:56.902973 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:56.902925    5002 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f86c006d-5087-4443-b907-c2a30815b206-istiod-ca-cert\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:22:56.902973 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:56.902949    5002 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f86c006d-5087-4443-b907-c2a30815b206-workload-socket\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:22:56.998556 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:56.998460    5002 generic.go:356] "Generic (PLEG): container finished" podID="f86c006d-5087-4443-b907-c2a30815b206" containerID="3f464cc547370328ffeb0384981e8e49111bd27743b52f2b77406696aa09d5c5" exitCode=0
Oct 30 01:22:56.998556 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:56.998516    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-cbs5z"
Oct 30 01:22:56.999098 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:56.998551    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-cbs5z" event={"ID":"f86c006d-5087-4443-b907-c2a30815b206","Type":"ContainerDied","Data":"3f464cc547370328ffeb0384981e8e49111bd27743b52f2b77406696aa09d5c5"}
Oct 30 01:22:56.999098 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:56.998642    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-cbs5z" event={"ID":"f86c006d-5087-4443-b907-c2a30815b206","Type":"ContainerDied","Data":"429886fee08c7d8302aa2f9c2f0a2c22cc4e6cbfd9d5924cb3a83d09a0fae90a"}
Oct 30 01:22:56.999098 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:56.998684    5002 scope.go:122] "RemoveContainer" containerID="3f464cc547370328ffeb0384981e8e49111bd27743b52f2b77406696aa09d5c5"
Oct 30 01:22:57.001188 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:57.001169    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-73435a-openshift-default-fc454477f-x6g2b"
Oct 30 01:22:57.001305 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:57.001205    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-73435a-openshift-default-fc454477f-x6g2b" event={"ID":"b38dbda1-9a06-461d-8d0a-a9413b76d333","Type":"ContainerDied","Data":"a7f730f453f33fe99b5976f4ee5b7f95453d1879fb7ee44d631f94659d76ca1f"}
Oct 30 01:22:57.004193 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:57.003296    5002 generic.go:356] "Generic (PLEG): container finished" podID="8c2a594b-3bbf-4c94-8a9f-7b79c663f135" containerID="24749d74a59c84ce731b6945d64b5d7397191f25a67a3b0836dfac7a2c54ff95" exitCode=0
Oct 30 01:22:57.004193 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:57.003344    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-b73232-openshift-default-59c5464ccc-2kpc7"
Oct 30 01:22:57.004193 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:57.003399    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-b73232-openshift-default-59c5464ccc-2kpc7" event={"ID":"8c2a594b-3bbf-4c94-8a9f-7b79c663f135","Type":"ContainerDied","Data":"24749d74a59c84ce731b6945d64b5d7397191f25a67a3b0836dfac7a2c54ff95"}
Oct 30 01:22:57.004193 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:57.003452    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-b73232-openshift-default-59c5464ccc-2kpc7" event={"ID":"8c2a594b-3bbf-4c94-8a9f-7b79c663f135","Type":"ContainerDied","Data":"d2f99feff5edcfe93cfb3b1b5f1d9dd7a5ed2a65f48529e130f59cf413bb7991"}
Oct 30 01:22:57.006088 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:57.006066    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-73435a-766b94d975-d8ztv" event={"ID":"a02e27db-2cc5-404f-8881-6efd27b6f48a","Type":"ContainerDied","Data":"39f0f7487f982bfcd27d88f5ee72f1d68f9a4a8f022111413ec11297d1f372c3"}
Oct 30 01:22:57.006192 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:57.006108    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-73435a-766b94d975-d8ztv"
Oct 30 01:22:57.012179 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:57.012163    5002 scope.go:122] "RemoveContainer" containerID="3f464cc547370328ffeb0384981e8e49111bd27743b52f2b77406696aa09d5c5"
Oct 30 01:22:57.012450 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:22:57.012430    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f464cc547370328ffeb0384981e8e49111bd27743b52f2b77406696aa09d5c5\": container with ID starting with 3f464cc547370328ffeb0384981e8e49111bd27743b52f2b77406696aa09d5c5 not found: ID does not exist" containerID="3f464cc547370328ffeb0384981e8e49111bd27743b52f2b77406696aa09d5c5"
Oct 30 01:22:57.012505 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:57.012458    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f464cc547370328ffeb0384981e8e49111bd27743b52f2b77406696aa09d5c5"} err="failed to get container status \"3f464cc547370328ffeb0384981e8e49111bd27743b52f2b77406696aa09d5c5\": rpc error: code = NotFound desc = could not find container \"3f464cc547370328ffeb0384981e8e49111bd27743b52f2b77406696aa09d5c5\": container with ID starting with 3f464cc547370328ffeb0384981e8e49111bd27743b52f2b77406696aa09d5c5 not found: ID does not exist"
Oct 30 01:22:57.012505 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:57.012475    5002 scope.go:122] "RemoveContainer" containerID="83add8a04a5e9f8469035acafb8674f0fbc3bf3e48da05e04540c95c8ba8038d"
Oct 30 01:22:57.025523 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:57.025495    5002 scope.go:122] "RemoveContainer" containerID="24749d74a59c84ce731b6945d64b5d7397191f25a67a3b0836dfac7a2c54ff95"
Oct 30 01:22:57.033330 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:57.033305    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5719d2a-tl4ff"]
Oct 30 01:22:57.037514 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:57.037485    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5719d2a-tl4ff"]
Oct 30 01:22:57.038625 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:57.038605    5002 scope.go:122] "RemoveContainer" containerID="bbec892f4b363921edf12e2be788aa12e7a2b4eda9645e63b5ecb9fdadb3d557"
Oct 30 01:22:57.048709 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:57.048683    5002 scope.go:122] "RemoveContainer" containerID="24749d74a59c84ce731b6945d64b5d7397191f25a67a3b0836dfac7a2c54ff95"
Oct 30 01:22:57.049008 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:22:57.048981    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24749d74a59c84ce731b6945d64b5d7397191f25a67a3b0836dfac7a2c54ff95\": container with ID starting with 24749d74a59c84ce731b6945d64b5d7397191f25a67a3b0836dfac7a2c54ff95 not found: ID does not exist" containerID="24749d74a59c84ce731b6945d64b5d7397191f25a67a3b0836dfac7a2c54ff95"
Oct 30 01:22:57.049073 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:57.049014    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24749d74a59c84ce731b6945d64b5d7397191f25a67a3b0836dfac7a2c54ff95"} err="failed to get container status \"24749d74a59c84ce731b6945d64b5d7397191f25a67a3b0836dfac7a2c54ff95\": rpc error: code = NotFound desc = could not find container \"24749d74a59c84ce731b6945d64b5d7397191f25a67a3b0836dfac7a2c54ff95\": container with ID starting with 24749d74a59c84ce731b6945d64b5d7397191f25a67a3b0836dfac7a2c54ff95 not found: ID does not exist"
Oct 30 01:22:57.049073 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:57.049032    5002 scope.go:122] "RemoveContainer" containerID="bbec892f4b363921edf12e2be788aa12e7a2b4eda9645e63b5ecb9fdadb3d557"
Oct 30 01:22:57.049324 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:22:57.049303    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bbec892f4b363921edf12e2be788aa12e7a2b4eda9645e63b5ecb9fdadb3d557\": container with ID starting with bbec892f4b363921edf12e2be788aa12e7a2b4eda9645e63b5ecb9fdadb3d557 not found: ID does not exist" containerID="bbec892f4b363921edf12e2be788aa12e7a2b4eda9645e63b5ecb9fdadb3d557"
Oct 30 01:22:57.049368 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:57.049331    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bbec892f4b363921edf12e2be788aa12e7a2b4eda9645e63b5ecb9fdadb3d557"} err="failed to get container status \"bbec892f4b363921edf12e2be788aa12e7a2b4eda9645e63b5ecb9fdadb3d557\": rpc error: code = NotFound desc = could not find container \"bbec892f4b363921edf12e2be788aa12e7a2b4eda9645e63b5ecb9fdadb3d557\": container with ID starting with bbec892f4b363921edf12e2be788aa12e7a2b4eda9645e63b5ecb9fdadb3d557 not found: ID does not exist"
Oct 30 01:22:57.049368 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:57.049347    5002 scope.go:122] "RemoveContainer" containerID="ae3ab6cc2a1adc5d05878cd80fb856333b98280bfd735d3b21b250cb1185a37a"
Oct 30 01:22:57.059723 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:57.059696    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-73435a-766b94d975-d8ztv"]
Oct 30 01:22:57.062909 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:57.062880    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-73435a-766b94d975-d8ztv"]
Oct 30 01:22:57.097459 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:57.097425    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-73435a-openshift-default-fc454477f-x6g2b"]
Oct 30 01:22:57.101223 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:57.101196    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-73435a-openshift-default-fc454477f-x6g2b"]
Oct 30 01:22:57.138850 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:57.138817    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-cbs5z"]
Oct 30 01:22:57.143898 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:57.143869    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-cbs5z"]
Oct 30 01:22:57.185770 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:57.185732    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-b73232-openshift-default-59c5464ccc-2kpc7"]
Oct 30 01:22:57.188376 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:57.188344    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-b73232-openshift-default-59c5464ccc-2kpc7"]
Oct 30 01:22:58.031119 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:58.031073    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b28e50b7-j7qkn"]
Oct 30 01:22:58.032273 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:58.032248    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b28e50b7-j7qkn"]
Oct 30 01:22:58.495523 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:58.495485    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17eaa864-381e-4d91-b358-862f1f709619" path="/var/lib/kubelet/pods/17eaa864-381e-4d91-b358-862f1f709619/volumes"
Oct 30 01:22:58.495984 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:58.495971    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c2a594b-3bbf-4c94-8a9f-7b79c663f135" path="/var/lib/kubelet/pods/8c2a594b-3bbf-4c94-8a9f-7b79c663f135/volumes"
Oct 30 01:22:58.496373 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:58.496362    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ed8b340-cc1d-4a9a-a662-e5a01147829b" path="/var/lib/kubelet/pods/8ed8b340-cc1d-4a9a-a662-e5a01147829b/volumes"
Oct 30 01:22:58.497035 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:58.497020    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a02e27db-2cc5-404f-8881-6efd27b6f48a" path="/var/lib/kubelet/pods/a02e27db-2cc5-404f-8881-6efd27b6f48a/volumes"
Oct 30 01:22:58.497469 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:58.497455    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b38dbda1-9a06-461d-8d0a-a9413b76d333" path="/var/lib/kubelet/pods/b38dbda1-9a06-461d-8d0a-a9413b76d333/volumes"
Oct 30 01:22:58.497883 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:58.497871    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f86c006d-5087-4443-b907-c2a30815b206" path="/var/lib/kubelet/pods/f86c006d-5087-4443-b907-c2a30815b206/volumes"
Oct 30 01:22:59.124571 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:59.124526    5002 patch_prober.go:29] interesting pod/e2e-ipp-b-1e8c0e-openshift-default-6fb676f64f-ld62q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 01:22:59.124999 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:22:59.124595    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-1e8c0e-openshift-default-6fb676f64f-ld62q" podUID="e09703cf-6ee3-45e9-893d-5a626ca636c9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 01:23:01.510094 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:01.510029    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-1e8c0e-openshift-default-6fb676f64f-ld62q"
Oct 30 01:23:01.569090 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:01.563762    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e09703cf-6ee3-45e9-893d-5a626ca636c9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e09703cf-6ee3-45e9-893d-5a626ca636c9-istio-data\") pod \"e09703cf-6ee3-45e9-893d-5a626ca636c9\" (UID: \"e09703cf-6ee3-45e9-893d-5a626ca636c9\") "
Oct 30 01:23:01.569090 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:01.563856    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e09703cf-6ee3-45e9-893d-5a626ca636c9-kube-api-access-vz89v\" (UniqueName: \"kubernetes.io/projected/e09703cf-6ee3-45e9-893d-5a626ca636c9-kube-api-access-vz89v\") pod \"e09703cf-6ee3-45e9-893d-5a626ca636c9\" (UID: \"e09703cf-6ee3-45e9-893d-5a626ca636c9\") "
Oct 30 01:23:01.569090 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:01.563895    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e09703cf-6ee3-45e9-893d-5a626ca636c9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e09703cf-6ee3-45e9-893d-5a626ca636c9-workload-certs\") pod \"e09703cf-6ee3-45e9-893d-5a626ca636c9\" (UID: \"e09703cf-6ee3-45e9-893d-5a626ca636c9\") "
Oct 30 01:23:01.569090 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:01.564001    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e09703cf-6ee3-45e9-893d-5a626ca636c9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e09703cf-6ee3-45e9-893d-5a626ca636c9-istiod-ca-cert\") pod \"e09703cf-6ee3-45e9-893d-5a626ca636c9\" (UID: \"e09703cf-6ee3-45e9-893d-5a626ca636c9\") "
Oct 30 01:23:01.569090 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:01.564121    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e09703cf-6ee3-45e9-893d-5a626ca636c9-istio-token\" (UniqueName: \"kubernetes.io/projected/e09703cf-6ee3-45e9-893d-5a626ca636c9-istio-token\") pod \"e09703cf-6ee3-45e9-893d-5a626ca636c9\" (UID: \"e09703cf-6ee3-45e9-893d-5a626ca636c9\") "
Oct 30 01:23:01.569090 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:01.564159    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e09703cf-6ee3-45e9-893d-5a626ca636c9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e09703cf-6ee3-45e9-893d-5a626ca636c9-credential-socket\") pod \"e09703cf-6ee3-45e9-893d-5a626ca636c9\" (UID: \"e09703cf-6ee3-45e9-893d-5a626ca636c9\") "
Oct 30 01:23:01.569090 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:01.564198    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e09703cf-6ee3-45e9-893d-5a626ca636c9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e09703cf-6ee3-45e9-893d-5a626ca636c9-workload-socket\") pod \"e09703cf-6ee3-45e9-893d-5a626ca636c9\" (UID: \"e09703cf-6ee3-45e9-893d-5a626ca636c9\") "
Oct 30 01:23:01.569090 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:01.564277    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e09703cf-6ee3-45e9-893d-5a626ca636c9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e09703cf-6ee3-45e9-893d-5a626ca636c9-istio-envoy\") pod \"e09703cf-6ee3-45e9-893d-5a626ca636c9\" (UID: \"e09703cf-6ee3-45e9-893d-5a626ca636c9\") "
Oct 30 01:23:01.569090 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:01.564326    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e09703cf-6ee3-45e9-893d-5a626ca636c9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e09703cf-6ee3-45e9-893d-5a626ca636c9-istio-podinfo\") pod \"e09703cf-6ee3-45e9-893d-5a626ca636c9\" (UID: \"e09703cf-6ee3-45e9-893d-5a626ca636c9\") "
Oct 30 01:23:01.571191 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:01.570842    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e09703cf-6ee3-45e9-893d-5a626ca636c9-workload-socket" pod "e09703cf-6ee3-45e9-893d-5a626ca636c9" (UID: "e09703cf-6ee3-45e9-893d-5a626ca636c9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:23:01.572105 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:01.572078    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e09703cf-6ee3-45e9-893d-5a626ca636c9-istio-data" pod "e09703cf-6ee3-45e9-893d-5a626ca636c9" (UID: "e09703cf-6ee3-45e9-893d-5a626ca636c9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:23:01.577399 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:01.573222    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e09703cf-6ee3-45e9-893d-5a626ca636c9-istiod-ca-cert" pod "e09703cf-6ee3-45e9-893d-5a626ca636c9" (UID: "e09703cf-6ee3-45e9-893d-5a626ca636c9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:23:01.577399 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:01.573493    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e09703cf-6ee3-45e9-893d-5a626ca636c9-workload-certs" pod "e09703cf-6ee3-45e9-893d-5a626ca636c9" (UID: "e09703cf-6ee3-45e9-893d-5a626ca636c9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:23:01.577399 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:01.574033    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e09703cf-6ee3-45e9-893d-5a626ca636c9-credential-socket" pod "e09703cf-6ee3-45e9-893d-5a626ca636c9" (UID: "e09703cf-6ee3-45e9-893d-5a626ca636c9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:23:01.583334 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:01.583272    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e09703cf-6ee3-45e9-893d-5a626ca636c9-istio-token" pod "e09703cf-6ee3-45e9-893d-5a626ca636c9" (UID: "e09703cf-6ee3-45e9-893d-5a626ca636c9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:23:01.584116 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:01.584091    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e09703cf-6ee3-45e9-893d-5a626ca636c9-istio-envoy" pod "e09703cf-6ee3-45e9-893d-5a626ca636c9" (UID: "e09703cf-6ee3-45e9-893d-5a626ca636c9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:23:01.584234 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:01.584152    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e09703cf-6ee3-45e9-893d-5a626ca636c9-kube-api-access-vz89v" pod "e09703cf-6ee3-45e9-893d-5a626ca636c9" (UID: "e09703cf-6ee3-45e9-893d-5a626ca636c9"). InnerVolumeSpecName "kube-api-access-vz89v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:23:01.585821 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:01.585788    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e09703cf-6ee3-45e9-893d-5a626ca636c9-istio-podinfo" pod "e09703cf-6ee3-45e9-893d-5a626ca636c9" (UID: "e09703cf-6ee3-45e9-893d-5a626ca636c9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:23:01.671348 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:01.671311    5002 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e09703cf-6ee3-45e9-893d-5a626ca636c9-istio-podinfo\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:23:01.671348 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:01.671345    5002 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e09703cf-6ee3-45e9-893d-5a626ca636c9-istio-data\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:23:01.671598 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:01.671358    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vz89v\" (UniqueName: \"kubernetes.io/projected/e09703cf-6ee3-45e9-893d-5a626ca636c9-kube-api-access-vz89v\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:23:01.671598 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:01.671368    5002 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e09703cf-6ee3-45e9-893d-5a626ca636c9-workload-certs\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:23:01.671598 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:01.671414    5002 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e09703cf-6ee3-45e9-893d-5a626ca636c9-istiod-ca-cert\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:23:01.671598 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:01.671439    5002 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e09703cf-6ee3-45e9-893d-5a626ca636c9-istio-token\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:23:01.671598 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:01.671450    5002 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e09703cf-6ee3-45e9-893d-5a626ca636c9-credential-socket\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:23:01.671598 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:01.671470    5002 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e09703cf-6ee3-45e9-893d-5a626ca636c9-workload-socket\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:23:01.671598 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:01.671484    5002 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e09703cf-6ee3-45e9-893d-5a626ca636c9-istio-envoy\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:23:02.045186 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:02.045146    5002 generic.go:356] "Generic (PLEG): container finished" podID="e09703cf-6ee3-45e9-893d-5a626ca636c9" containerID="7efdd2fa7f3ce097256c5cf2e582e81107eea9a013d1a6e703608ecd06c063b6" exitCode=0
Oct 30 01:23:02.045520 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:02.045205    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-1e8c0e-openshift-default-6fb676f64f-ld62q"
Oct 30 01:23:02.045520 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:02.045235    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-1e8c0e-openshift-default-6fb676f64f-ld62q" event={"ID":"e09703cf-6ee3-45e9-893d-5a626ca636c9","Type":"ContainerDied","Data":"7efdd2fa7f3ce097256c5cf2e582e81107eea9a013d1a6e703608ecd06c063b6"}
Oct 30 01:23:02.045520 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:02.045307    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-1e8c0e-openshift-default-6fb676f64f-ld62q" event={"ID":"e09703cf-6ee3-45e9-893d-5a626ca636c9","Type":"ContainerDied","Data":"d018ff30bfa75b77c6559a8d2e3b8554b923eb11f396a978ffd9f6d9662bb134"}
Oct 30 01:23:02.045520 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:02.045338    5002 scope.go:122] "RemoveContainer" containerID="7efdd2fa7f3ce097256c5cf2e582e81107eea9a013d1a6e703608ecd06c063b6"
Oct 30 01:23:02.057602 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:02.057572    5002 scope.go:122] "RemoveContainer" containerID="4ffab9d9a6fd83e8b2639aaf218cbf94607909f3ff720ba0d2aa6083b0d37bcd"
Oct 30 01:23:02.067691 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:02.067663    5002 scope.go:122] "RemoveContainer" containerID="7efdd2fa7f3ce097256c5cf2e582e81107eea9a013d1a6e703608ecd06c063b6"
Oct 30 01:23:02.068010 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:23:02.067988    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7efdd2fa7f3ce097256c5cf2e582e81107eea9a013d1a6e703608ecd06c063b6\": container with ID starting with 7efdd2fa7f3ce097256c5cf2e582e81107eea9a013d1a6e703608ecd06c063b6 not found: ID does not exist" containerID="7efdd2fa7f3ce097256c5cf2e582e81107eea9a013d1a6e703608ecd06c063b6"
Oct 30 01:23:02.068084 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:02.068018    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7efdd2fa7f3ce097256c5cf2e582e81107eea9a013d1a6e703608ecd06c063b6"} err="failed to get container status \"7efdd2fa7f3ce097256c5cf2e582e81107eea9a013d1a6e703608ecd06c063b6\": rpc error: code = NotFound desc = could not find container \"7efdd2fa7f3ce097256c5cf2e582e81107eea9a013d1a6e703608ecd06c063b6\": container with ID starting with 7efdd2fa7f3ce097256c5cf2e582e81107eea9a013d1a6e703608ecd06c063b6 not found: ID does not exist"
Oct 30 01:23:02.068084 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:02.068048    5002 scope.go:122] "RemoveContainer" containerID="4ffab9d9a6fd83e8b2639aaf218cbf94607909f3ff720ba0d2aa6083b0d37bcd"
Oct 30 01:23:02.068367 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:23:02.068339    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ffab9d9a6fd83e8b2639aaf218cbf94607909f3ff720ba0d2aa6083b0d37bcd\": container with ID starting with 4ffab9d9a6fd83e8b2639aaf218cbf94607909f3ff720ba0d2aa6083b0d37bcd not found: ID does not exist" containerID="4ffab9d9a6fd83e8b2639aaf218cbf94607909f3ff720ba0d2aa6083b0d37bcd"
Oct 30 01:23:02.068570 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:02.068374    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ffab9d9a6fd83e8b2639aaf218cbf94607909f3ff720ba0d2aa6083b0d37bcd"} err="failed to get container status \"4ffab9d9a6fd83e8b2639aaf218cbf94607909f3ff720ba0d2aa6083b0d37bcd\": rpc error: code = NotFound desc = could not find container \"4ffab9d9a6fd83e8b2639aaf218cbf94607909f3ff720ba0d2aa6083b0d37bcd\": container with ID starting with 4ffab9d9a6fd83e8b2639aaf218cbf94607909f3ff720ba0d2aa6083b0d37bcd not found: ID does not exist"
Oct 30 01:23:02.103793 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:02.103757    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-1e8c0e-openshift-default-6fb676f64f-ld62q"]
Oct 30 01:23:02.116603 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:02.116568    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-1e8c0e-openshift-default-6fb676f64f-ld62q"]
Oct 30 01:23:02.497011 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:02.496968    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e09703cf-6ee3-45e9-893d-5a626ca636c9" path="/var/lib/kubelet/pods/e09703cf-6ee3-45e9-893d-5a626ca636c9/volumes"
Oct 30 01:23:03.507784 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:03.507749    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-a3619c-openshift-default-675f567c95-jsx2v"]
Oct 30 01:23:03.508236 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:03.507969    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-a3619c-openshift-default-675f567c95-jsx2v" podUID="b9058f9e-a7e7-4f30-87be-e037e5378b3c" containerName="istio-proxy" containerID="cri-o://9e600f9dc55a970f153f67517d7cfcd8cc39a6ad43ac338a7c64cc453af40360" gracePeriod=30
Oct 30 01:23:05.989482 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:05.989444    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7d8z4"]
Oct 30 01:23:07.224483 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:07.224445    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f64dc0-b4f9c4899-cnjhz"]
Oct 30 01:23:07.438055 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:07.437971    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-cefb27/simulator-w3-cefb27-kserve-6b669c9d9f-zx2vm" podUID="7121e81e-f903-42c0-a7a4-bc18d49194e9" containerName="main" containerID="cri-o://a882c5a3943ec03c8f2fb3309811988b702de92703a7ca1d17eafa3b4a115adb" gracePeriod=45
Oct 30 01:23:07.707035 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:07.707005    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-cefb27/simulator-w3-cefb27-kserve-6b669c9d9f-zx2vm"
Oct 30 01:23:07.841837 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:07.841756    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7121e81e-f903-42c0-a7a4-bc18d49194e9-tls-certs\" (UniqueName: \"kubernetes.io/secret/7121e81e-f903-42c0-a7a4-bc18d49194e9-tls-certs\") pod \"7121e81e-f903-42c0-a7a4-bc18d49194e9\" (UID: \"7121e81e-f903-42c0-a7a4-bc18d49194e9\") "
Oct 30 01:23:07.841837 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:07.841832    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7121e81e-f903-42c0-a7a4-bc18d49194e9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7121e81e-f903-42c0-a7a4-bc18d49194e9-tmp-dir\") pod \"7121e81e-f903-42c0-a7a4-bc18d49194e9\" (UID: \"7121e81e-f903-42c0-a7a4-bc18d49194e9\") "
Oct 30 01:23:07.842034 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:07.841920    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7121e81e-f903-42c0-a7a4-bc18d49194e9-home\" (UniqueName: \"kubernetes.io/empty-dir/7121e81e-f903-42c0-a7a4-bc18d49194e9-home\") pod \"7121e81e-f903-42c0-a7a4-bc18d49194e9\" (UID: \"7121e81e-f903-42c0-a7a4-bc18d49194e9\") "
Oct 30 01:23:07.842034 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:07.841959    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7121e81e-f903-42c0-a7a4-bc18d49194e9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7121e81e-f903-42c0-a7a4-bc18d49194e9-dshm\") pod \"7121e81e-f903-42c0-a7a4-bc18d49194e9\" (UID: \"7121e81e-f903-42c0-a7a4-bc18d49194e9\") "
Oct 30 01:23:07.842126 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:07.842033    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7121e81e-f903-42c0-a7a4-bc18d49194e9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7121e81e-f903-42c0-a7a4-bc18d49194e9-model-cache\") pod \"7121e81e-f903-42c0-a7a4-bc18d49194e9\" (UID: \"7121e81e-f903-42c0-a7a4-bc18d49194e9\") "
Oct 30 01:23:07.842126 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:07.842100    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7121e81e-f903-42c0-a7a4-bc18d49194e9-kube-api-access-4pvsr\" (UniqueName: \"kubernetes.io/projected/7121e81e-f903-42c0-a7a4-bc18d49194e9-kube-api-access-4pvsr\") pod \"7121e81e-f903-42c0-a7a4-bc18d49194e9\" (UID: \"7121e81e-f903-42c0-a7a4-bc18d49194e9\") "
Oct 30 01:23:07.842249 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:07.842132    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7121e81e-f903-42c0-a7a4-bc18d49194e9-tmp-dir" pod "7121e81e-f903-42c0-a7a4-bc18d49194e9" (UID: "7121e81e-f903-42c0-a7a4-bc18d49194e9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:23:07.842249 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:07.842169    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7121e81e-f903-42c0-a7a4-bc18d49194e9-home" pod "7121e81e-f903-42c0-a7a4-bc18d49194e9" (UID: "7121e81e-f903-42c0-a7a4-bc18d49194e9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:23:07.842429 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:07.842404    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7121e81e-f903-42c0-a7a4-bc18d49194e9-model-cache" pod "7121e81e-f903-42c0-a7a4-bc18d49194e9" (UID: "7121e81e-f903-42c0-a7a4-bc18d49194e9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:23:07.842605 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:07.842587    5002 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7121e81e-f903-42c0-a7a4-bc18d49194e9-home\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:23:07.842670 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:07.842612    5002 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7121e81e-f903-42c0-a7a4-bc18d49194e9-model-cache\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:23:07.842670 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:07.842629    5002 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7121e81e-f903-42c0-a7a4-bc18d49194e9-tmp-dir\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:23:07.844528 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:07.844506    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7121e81e-f903-42c0-a7a4-bc18d49194e9-kube-api-access-4pvsr" pod "7121e81e-f903-42c0-a7a4-bc18d49194e9" (UID: "7121e81e-f903-42c0-a7a4-bc18d49194e9"). InnerVolumeSpecName "kube-api-access-4pvsr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:23:07.844677 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:07.844653    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7121e81e-f903-42c0-a7a4-bc18d49194e9-tls-certs" pod "7121e81e-f903-42c0-a7a4-bc18d49194e9" (UID: "7121e81e-f903-42c0-a7a4-bc18d49194e9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:23:07.844677 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:07.844661    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7121e81e-f903-42c0-a7a4-bc18d49194e9-dshm" pod "7121e81e-f903-42c0-a7a4-bc18d49194e9" (UID: "7121e81e-f903-42c0-a7a4-bc18d49194e9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:23:07.944054 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:07.944003    5002 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7121e81e-f903-42c0-a7a4-bc18d49194e9-dshm\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:23:07.944054 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:07.944051    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4pvsr\" (UniqueName: \"kubernetes.io/projected/7121e81e-f903-42c0-a7a4-bc18d49194e9-kube-api-access-4pvsr\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:23:07.944340 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:07.944068    5002 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7121e81e-f903-42c0-a7a4-bc18d49194e9-tls-certs\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:23:08.093943 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:08.093835    5002 generic.go:356] "Generic (PLEG): container finished" podID="7121e81e-f903-42c0-a7a4-bc18d49194e9" containerID="a882c5a3943ec03c8f2fb3309811988b702de92703a7ca1d17eafa3b4a115adb" exitCode=0
Oct 30 01:23:08.093943 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:08.093889    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-cefb27/simulator-w3-cefb27-kserve-6b669c9d9f-zx2vm"
Oct 30 01:23:08.093943 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:08.093915    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-cefb27/simulator-w3-cefb27-kserve-6b669c9d9f-zx2vm" event={"ID":"7121e81e-f903-42c0-a7a4-bc18d49194e9","Type":"ContainerDied","Data":"a882c5a3943ec03c8f2fb3309811988b702de92703a7ca1d17eafa3b4a115adb"}
Oct 30 01:23:08.094195 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:08.093973    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-cefb27/simulator-w3-cefb27-kserve-6b669c9d9f-zx2vm" event={"ID":"7121e81e-f903-42c0-a7a4-bc18d49194e9","Type":"ContainerDied","Data":"f8c2389e04582bc971a71271a75ed7ebb2c225cfa128ac171a1a25c5df26f926"}
Oct 30 01:23:08.094195 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:08.093992    5002 scope.go:122] "RemoveContainer" containerID="a882c5a3943ec03c8f2fb3309811988b702de92703a7ca1d17eafa3b4a115adb"
Oct 30 01:23:08.104446 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:08.104428    5002 scope.go:122] "RemoveContainer" containerID="a882c5a3943ec03c8f2fb3309811988b702de92703a7ca1d17eafa3b4a115adb"
Oct 30 01:23:08.104733 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:23:08.104710    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a882c5a3943ec03c8f2fb3309811988b702de92703a7ca1d17eafa3b4a115adb\": container with ID starting with a882c5a3943ec03c8f2fb3309811988b702de92703a7ca1d17eafa3b4a115adb not found: ID does not exist" containerID="a882c5a3943ec03c8f2fb3309811988b702de92703a7ca1d17eafa3b4a115adb"
Oct 30 01:23:08.104821 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:08.104739    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a882c5a3943ec03c8f2fb3309811988b702de92703a7ca1d17eafa3b4a115adb"} err="failed to get container status \"a882c5a3943ec03c8f2fb3309811988b702de92703a7ca1d17eafa3b4a115adb\": rpc error: code = NotFound desc = could not find container \"a882c5a3943ec03c8f2fb3309811988b702de92703a7ca1d17eafa3b4a115adb\": container with ID starting with a882c5a3943ec03c8f2fb3309811988b702de92703a7ca1d17eafa3b4a115adb not found: ID does not exist"
Oct 30 01:23:08.151178 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:08.151139    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-cefb27/simulator-w3-cefb27-kserve-6b669c9d9f-zx2vm"]
Oct 30 01:23:08.154395 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:08.154354    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-cefb27/simulator-w3-cefb27-kserve-6b669c9d9f-zx2vm"]
Oct 30 01:23:08.493991 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:08.493947    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7121e81e-f903-42c0-a7a4-bc18d49194e9" path="/var/lib/kubelet/pods/7121e81e-f903-42c0-a7a4-bc18d49194e9/volumes"
Oct 30 01:23:08.775327 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:08.775296    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-a3619c-openshift-default-675f567c95-jsx2v"
Oct 30 01:23:08.852965 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:08.852925    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9058f9e-a7e7-4f30-87be-e037e5378b3c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9058f9e-a7e7-4f30-87be-e037e5378b3c-credential-socket\") pod \"b9058f9e-a7e7-4f30-87be-e037e5378b3c\" (UID: \"b9058f9e-a7e7-4f30-87be-e037e5378b3c\") "
Oct 30 01:23:08.853176 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:08.852977    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9058f9e-a7e7-4f30-87be-e037e5378b3c-kube-api-access-7xhnc\" (UniqueName: \"kubernetes.io/projected/b9058f9e-a7e7-4f30-87be-e037e5378b3c-kube-api-access-7xhnc\") pod \"b9058f9e-a7e7-4f30-87be-e037e5378b3c\" (UID: \"b9058f9e-a7e7-4f30-87be-e037e5378b3c\") "
Oct 30 01:23:08.853176 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:08.853020    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9058f9e-a7e7-4f30-87be-e037e5378b3c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9058f9e-a7e7-4f30-87be-e037e5378b3c-workload-certs\") pod \"b9058f9e-a7e7-4f30-87be-e037e5378b3c\" (UID: \"b9058f9e-a7e7-4f30-87be-e037e5378b3c\") "
Oct 30 01:23:08.853176 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:08.853071    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9058f9e-a7e7-4f30-87be-e037e5378b3c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9058f9e-a7e7-4f30-87be-e037e5378b3c-istio-envoy\") pod \"b9058f9e-a7e7-4f30-87be-e037e5378b3c\" (UID: \"b9058f9e-a7e7-4f30-87be-e037e5378b3c\") "
Oct 30 01:23:08.853366 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:08.853178    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9058f9e-a7e7-4f30-87be-e037e5378b3c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9058f9e-a7e7-4f30-87be-e037e5378b3c-istio-data\") pod \"b9058f9e-a7e7-4f30-87be-e037e5378b3c\" (UID: \"b9058f9e-a7e7-4f30-87be-e037e5378b3c\") "
Oct 30 01:23:08.853366 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:08.853215    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9058f9e-a7e7-4f30-87be-e037e5378b3c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9058f9e-a7e7-4f30-87be-e037e5378b3c-workload-socket\") pod \"b9058f9e-a7e7-4f30-87be-e037e5378b3c\" (UID: \"b9058f9e-a7e7-4f30-87be-e037e5378b3c\") "
Oct 30 01:23:08.853366 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:08.853275    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9058f9e-a7e7-4f30-87be-e037e5378b3c-istio-token\" (UniqueName: \"kubernetes.io/projected/b9058f9e-a7e7-4f30-87be-e037e5378b3c-istio-token\") pod \"b9058f9e-a7e7-4f30-87be-e037e5378b3c\" (UID: \"b9058f9e-a7e7-4f30-87be-e037e5378b3c\") "
Oct 30 01:23:08.853366 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:08.853341    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9058f9e-a7e7-4f30-87be-e037e5378b3c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9058f9e-a7e7-4f30-87be-e037e5378b3c-istiod-ca-cert\") pod \"b9058f9e-a7e7-4f30-87be-e037e5378b3c\" (UID: \"b9058f9e-a7e7-4f30-87be-e037e5378b3c\") "
Oct 30 01:23:08.853606 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:08.853439    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b9058f9e-a7e7-4f30-87be-e037e5378b3c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9058f9e-a7e7-4f30-87be-e037e5378b3c-istio-podinfo\") pod \"b9058f9e-a7e7-4f30-87be-e037e5378b3c\" (UID: \"b9058f9e-a7e7-4f30-87be-e037e5378b3c\") "
Oct 30 01:23:08.853606 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:08.853472    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9058f9e-a7e7-4f30-87be-e037e5378b3c-workload-certs" pod "b9058f9e-a7e7-4f30-87be-e037e5378b3c" (UID: "b9058f9e-a7e7-4f30-87be-e037e5378b3c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:23:08.853606 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:08.853588    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9058f9e-a7e7-4f30-87be-e037e5378b3c-credential-socket" pod "b9058f9e-a7e7-4f30-87be-e037e5378b3c" (UID: "b9058f9e-a7e7-4f30-87be-e037e5378b3c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:23:08.853998 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:08.853732    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9058f9e-a7e7-4f30-87be-e037e5378b3c-istio-data" pod "b9058f9e-a7e7-4f30-87be-e037e5378b3c" (UID: "b9058f9e-a7e7-4f30-87be-e037e5378b3c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:23:08.854074 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:08.854017    5002 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9058f9e-a7e7-4f30-87be-e037e5378b3c-istio-data\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:23:08.854074 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:08.854038    5002 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9058f9e-a7e7-4f30-87be-e037e5378b3c-credential-socket\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:23:08.854074 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:08.854048    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9058f9e-a7e7-4f30-87be-e037e5378b3c-istiod-ca-cert" pod "b9058f9e-a7e7-4f30-87be-e037e5378b3c" (UID: "b9058f9e-a7e7-4f30-87be-e037e5378b3c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:23:08.854074 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:08.854054    5002 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9058f9e-a7e7-4f30-87be-e037e5378b3c-workload-certs\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:23:08.854352 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:08.854328    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9058f9e-a7e7-4f30-87be-e037e5378b3c-workload-socket" pod "b9058f9e-a7e7-4f30-87be-e037e5378b3c" (UID: "b9058f9e-a7e7-4f30-87be-e037e5378b3c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:23:08.856312 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:08.856273    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b9058f9e-a7e7-4f30-87be-e037e5378b3c-istio-podinfo" pod "b9058f9e-a7e7-4f30-87be-e037e5378b3c" (UID: "b9058f9e-a7e7-4f30-87be-e037e5378b3c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:23:08.856632 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:08.856605    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9058f9e-a7e7-4f30-87be-e037e5378b3c-istio-token" pod "b9058f9e-a7e7-4f30-87be-e037e5378b3c" (UID: "b9058f9e-a7e7-4f30-87be-e037e5378b3c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:23:08.856839 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:08.856819    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9058f9e-a7e7-4f30-87be-e037e5378b3c-kube-api-access-7xhnc" pod "b9058f9e-a7e7-4f30-87be-e037e5378b3c" (UID: "b9058f9e-a7e7-4f30-87be-e037e5378b3c"). InnerVolumeSpecName "kube-api-access-7xhnc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:23:08.857031 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:08.856997    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9058f9e-a7e7-4f30-87be-e037e5378b3c-istio-envoy" pod "b9058f9e-a7e7-4f30-87be-e037e5378b3c" (UID: "b9058f9e-a7e7-4f30-87be-e037e5378b3c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:23:08.962848 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:08.962654    5002 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9058f9e-a7e7-4f30-87be-e037e5378b3c-workload-socket\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:23:08.962848 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:08.962699    5002 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9058f9e-a7e7-4f30-87be-e037e5378b3c-istio-token\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:23:08.962848 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:08.962713    5002 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9058f9e-a7e7-4f30-87be-e037e5378b3c-istiod-ca-cert\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:23:08.962848 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:08.962726    5002 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9058f9e-a7e7-4f30-87be-e037e5378b3c-istio-podinfo\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:23:08.962848 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:08.962741    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7xhnc\" (UniqueName: \"kubernetes.io/projected/b9058f9e-a7e7-4f30-87be-e037e5378b3c-kube-api-access-7xhnc\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:23:08.962848 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:08.962755    5002 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9058f9e-a7e7-4f30-87be-e037e5378b3c-istio-envoy\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:23:09.103208 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:09.103117    5002 generic.go:356] "Generic (PLEG): container finished" podID="b9058f9e-a7e7-4f30-87be-e037e5378b3c" containerID="9e600f9dc55a970f153f67517d7cfcd8cc39a6ad43ac338a7c64cc453af40360" exitCode=0
Oct 30 01:23:09.103208 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:09.103166    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-a3619c-openshift-default-675f567c95-jsx2v"
Oct 30 01:23:09.103208 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:09.103180    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-a3619c-openshift-default-675f567c95-jsx2v" event={"ID":"b9058f9e-a7e7-4f30-87be-e037e5378b3c","Type":"ContainerDied","Data":"9e600f9dc55a970f153f67517d7cfcd8cc39a6ad43ac338a7c64cc453af40360"}
Oct 30 01:23:09.103508 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:09.103222    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-a3619c-openshift-default-675f567c95-jsx2v" event={"ID":"b9058f9e-a7e7-4f30-87be-e037e5378b3c","Type":"ContainerDied","Data":"e62ecc26493257d537196a656760ba3f7bb18d69a2786687be6bee286711fd41"}
Oct 30 01:23:09.103508 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:09.103244    5002 scope.go:122] "RemoveContainer" containerID="9e600f9dc55a970f153f67517d7cfcd8cc39a6ad43ac338a7c64cc453af40360"
Oct 30 01:23:09.114094 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:09.114073    5002 scope.go:122] "RemoveContainer" containerID="9e600f9dc55a970f153f67517d7cfcd8cc39a6ad43ac338a7c64cc453af40360"
Oct 30 01:23:09.114372 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:23:09.114351    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e600f9dc55a970f153f67517d7cfcd8cc39a6ad43ac338a7c64cc453af40360\": container with ID starting with 9e600f9dc55a970f153f67517d7cfcd8cc39a6ad43ac338a7c64cc453af40360 not found: ID does not exist" containerID="9e600f9dc55a970f153f67517d7cfcd8cc39a6ad43ac338a7c64cc453af40360"
Oct 30 01:23:09.114479 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:09.114408    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e600f9dc55a970f153f67517d7cfcd8cc39a6ad43ac338a7c64cc453af40360"} err="failed to get container status \"9e600f9dc55a970f153f67517d7cfcd8cc39a6ad43ac338a7c64cc453af40360\": rpc error: code = NotFound desc = could not find container \"9e600f9dc55a970f153f67517d7cfcd8cc39a6ad43ac338a7c64cc453af40360\": container with ID starting with 9e600f9dc55a970f153f67517d7cfcd8cc39a6ad43ac338a7c64cc453af40360 not found: ID does not exist"
Oct 30 01:23:09.161367 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:09.161329    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-a3619c-openshift-default-675f567c95-jsx2v"]
Oct 30 01:23:09.174795 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:09.174761    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-a3619c-openshift-default-675f567c95-jsx2v"]
Oct 30 01:23:10.502400 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:10.497587    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9058f9e-a7e7-4f30-87be-e037e5378b3c" path="/var/lib/kubelet/pods/b9058f9e-a7e7-4f30-87be-e037e5378b3c/volumes"
Oct 30 01:23:11.141103 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:11.141045    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-ca5363/distinct-2-w4-ca5363-kserve-576576f9b5-xkkqc" podUID="5de8bebf-cfad-49da-9a98-08aedde4442c" containerName="main" containerID="cri-o://799190f41cf9cd2205e5603084cd81d6af97fb7562ff18ef90213ac781319742" gracePeriod=45
Oct 30 01:23:11.413352 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:11.413326    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ca5363/distinct-2-w4-ca5363-kserve-576576f9b5-xkkqc"
Oct 30 01:23:11.495356 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:11.495308    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5de8bebf-cfad-49da-9a98-08aedde4442c-kube-api-access-5v78h\" (UniqueName: \"kubernetes.io/projected/5de8bebf-cfad-49da-9a98-08aedde4442c-kube-api-access-5v78h\") pod \"5de8bebf-cfad-49da-9a98-08aedde4442c\" (UID: \"5de8bebf-cfad-49da-9a98-08aedde4442c\") "
Oct 30 01:23:11.495584 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:11.495428    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5de8bebf-cfad-49da-9a98-08aedde4442c-home\" (UniqueName: \"kubernetes.io/empty-dir/5de8bebf-cfad-49da-9a98-08aedde4442c-home\") pod \"5de8bebf-cfad-49da-9a98-08aedde4442c\" (UID: \"5de8bebf-cfad-49da-9a98-08aedde4442c\") "
Oct 30 01:23:11.495584 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:11.495552    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5de8bebf-cfad-49da-9a98-08aedde4442c-tls-certs\" (UniqueName: \"kubernetes.io/secret/5de8bebf-cfad-49da-9a98-08aedde4442c-tls-certs\") pod \"5de8bebf-cfad-49da-9a98-08aedde4442c\" (UID: \"5de8bebf-cfad-49da-9a98-08aedde4442c\") "
Oct 30 01:23:11.495733 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:11.495689    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5de8bebf-cfad-49da-9a98-08aedde4442c-home" pod "5de8bebf-cfad-49da-9a98-08aedde4442c" (UID: "5de8bebf-cfad-49da-9a98-08aedde4442c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:23:11.496007 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:11.495981    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5de8bebf-cfad-49da-9a98-08aedde4442c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5de8bebf-cfad-49da-9a98-08aedde4442c-tmp-dir\") pod \"5de8bebf-cfad-49da-9a98-08aedde4442c\" (UID: \"5de8bebf-cfad-49da-9a98-08aedde4442c\") "
Oct 30 01:23:11.496152 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:11.496103    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5de8bebf-cfad-49da-9a98-08aedde4442c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5de8bebf-cfad-49da-9a98-08aedde4442c-dshm\") pod \"5de8bebf-cfad-49da-9a98-08aedde4442c\" (UID: \"5de8bebf-cfad-49da-9a98-08aedde4442c\") "
Oct 30 01:23:11.496152 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:11.496141    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5de8bebf-cfad-49da-9a98-08aedde4442c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5de8bebf-cfad-49da-9a98-08aedde4442c-model-cache\") pod \"5de8bebf-cfad-49da-9a98-08aedde4442c\" (UID: \"5de8bebf-cfad-49da-9a98-08aedde4442c\") "
Oct 30 01:23:11.497013 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:11.496574    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5de8bebf-cfad-49da-9a98-08aedde4442c-tmp-dir" pod "5de8bebf-cfad-49da-9a98-08aedde4442c" (UID: "5de8bebf-cfad-49da-9a98-08aedde4442c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:23:11.497013 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:11.496700    5002 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5de8bebf-cfad-49da-9a98-08aedde4442c-home\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:23:11.497013 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:11.496718    5002 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5de8bebf-cfad-49da-9a98-08aedde4442c-tmp-dir\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:23:11.497013 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:11.496791    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5de8bebf-cfad-49da-9a98-08aedde4442c-model-cache" pod "5de8bebf-cfad-49da-9a98-08aedde4442c" (UID: "5de8bebf-cfad-49da-9a98-08aedde4442c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:23:11.498308 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:11.498282    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5de8bebf-cfad-49da-9a98-08aedde4442c-tls-certs" pod "5de8bebf-cfad-49da-9a98-08aedde4442c" (UID: "5de8bebf-cfad-49da-9a98-08aedde4442c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:23:11.498513 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:11.498492    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5de8bebf-cfad-49da-9a98-08aedde4442c-kube-api-access-5v78h" pod "5de8bebf-cfad-49da-9a98-08aedde4442c" (UID: "5de8bebf-cfad-49da-9a98-08aedde4442c"). InnerVolumeSpecName "kube-api-access-5v78h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:23:11.499153 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:11.499126    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5de8bebf-cfad-49da-9a98-08aedde4442c-dshm" pod "5de8bebf-cfad-49da-9a98-08aedde4442c" (UID: "5de8bebf-cfad-49da-9a98-08aedde4442c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:23:11.597935 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:11.597896    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5v78h\" (UniqueName: \"kubernetes.io/projected/5de8bebf-cfad-49da-9a98-08aedde4442c-kube-api-access-5v78h\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:23:11.597935 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:11.597936    5002 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5de8bebf-cfad-49da-9a98-08aedde4442c-tls-certs\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:23:11.597935 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:11.597951    5002 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5de8bebf-cfad-49da-9a98-08aedde4442c-dshm\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:23:11.598662 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:11.597964    5002 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5de8bebf-cfad-49da-9a98-08aedde4442c-model-cache\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:23:12.125577 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:12.125541    5002 generic.go:356] "Generic (PLEG): container finished" podID="5de8bebf-cfad-49da-9a98-08aedde4442c" containerID="799190f41cf9cd2205e5603084cd81d6af97fb7562ff18ef90213ac781319742" exitCode=0
Oct 30 01:23:12.125770 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:12.125603    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ca5363/distinct-2-w4-ca5363-kserve-576576f9b5-xkkqc"
Oct 30 01:23:12.125770 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:12.125622    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ca5363/distinct-2-w4-ca5363-kserve-576576f9b5-xkkqc" event={"ID":"5de8bebf-cfad-49da-9a98-08aedde4442c","Type":"ContainerDied","Data":"799190f41cf9cd2205e5603084cd81d6af97fb7562ff18ef90213ac781319742"}
Oct 30 01:23:12.125770 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:12.125666    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ca5363/distinct-2-w4-ca5363-kserve-576576f9b5-xkkqc" event={"ID":"5de8bebf-cfad-49da-9a98-08aedde4442c","Type":"ContainerDied","Data":"ad84d9fc58ea0b3962e42b82bd94ed709c56817c604f1f4396b5321d537f0431"}
Oct 30 01:23:12.125770 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:12.125685    5002 scope.go:122] "RemoveContainer" containerID="799190f41cf9cd2205e5603084cd81d6af97fb7562ff18ef90213ac781319742"
Oct 30 01:23:12.137733 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:12.137711    5002 scope.go:122] "RemoveContainer" containerID="799190f41cf9cd2205e5603084cd81d6af97fb7562ff18ef90213ac781319742"
Oct 30 01:23:12.138047 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:23:12.138017    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"799190f41cf9cd2205e5603084cd81d6af97fb7562ff18ef90213ac781319742\": container with ID starting with 799190f41cf9cd2205e5603084cd81d6af97fb7562ff18ef90213ac781319742 not found: ID does not exist" containerID="799190f41cf9cd2205e5603084cd81d6af97fb7562ff18ef90213ac781319742"
Oct 30 01:23:12.138122 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:12.138060    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"799190f41cf9cd2205e5603084cd81d6af97fb7562ff18ef90213ac781319742"} err="failed to get container status \"799190f41cf9cd2205e5603084cd81d6af97fb7562ff18ef90213ac781319742\": rpc error: code = NotFound desc = could not find container \"799190f41cf9cd2205e5603084cd81d6af97fb7562ff18ef90213ac781319742\": container with ID starting with 799190f41cf9cd2205e5603084cd81d6af97fb7562ff18ef90213ac781319742 not found: ID does not exist"
Oct 30 01:23:12.183057 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:12.183022    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-ca5363/distinct-2-w4-ca5363-kserve-576576f9b5-xkkqc"]
Oct 30 01:23:12.186425 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:12.186397    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-ca5363/distinct-2-w4-ca5363-kserve-576576f9b5-xkkqc"]
Oct 30 01:23:12.224935 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:12.224890    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f64dc0-b4f9c4899-cnjhz" podUID="b02a73f0-b8fd-4b73-8437-c940fc676a67" containerName="payload-pre-processing" containerID="cri-o://d08a9e109ffb7402396024741367b71a6d600823cc44ad3bc0143da58e28d5c2" gracePeriod=25
Oct 30 01:23:12.494295 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:12.494259    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5de8bebf-cfad-49da-9a98-08aedde4442c" path="/var/lib/kubelet/pods/5de8bebf-cfad-49da-9a98-08aedde4442c/volumes"
Oct 30 01:23:12.824180 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:12.824093    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f64dc0-b4f9c4899-cnjhz" podUID="b02a73f0-b8fd-4b73-8437-c940fc676a67" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:23:12.860392 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:23:12.860342    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbjdq" containerName="extractor"
Oct 30 01:23:18.154196 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:18.154161    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-658c99-openshift-default-54d78694df-cfhwj"]
Oct 30 01:23:18.154881 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:18.154462    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-658c99-openshift-default-54d78694df-cfhwj" podUID="7077b8b8-1f35-49c4-9e58-762297c046cf" containerName="istio-proxy" containerID="cri-o://230d62681ee1a91e15bbf16f693d6386e160dc893faefaeadff7312dc9807422" gracePeriod=30
Oct 30 01:23:22.717635 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:22.717594    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-6m5cj"]
Oct 30 01:23:22.718442 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:22.718406    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-6m5cj" podUID="0ff20380-75e5-4089-9f54-50d1917eff55" containerName="istio-proxy" containerID="cri-o://2298517812d470df4b182cf6e39d29cb2d02de7612a0c0ab201dda442fa3b9cb" gracePeriod=30
Oct 30 01:23:22.825175 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:22.825108    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f64dc0-b4f9c4899-cnjhz" podUID="b02a73f0-b8fd-4b73-8437-c940fc676a67" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:23:23.688448 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:23.688418    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-658c99-openshift-default-54d78694df-cfhwj"
Oct 30 01:23:23.786250 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:23.785156    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7077b8b8-1f35-49c4-9e58-762297c046cf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7077b8b8-1f35-49c4-9e58-762297c046cf-istio-data\") pod \"7077b8b8-1f35-49c4-9e58-762297c046cf\" (UID: \"7077b8b8-1f35-49c4-9e58-762297c046cf\") "
Oct 30 01:23:23.786250 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:23.785226    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7077b8b8-1f35-49c4-9e58-762297c046cf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7077b8b8-1f35-49c4-9e58-762297c046cf-credential-socket\") pod \"7077b8b8-1f35-49c4-9e58-762297c046cf\" (UID: \"7077b8b8-1f35-49c4-9e58-762297c046cf\") "
Oct 30 01:23:23.786250 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:23.785267    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7077b8b8-1f35-49c4-9e58-762297c046cf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7077b8b8-1f35-49c4-9e58-762297c046cf-istiod-ca-cert\") pod \"7077b8b8-1f35-49c4-9e58-762297c046cf\" (UID: \"7077b8b8-1f35-49c4-9e58-762297c046cf\") "
Oct 30 01:23:23.786250 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:23.785311    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7077b8b8-1f35-49c4-9e58-762297c046cf-istio-token\" (UniqueName: \"kubernetes.io/projected/7077b8b8-1f35-49c4-9e58-762297c046cf-istio-token\") pod \"7077b8b8-1f35-49c4-9e58-762297c046cf\" (UID: \"7077b8b8-1f35-49c4-9e58-762297c046cf\") "
Oct 30 01:23:23.786250 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:23.785338    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7077b8b8-1f35-49c4-9e58-762297c046cf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7077b8b8-1f35-49c4-9e58-762297c046cf-istio-podinfo\") pod \"7077b8b8-1f35-49c4-9e58-762297c046cf\" (UID: \"7077b8b8-1f35-49c4-9e58-762297c046cf\") "
Oct 30 01:23:23.786250 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:23.785465    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7077b8b8-1f35-49c4-9e58-762297c046cf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7077b8b8-1f35-49c4-9e58-762297c046cf-istio-envoy\") pod \"7077b8b8-1f35-49c4-9e58-762297c046cf\" (UID: \"7077b8b8-1f35-49c4-9e58-762297c046cf\") "
Oct 30 01:23:23.786250 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:23.785518    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7077b8b8-1f35-49c4-9e58-762297c046cf-kube-api-access-8w4t4\" (UniqueName: \"kubernetes.io/projected/7077b8b8-1f35-49c4-9e58-762297c046cf-kube-api-access-8w4t4\") pod \"7077b8b8-1f35-49c4-9e58-762297c046cf\" (UID: \"7077b8b8-1f35-49c4-9e58-762297c046cf\") "
Oct 30 01:23:23.786250 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:23.785558    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7077b8b8-1f35-49c4-9e58-762297c046cf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7077b8b8-1f35-49c4-9e58-762297c046cf-workload-socket\") pod \"7077b8b8-1f35-49c4-9e58-762297c046cf\" (UID: \"7077b8b8-1f35-49c4-9e58-762297c046cf\") "
Oct 30 01:23:23.786250 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:23.785615    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7077b8b8-1f35-49c4-9e58-762297c046cf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7077b8b8-1f35-49c4-9e58-762297c046cf-workload-certs\") pod \"7077b8b8-1f35-49c4-9e58-762297c046cf\" (UID: \"7077b8b8-1f35-49c4-9e58-762297c046cf\") "
Oct 30 01:23:23.787478 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:23.787257    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7077b8b8-1f35-49c4-9e58-762297c046cf-workload-certs" pod "7077b8b8-1f35-49c4-9e58-762297c046cf" (UID: "7077b8b8-1f35-49c4-9e58-762297c046cf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:23:23.787758 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:23.787612    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7077b8b8-1f35-49c4-9e58-762297c046cf-istio-data" pod "7077b8b8-1f35-49c4-9e58-762297c046cf" (UID: "7077b8b8-1f35-49c4-9e58-762297c046cf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:23:23.787863 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:23.787763    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7077b8b8-1f35-49c4-9e58-762297c046cf-credential-socket" pod "7077b8b8-1f35-49c4-9e58-762297c046cf" (UID: "7077b8b8-1f35-49c4-9e58-762297c046cf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:23:23.788568 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:23.788229    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7077b8b8-1f35-49c4-9e58-762297c046cf-istiod-ca-cert" pod "7077b8b8-1f35-49c4-9e58-762297c046cf" (UID: "7077b8b8-1f35-49c4-9e58-762297c046cf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:23:23.788568 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:23.788520    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7077b8b8-1f35-49c4-9e58-762297c046cf-workload-socket" pod "7077b8b8-1f35-49c4-9e58-762297c046cf" (UID: "7077b8b8-1f35-49c4-9e58-762297c046cf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:23:23.791337 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:23.791315    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7077b8b8-1f35-49c4-9e58-762297c046cf-kube-api-access-8w4t4" pod "7077b8b8-1f35-49c4-9e58-762297c046cf" (UID: "7077b8b8-1f35-49c4-9e58-762297c046cf"). InnerVolumeSpecName "kube-api-access-8w4t4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:23:23.791642 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:23.791614    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7077b8b8-1f35-49c4-9e58-762297c046cf-istio-podinfo" pod "7077b8b8-1f35-49c4-9e58-762297c046cf" (UID: "7077b8b8-1f35-49c4-9e58-762297c046cf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:23:23.793305 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:23.793271    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7077b8b8-1f35-49c4-9e58-762297c046cf-istio-envoy" pod "7077b8b8-1f35-49c4-9e58-762297c046cf" (UID: "7077b8b8-1f35-49c4-9e58-762297c046cf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:23:23.793636 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:23.793599    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7077b8b8-1f35-49c4-9e58-762297c046cf-istio-token" pod "7077b8b8-1f35-49c4-9e58-762297c046cf" (UID: "7077b8b8-1f35-49c4-9e58-762297c046cf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:23:23.887327 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:23.887231    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8w4t4\" (UniqueName: \"kubernetes.io/projected/7077b8b8-1f35-49c4-9e58-762297c046cf-kube-api-access-8w4t4\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:23:23.887327 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:23.887274    5002 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7077b8b8-1f35-49c4-9e58-762297c046cf-workload-socket\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:23:23.887327 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:23.887290    5002 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7077b8b8-1f35-49c4-9e58-762297c046cf-workload-certs\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:23:23.887327 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:23.887303    5002 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7077b8b8-1f35-49c4-9e58-762297c046cf-istio-data\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:23:23.887327 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:23.887316    5002 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7077b8b8-1f35-49c4-9e58-762297c046cf-credential-socket\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:23:23.887327 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:23.887329    5002 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7077b8b8-1f35-49c4-9e58-762297c046cf-istiod-ca-cert\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:23:23.887729 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:23.887342    5002 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7077b8b8-1f35-49c4-9e58-762297c046cf-istio-token\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:23:23.887729 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:23.887355    5002 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7077b8b8-1f35-49c4-9e58-762297c046cf-istio-podinfo\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:23:23.887729 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:23.887371    5002 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7077b8b8-1f35-49c4-9e58-762297c046cf-istio-envoy\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:23:24.236049 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:24.236003    5002 generic.go:356] "Generic (PLEG): container finished" podID="7077b8b8-1f35-49c4-9e58-762297c046cf" containerID="230d62681ee1a91e15bbf16f693d6386e160dc893faefaeadff7312dc9807422" exitCode=0
Oct 30 01:23:24.236259 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:24.236070    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-658c99-openshift-default-54d78694df-cfhwj"
Oct 30 01:23:24.236259 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:24.236089    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-658c99-openshift-default-54d78694df-cfhwj" event={"ID":"7077b8b8-1f35-49c4-9e58-762297c046cf","Type":"ContainerDied","Data":"230d62681ee1a91e15bbf16f693d6386e160dc893faefaeadff7312dc9807422"}
Oct 30 01:23:24.236259 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:24.236189    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-658c99-openshift-default-54d78694df-cfhwj" event={"ID":"7077b8b8-1f35-49c4-9e58-762297c046cf","Type":"ContainerDied","Data":"ee0e51aaa09d576ce44d698d46b37c6d636b14d6b7ca14d940fcb43cc53f4306"}
Oct 30 01:23:24.236259 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:24.236221    5002 scope.go:122] "RemoveContainer" containerID="230d62681ee1a91e15bbf16f693d6386e160dc893faefaeadff7312dc9807422"
Oct 30 01:23:24.249909 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:24.249890    5002 scope.go:122] "RemoveContainer" containerID="3f0d5174659d934cb002b76b41d644fd6133deca3d9b55042a24a21b138cf830"
Oct 30 01:23:24.260401 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:24.260363    5002 scope.go:122] "RemoveContainer" containerID="230d62681ee1a91e15bbf16f693d6386e160dc893faefaeadff7312dc9807422"
Oct 30 01:23:24.260733 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:23:24.260709    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"230d62681ee1a91e15bbf16f693d6386e160dc893faefaeadff7312dc9807422\": container with ID starting with 230d62681ee1a91e15bbf16f693d6386e160dc893faefaeadff7312dc9807422 not found: ID does not exist" containerID="230d62681ee1a91e15bbf16f693d6386e160dc893faefaeadff7312dc9807422"
Oct 30 01:23:24.260802 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:24.260742    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"230d62681ee1a91e15bbf16f693d6386e160dc893faefaeadff7312dc9807422"} err="failed to get container status \"230d62681ee1a91e15bbf16f693d6386e160dc893faefaeadff7312dc9807422\": rpc error: code = NotFound desc = could not find container \"230d62681ee1a91e15bbf16f693d6386e160dc893faefaeadff7312dc9807422\": container with ID starting with 230d62681ee1a91e15bbf16f693d6386e160dc893faefaeadff7312dc9807422 not found: ID does not exist"
Oct 30 01:23:24.260802 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:24.260760    5002 scope.go:122] "RemoveContainer" containerID="3f0d5174659d934cb002b76b41d644fd6133deca3d9b55042a24a21b138cf830"
Oct 30 01:23:24.261010 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:23:24.260990    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f0d5174659d934cb002b76b41d644fd6133deca3d9b55042a24a21b138cf830\": container with ID starting with 3f0d5174659d934cb002b76b41d644fd6133deca3d9b55042a24a21b138cf830 not found: ID does not exist" containerID="3f0d5174659d934cb002b76b41d644fd6133deca3d9b55042a24a21b138cf830"
Oct 30 01:23:24.261065 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:24.261015    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f0d5174659d934cb002b76b41d644fd6133deca3d9b55042a24a21b138cf830"} err="failed to get container status \"3f0d5174659d934cb002b76b41d644fd6133deca3d9b55042a24a21b138cf830\": rpc error: code = NotFound desc = could not find container \"3f0d5174659d934cb002b76b41d644fd6133deca3d9b55042a24a21b138cf830\": container with ID starting with 3f0d5174659d934cb002b76b41d644fd6133deca3d9b55042a24a21b138cf830 not found: ID does not exist"
Oct 30 01:23:24.293456 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:24.293415    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-658c99-openshift-default-54d78694df-cfhwj"]
Oct 30 01:23:24.307298 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:24.307267    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-658c99-openshift-default-54d78694df-cfhwj"]
Oct 30 01:23:24.495317 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:24.495233    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7077b8b8-1f35-49c4-9e58-762297c046cf" path="/var/lib/kubelet/pods/7077b8b8-1f35-49c4-9e58-762297c046cf/volumes"
Oct 30 01:23:27.630834 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:27.630748    5002 patch_prober.go:29] interesting pod/e2e-worker-w3-cefb27-openshift-default-577b649d97-6m5cj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 01:23:27.630834 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:27.630806    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-6m5cj" podUID="0ff20380-75e5-4089-9f54-50d1917eff55" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 01:23:29.275994 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:29.275955    5002 generic.go:356] "Generic (PLEG): container finished" podID="0ff20380-75e5-4089-9f54-50d1917eff55" containerID="2298517812d470df4b182cf6e39d29cb2d02de7612a0c0ab201dda442fa3b9cb" exitCode=0
Oct 30 01:23:29.276495 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:29.276037    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-6m5cj" event={"ID":"0ff20380-75e5-4089-9f54-50d1917eff55","Type":"ContainerDied","Data":"2298517812d470df4b182cf6e39d29cb2d02de7612a0c0ab201dda442fa3b9cb"}
Oct 30 01:23:29.496191 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:29.496156    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-009e78-847cbfbf7d-vgf6m"]
Oct 30 01:23:29.531620 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:29.531531    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-009e78-54d78f6cf9-sjgcl"]
Oct 30 01:23:29.589889 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:29.589651    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-6m5cj"
Oct 30 01:23:29.650915 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:29.650875    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ff20380-75e5-4089-9f54-50d1917eff55-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ff20380-75e5-4089-9f54-50d1917eff55-istio-data\") pod \"0ff20380-75e5-4089-9f54-50d1917eff55\" (UID: \"0ff20380-75e5-4089-9f54-50d1917eff55\") "
Oct 30 01:23:29.651107 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:29.650932    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0ff20380-75e5-4089-9f54-50d1917eff55-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ff20380-75e5-4089-9f54-50d1917eff55-istio-podinfo\") pod \"0ff20380-75e5-4089-9f54-50d1917eff55\" (UID: \"0ff20380-75e5-4089-9f54-50d1917eff55\") "
Oct 30 01:23:29.651107 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:29.650982    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ff20380-75e5-4089-9f54-50d1917eff55-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ff20380-75e5-4089-9f54-50d1917eff55-credential-socket\") pod \"0ff20380-75e5-4089-9f54-50d1917eff55\" (UID: \"0ff20380-75e5-4089-9f54-50d1917eff55\") "
Oct 30 01:23:29.651107 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:29.651033    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ff20380-75e5-4089-9f54-50d1917eff55-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ff20380-75e5-4089-9f54-50d1917eff55-istiod-ca-cert\") pod \"0ff20380-75e5-4089-9f54-50d1917eff55\" (UID: \"0ff20380-75e5-4089-9f54-50d1917eff55\") "
Oct 30 01:23:29.651107 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:29.651062    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ff20380-75e5-4089-9f54-50d1917eff55-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ff20380-75e5-4089-9f54-50d1917eff55-workload-certs\") pod \"0ff20380-75e5-4089-9f54-50d1917eff55\" (UID: \"0ff20380-75e5-4089-9f54-50d1917eff55\") "
Oct 30 01:23:29.651349 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:29.651171    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ff20380-75e5-4089-9f54-50d1917eff55-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ff20380-75e5-4089-9f54-50d1917eff55-workload-socket\") pod \"0ff20380-75e5-4089-9f54-50d1917eff55\" (UID: \"0ff20380-75e5-4089-9f54-50d1917eff55\") "
Oct 30 01:23:29.651349 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:29.651198    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ff20380-75e5-4089-9f54-50d1917eff55-istio-token\" (UniqueName: \"kubernetes.io/projected/0ff20380-75e5-4089-9f54-50d1917eff55-istio-token\") pod \"0ff20380-75e5-4089-9f54-50d1917eff55\" (UID: \"0ff20380-75e5-4089-9f54-50d1917eff55\") "
Oct 30 01:23:29.651349 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:29.651227    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ff20380-75e5-4089-9f54-50d1917eff55-kube-api-access-fzg9n\" (UniqueName: \"kubernetes.io/projected/0ff20380-75e5-4089-9f54-50d1917eff55-kube-api-access-fzg9n\") pod \"0ff20380-75e5-4089-9f54-50d1917eff55\" (UID: \"0ff20380-75e5-4089-9f54-50d1917eff55\") "
Oct 30 01:23:29.651349 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:29.651273    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ff20380-75e5-4089-9f54-50d1917eff55-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ff20380-75e5-4089-9f54-50d1917eff55-istio-envoy\") pod \"0ff20380-75e5-4089-9f54-50d1917eff55\" (UID: \"0ff20380-75e5-4089-9f54-50d1917eff55\") "
Oct 30 01:23:29.651626 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:29.651363    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ff20380-75e5-4089-9f54-50d1917eff55-credential-socket" pod "0ff20380-75e5-4089-9f54-50d1917eff55" (UID: "0ff20380-75e5-4089-9f54-50d1917eff55"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:23:29.651735 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:29.651710    5002 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ff20380-75e5-4089-9f54-50d1917eff55-credential-socket\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:23:29.652249 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:29.652112    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ff20380-75e5-4089-9f54-50d1917eff55-istiod-ca-cert" pod "0ff20380-75e5-4089-9f54-50d1917eff55" (UID: "0ff20380-75e5-4089-9f54-50d1917eff55"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:23:29.652249 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:29.652215    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ff20380-75e5-4089-9f54-50d1917eff55-workload-certs" pod "0ff20380-75e5-4089-9f54-50d1917eff55" (UID: "0ff20380-75e5-4089-9f54-50d1917eff55"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:23:29.652249 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:29.652244    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ff20380-75e5-4089-9f54-50d1917eff55-istio-data" pod "0ff20380-75e5-4089-9f54-50d1917eff55" (UID: "0ff20380-75e5-4089-9f54-50d1917eff55"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:23:29.653662 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:29.653634    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ff20380-75e5-4089-9f54-50d1917eff55-workload-socket" pod "0ff20380-75e5-4089-9f54-50d1917eff55" (UID: "0ff20380-75e5-4089-9f54-50d1917eff55"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:23:29.654854 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:29.654718    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ff20380-75e5-4089-9f54-50d1917eff55-istio-envoy" pod "0ff20380-75e5-4089-9f54-50d1917eff55" (UID: "0ff20380-75e5-4089-9f54-50d1917eff55"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:23:29.654854 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:29.654824    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0ff20380-75e5-4089-9f54-50d1917eff55-istio-podinfo" pod "0ff20380-75e5-4089-9f54-50d1917eff55" (UID: "0ff20380-75e5-4089-9f54-50d1917eff55"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:23:29.655786 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:29.655758    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ff20380-75e5-4089-9f54-50d1917eff55-istio-token" pod "0ff20380-75e5-4089-9f54-50d1917eff55" (UID: "0ff20380-75e5-4089-9f54-50d1917eff55"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:23:29.656011 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:29.655993    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ff20380-75e5-4089-9f54-50d1917eff55-kube-api-access-fzg9n" pod "0ff20380-75e5-4089-9f54-50d1917eff55" (UID: "0ff20380-75e5-4089-9f54-50d1917eff55"). InnerVolumeSpecName "kube-api-access-fzg9n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:23:29.755257 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:29.755080    5002 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ff20380-75e5-4089-9f54-50d1917eff55-workload-socket\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:23:29.755257 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:29.755126    5002 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ff20380-75e5-4089-9f54-50d1917eff55-istio-token\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:23:29.755257 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:29.755141    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fzg9n\" (UniqueName: \"kubernetes.io/projected/0ff20380-75e5-4089-9f54-50d1917eff55-kube-api-access-fzg9n\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:23:29.755257 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:29.755156    5002 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ff20380-75e5-4089-9f54-50d1917eff55-istio-envoy\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:23:29.755257 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:29.755169    5002 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ff20380-75e5-4089-9f54-50d1917eff55-istio-data\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:23:29.755257 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:29.755182    5002 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ff20380-75e5-4089-9f54-50d1917eff55-istio-podinfo\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:23:29.755257 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:29.755195    5002 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ff20380-75e5-4089-9f54-50d1917eff55-istiod-ca-cert\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:23:29.755257 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:29.755209    5002 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ff20380-75e5-4089-9f54-50d1917eff55-workload-certs\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:23:30.285603 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:30.285560    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-6m5cj" event={"ID":"0ff20380-75e5-4089-9f54-50d1917eff55","Type":"ContainerDied","Data":"0c5f5e8465542b9904c5ba4e50c2d86c908df4d9b764789f8868e04b4ef247b3"}
Oct 30 01:23:30.286113 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:30.285609    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-6m5cj"
Oct 30 01:23:30.286113 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:30.285658    5002 scope.go:122] "RemoveContainer" containerID="2298517812d470df4b182cf6e39d29cb2d02de7612a0c0ab201dda442fa3b9cb"
Oct 30 01:23:30.343894 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:30.343857    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-6m5cj"]
Oct 30 01:23:30.358036 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:30.358001    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-6m5cj"]
Oct 30 01:23:30.493503 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:30.493468    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ff20380-75e5-4089-9f54-50d1917eff55" path="/var/lib/kubelet/pods/0ff20380-75e5-4089-9f54-50d1917eff55/volumes"
Oct 30 01:23:31.451568 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:31.451454    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7d8z4"]
Oct 30 01:23:32.304519 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:32.304363    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-xfr4r"]
Oct 30 01:23:32.304762 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:32.304730    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-xfr4r" podUID="735d76ff-0add-4acc-b5e9-4ebeb27c0e9a" containerName="istio-proxy" containerID="cri-o://6dd8fa824e1c5a0d12aab0bf1b534816167511bc151c9dc23fa04fef6e7e01c7" gracePeriod=30
Oct 30 01:23:32.557921 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:32.557892    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f64dc0-b4f9c4899-cnjhz"
Oct 30 01:23:32.693650 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:32.693616    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b02a73f0-b8fd-4b73-8437-c940fc676a67-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b02a73f0-b8fd-4b73-8437-c940fc676a67-plugins-config-volume\") pod \"b02a73f0-b8fd-4b73-8437-c940fc676a67\" (UID: \"b02a73f0-b8fd-4b73-8437-c940fc676a67\") "
Oct 30 01:23:32.693848 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:32.693798    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b02a73f0-b8fd-4b73-8437-c940fc676a67-kube-api-access-fn82f\" (UniqueName: \"kubernetes.io/projected/b02a73f0-b8fd-4b73-8437-c940fc676a67-kube-api-access-fn82f\") pod \"b02a73f0-b8fd-4b73-8437-c940fc676a67\" (UID: \"b02a73f0-b8fd-4b73-8437-c940fc676a67\") "
Oct 30 01:23:32.694101 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:32.694072    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b02a73f0-b8fd-4b73-8437-c940fc676a67-plugins-config-volume" pod "b02a73f0-b8fd-4b73-8437-c940fc676a67" (UID: "b02a73f0-b8fd-4b73-8437-c940fc676a67"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:23:32.694235 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:32.694219    5002 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b02a73f0-b8fd-4b73-8437-c940fc676a67-plugins-config-volume\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:23:32.696170 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:32.696147    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b02a73f0-b8fd-4b73-8437-c940fc676a67-kube-api-access-fn82f" pod "b02a73f0-b8fd-4b73-8437-c940fc676a67" (UID: "b02a73f0-b8fd-4b73-8437-c940fc676a67"). InnerVolumeSpecName "kube-api-access-fn82f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:23:32.795569 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:32.795495    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fn82f\" (UniqueName: \"kubernetes.io/projected/b02a73f0-b8fd-4b73-8437-c940fc676a67-kube-api-access-fn82f\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:23:33.323079 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:33.323039    5002 generic.go:356] "Generic (PLEG): container finished" podID="b02a73f0-b8fd-4b73-8437-c940fc676a67" containerID="d08a9e109ffb7402396024741367b71a6d600823cc44ad3bc0143da58e28d5c2" exitCode=0
Oct 30 01:23:33.323448 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:33.323116    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f64dc0-b4f9c4899-cnjhz"
Oct 30 01:23:33.323448 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:33.323138    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f64dc0-b4f9c4899-cnjhz" event={"ID":"b02a73f0-b8fd-4b73-8437-c940fc676a67","Type":"ContainerDied","Data":"d08a9e109ffb7402396024741367b71a6d600823cc44ad3bc0143da58e28d5c2"}
Oct 30 01:23:33.323448 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:33.323206    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f64dc0-b4f9c4899-cnjhz" event={"ID":"b02a73f0-b8fd-4b73-8437-c940fc676a67","Type":"ContainerDied","Data":"d5efea925739cd5072eb8cbabaf92be67996001da603f6adf23eeedec353d75a"}
Oct 30 01:23:33.323448 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:33.323256    5002 scope.go:122] "RemoveContainer" containerID="d08a9e109ffb7402396024741367b71a6d600823cc44ad3bc0143da58e28d5c2"
Oct 30 01:23:33.345644 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:33.338976    5002 scope.go:122] "RemoveContainer" containerID="d08a9e109ffb7402396024741367b71a6d600823cc44ad3bc0143da58e28d5c2"
Oct 30 01:23:33.345644 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:23:33.344309    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d08a9e109ffb7402396024741367b71a6d600823cc44ad3bc0143da58e28d5c2\": container with ID starting with d08a9e109ffb7402396024741367b71a6d600823cc44ad3bc0143da58e28d5c2 not found: ID does not exist" containerID="d08a9e109ffb7402396024741367b71a6d600823cc44ad3bc0143da58e28d5c2"
Oct 30 01:23:33.345644 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:33.344349    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d08a9e109ffb7402396024741367b71a6d600823cc44ad3bc0143da58e28d5c2"} err="failed to get container status \"d08a9e109ffb7402396024741367b71a6d600823cc44ad3bc0143da58e28d5c2\": rpc error: code = NotFound desc = could not find container \"d08a9e109ffb7402396024741367b71a6d600823cc44ad3bc0143da58e28d5c2\": container with ID starting with d08a9e109ffb7402396024741367b71a6d600823cc44ad3bc0143da58e28d5c2 not found: ID does not exist"
Oct 30 01:23:33.385567 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:33.385536    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f64dc0-b4f9c4899-cnjhz"]
Oct 30 01:23:33.391654 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:33.391620    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f64dc0-b4f9c4899-cnjhz"]
Oct 30 01:23:33.627586 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:33.627497    5002 patch_prober.go:29] interesting pod/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-xfr4r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 01:23:33.627586 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:33.627573    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-xfr4r" podUID="735d76ff-0add-4acc-b5e9-4ebeb27c0e9a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 01:23:34.495733 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:34.495693    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b02a73f0-b8fd-4b73-8437-c940fc676a67" path="/var/lib/kubelet/pods/b02a73f0-b8fd-4b73-8437-c940fc676a67/volumes"
Oct 30 01:23:34.497541 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:34.497500    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-009e78-847cbfbf7d-vgf6m" podUID="5a0523a2-66bc-4ec0-a5bf-ac5cbafcdbfd" containerName="payload-processing" containerID="cri-o://de20524b1394285bbc1d024f323d1fee8647693e0395af6a079b4befd317d87a" gracePeriod=25
Oct 30 01:23:34.531934 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:34.531871    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-009e78-54d78f6cf9-sjgcl" podUID="a0a93655-85f4-4cf7-9e80-2ea8ce25bcb5" containerName="payload-pre-processing" containerID="cri-o://70e6127b29dc6420849daa9cd497a262f43584dacb52352a76584bcfbd32c3aa" gracePeriod=25
Oct 30 01:23:37.575601 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:37.575527    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-xfr4r"
Oct 30 01:23:37.656425 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:37.655662    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/735d76ff-0add-4acc-b5e9-4ebeb27c0e9a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/735d76ff-0add-4acc-b5e9-4ebeb27c0e9a-istio-podinfo\") pod \"735d76ff-0add-4acc-b5e9-4ebeb27c0e9a\" (UID: \"735d76ff-0add-4acc-b5e9-4ebeb27c0e9a\") "
Oct 30 01:23:37.656425 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:37.655721    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/735d76ff-0add-4acc-b5e9-4ebeb27c0e9a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/735d76ff-0add-4acc-b5e9-4ebeb27c0e9a-workload-certs\") pod \"735d76ff-0add-4acc-b5e9-4ebeb27c0e9a\" (UID: \"735d76ff-0add-4acc-b5e9-4ebeb27c0e9a\") "
Oct 30 01:23:37.656425 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:37.655773    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/735d76ff-0add-4acc-b5e9-4ebeb27c0e9a-kube-api-access-jzp59\" (UniqueName: \"kubernetes.io/projected/735d76ff-0add-4acc-b5e9-4ebeb27c0e9a-kube-api-access-jzp59\") pod \"735d76ff-0add-4acc-b5e9-4ebeb27c0e9a\" (UID: \"735d76ff-0add-4acc-b5e9-4ebeb27c0e9a\") "
Oct 30 01:23:37.656425 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:37.655847    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/735d76ff-0add-4acc-b5e9-4ebeb27c0e9a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/735d76ff-0add-4acc-b5e9-4ebeb27c0e9a-istiod-ca-cert\") pod \"735d76ff-0add-4acc-b5e9-4ebeb27c0e9a\" (UID: \"735d76ff-0add-4acc-b5e9-4ebeb27c0e9a\") "
Oct 30 01:23:37.656425 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:37.655901    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/735d76ff-0add-4acc-b5e9-4ebeb27c0e9a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/735d76ff-0add-4acc-b5e9-4ebeb27c0e9a-credential-socket\") pod \"735d76ff-0add-4acc-b5e9-4ebeb27c0e9a\" (UID: \"735d76ff-0add-4acc-b5e9-4ebeb27c0e9a\") "
Oct 30 01:23:37.656425 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:37.655934    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/735d76ff-0add-4acc-b5e9-4ebeb27c0e9a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/735d76ff-0add-4acc-b5e9-4ebeb27c0e9a-istio-data\") pod \"735d76ff-0add-4acc-b5e9-4ebeb27c0e9a\" (UID: \"735d76ff-0add-4acc-b5e9-4ebeb27c0e9a\") "
Oct 30 01:23:37.656425 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:37.656042    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/735d76ff-0add-4acc-b5e9-4ebeb27c0e9a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/735d76ff-0add-4acc-b5e9-4ebeb27c0e9a-workload-socket\") pod \"735d76ff-0add-4acc-b5e9-4ebeb27c0e9a\" (UID: \"735d76ff-0add-4acc-b5e9-4ebeb27c0e9a\") "
Oct 30 01:23:37.656425 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:37.656068    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/735d76ff-0add-4acc-b5e9-4ebeb27c0e9a-istio-token\" (UniqueName: \"kubernetes.io/projected/735d76ff-0add-4acc-b5e9-4ebeb27c0e9a-istio-token\") pod \"735d76ff-0add-4acc-b5e9-4ebeb27c0e9a\" (UID: \"735d76ff-0add-4acc-b5e9-4ebeb27c0e9a\") "
Oct 30 01:23:37.656425 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:37.656113    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/735d76ff-0add-4acc-b5e9-4ebeb27c0e9a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/735d76ff-0add-4acc-b5e9-4ebeb27c0e9a-istio-envoy\") pod \"735d76ff-0add-4acc-b5e9-4ebeb27c0e9a\" (UID: \"735d76ff-0add-4acc-b5e9-4ebeb27c0e9a\") "
Oct 30 01:23:37.673635 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:37.672144    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/735d76ff-0add-4acc-b5e9-4ebeb27c0e9a-istio-data" pod "735d76ff-0add-4acc-b5e9-4ebeb27c0e9a" (UID: "735d76ff-0add-4acc-b5e9-4ebeb27c0e9a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:23:37.673635 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:37.672329    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/735d76ff-0add-4acc-b5e9-4ebeb27c0e9a-credential-socket" pod "735d76ff-0add-4acc-b5e9-4ebeb27c0e9a" (UID: "735d76ff-0add-4acc-b5e9-4ebeb27c0e9a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:23:37.673635 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:37.672504    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/735d76ff-0add-4acc-b5e9-4ebeb27c0e9a-workload-socket" pod "735d76ff-0add-4acc-b5e9-4ebeb27c0e9a" (UID: "735d76ff-0add-4acc-b5e9-4ebeb27c0e9a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:23:37.673635 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:37.672925    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/735d76ff-0add-4acc-b5e9-4ebeb27c0e9a-istiod-ca-cert" pod "735d76ff-0add-4acc-b5e9-4ebeb27c0e9a" (UID: "735d76ff-0add-4acc-b5e9-4ebeb27c0e9a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:23:37.673850 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:37.673759    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/735d76ff-0add-4acc-b5e9-4ebeb27c0e9a-workload-certs" pod "735d76ff-0add-4acc-b5e9-4ebeb27c0e9a" (UID: "735d76ff-0add-4acc-b5e9-4ebeb27c0e9a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:23:37.675322 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:37.675294    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/735d76ff-0add-4acc-b5e9-4ebeb27c0e9a-istio-envoy" pod "735d76ff-0add-4acc-b5e9-4ebeb27c0e9a" (UID: "735d76ff-0add-4acc-b5e9-4ebeb27c0e9a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:23:37.676803 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:37.676761    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/735d76ff-0add-4acc-b5e9-4ebeb27c0e9a-kube-api-access-jzp59" pod "735d76ff-0add-4acc-b5e9-4ebeb27c0e9a" (UID: "735d76ff-0add-4acc-b5e9-4ebeb27c0e9a"). InnerVolumeSpecName "kube-api-access-jzp59". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:23:37.677203 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:37.677165    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/735d76ff-0add-4acc-b5e9-4ebeb27c0e9a-istio-token" pod "735d76ff-0add-4acc-b5e9-4ebeb27c0e9a" (UID: "735d76ff-0add-4acc-b5e9-4ebeb27c0e9a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:23:37.677203 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:37.677178    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/735d76ff-0add-4acc-b5e9-4ebeb27c0e9a-istio-podinfo" pod "735d76ff-0add-4acc-b5e9-4ebeb27c0e9a" (UID: "735d76ff-0add-4acc-b5e9-4ebeb27c0e9a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:23:37.758605 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:37.758564    5002 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/735d76ff-0add-4acc-b5e9-4ebeb27c0e9a-workload-socket\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:23:37.758605 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:37.758599    5002 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/735d76ff-0add-4acc-b5e9-4ebeb27c0e9a-istio-token\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:23:37.758605 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:37.758607    5002 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/735d76ff-0add-4acc-b5e9-4ebeb27c0e9a-istio-envoy\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:23:37.758605 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:37.758615    5002 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/735d76ff-0add-4acc-b5e9-4ebeb27c0e9a-istio-podinfo\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:23:37.758893 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:37.758624    5002 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/735d76ff-0add-4acc-b5e9-4ebeb27c0e9a-workload-certs\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:23:37.758893 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:37.758632    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jzp59\" (UniqueName: \"kubernetes.io/projected/735d76ff-0add-4acc-b5e9-4ebeb27c0e9a-kube-api-access-jzp59\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:23:37.758893 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:37.758643    5002 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/735d76ff-0add-4acc-b5e9-4ebeb27c0e9a-istiod-ca-cert\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:23:37.758893 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:37.758651    5002 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/735d76ff-0add-4acc-b5e9-4ebeb27c0e9a-credential-socket\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:23:37.758893 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:37.758664    5002 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/735d76ff-0add-4acc-b5e9-4ebeb27c0e9a-istio-data\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:23:38.357619 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:38.357575    5002 generic.go:356] "Generic (PLEG): container finished" podID="735d76ff-0add-4acc-b5e9-4ebeb27c0e9a" containerID="6dd8fa824e1c5a0d12aab0bf1b534816167511bc151c9dc23fa04fef6e7e01c7" exitCode=0
Oct 30 01:23:38.357779 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:38.357626    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-xfr4r"
Oct 30 01:23:38.357779 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:38.357660    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-xfr4r" event={"ID":"735d76ff-0add-4acc-b5e9-4ebeb27c0e9a","Type":"ContainerDied","Data":"6dd8fa824e1c5a0d12aab0bf1b534816167511bc151c9dc23fa04fef6e7e01c7"}
Oct 30 01:23:38.357779 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:38.357716    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-xfr4r" event={"ID":"735d76ff-0add-4acc-b5e9-4ebeb27c0e9a","Type":"ContainerDied","Data":"4df93890b733b3e6591f722aa6814b0d7eeff070f61a686c57331afb1bbe726f"}
Oct 30 01:23:38.357779 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:38.357748    5002 scope.go:122] "RemoveContainer" containerID="6dd8fa824e1c5a0d12aab0bf1b534816167511bc151c9dc23fa04fef6e7e01c7"
Oct 30 01:23:38.369859 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:38.369831    5002 scope.go:122] "RemoveContainer" containerID="6dd8fa824e1c5a0d12aab0bf1b534816167511bc151c9dc23fa04fef6e7e01c7"
Oct 30 01:23:38.370167 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:23:38.370147    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6dd8fa824e1c5a0d12aab0bf1b534816167511bc151c9dc23fa04fef6e7e01c7\": container with ID starting with 6dd8fa824e1c5a0d12aab0bf1b534816167511bc151c9dc23fa04fef6e7e01c7 not found: ID does not exist" containerID="6dd8fa824e1c5a0d12aab0bf1b534816167511bc151c9dc23fa04fef6e7e01c7"
Oct 30 01:23:38.370226 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:38.370180    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6dd8fa824e1c5a0d12aab0bf1b534816167511bc151c9dc23fa04fef6e7e01c7"} err="failed to get container status \"6dd8fa824e1c5a0d12aab0bf1b534816167511bc151c9dc23fa04fef6e7e01c7\": rpc error: code = NotFound desc = could not find container \"6dd8fa824e1c5a0d12aab0bf1b534816167511bc151c9dc23fa04fef6e7e01c7\": container with ID starting with 6dd8fa824e1c5a0d12aab0bf1b534816167511bc151c9dc23fa04fef6e7e01c7 not found: ID does not exist"
Oct 30 01:23:38.427909 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:38.427875    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-xfr4r"]
Oct 30 01:23:38.441765 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:38.441728    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-xfr4r"]
Oct 30 01:23:38.496551 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:38.496515    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="735d76ff-0add-4acc-b5e9-4ebeb27c0e9a" path="/var/lib/kubelet/pods/735d76ff-0add-4acc-b5e9-4ebeb27c0e9a/volumes"
Oct 30 01:23:38.777323 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:38.777277    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-009e78-openshift-default-5674f58c7-9fkcq"]
Oct 30 01:23:38.784021 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:38.783891    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w6-009e78-openshift-default-5674f58c7-9fkcq" podUID="ed247ba4-ebad-4ef0-bffe-5f85a2794a05" containerName="istio-proxy" containerID="cri-o://07890073533e0669fe179142ae54a40b9c0c01eb2025c91f7e78c8098ff6c414" gracePeriod=30
Oct 30 01:23:42.865654 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:23:42.865613    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbjdq" containerName="extractor"
Oct 30 01:23:42.999949 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:42.999908    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-009e78-54d78f6cf9-sjgcl" podUID="a0a93655-85f4-4cf7-9e80-2ea8ce25bcb5" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:23:42.999949 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:42.999940    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-009e78-847cbfbf7d-vgf6m" podUID="5a0523a2-66bc-4ec0-a5bf-ac5cbafcdbfd" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:23:43.410151 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:43.410101    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7d8z4"]
Oct 30 01:23:44.048445 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:44.048418    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-009e78-openshift-default-5674f58c7-9fkcq"
Oct 30 01:23:44.149713 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:44.149607    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed247ba4-ebad-4ef0-bffe-5f85a2794a05-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed247ba4-ebad-4ef0-bffe-5f85a2794a05-istio-envoy\") pod \"ed247ba4-ebad-4ef0-bffe-5f85a2794a05\" (UID: \"ed247ba4-ebad-4ef0-bffe-5f85a2794a05\") "
Oct 30 01:23:44.149713 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:44.149663    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed247ba4-ebad-4ef0-bffe-5f85a2794a05-istio-token\" (UniqueName: \"kubernetes.io/projected/ed247ba4-ebad-4ef0-bffe-5f85a2794a05-istio-token\") pod \"ed247ba4-ebad-4ef0-bffe-5f85a2794a05\" (UID: \"ed247ba4-ebad-4ef0-bffe-5f85a2794a05\") "
Oct 30 01:23:44.149957 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:44.149737    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed247ba4-ebad-4ef0-bffe-5f85a2794a05-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed247ba4-ebad-4ef0-bffe-5f85a2794a05-workload-socket\") pod \"ed247ba4-ebad-4ef0-bffe-5f85a2794a05\" (UID: \"ed247ba4-ebad-4ef0-bffe-5f85a2794a05\") "
Oct 30 01:23:44.149957 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:44.149786    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed247ba4-ebad-4ef0-bffe-5f85a2794a05-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed247ba4-ebad-4ef0-bffe-5f85a2794a05-istiod-ca-cert\") pod \"ed247ba4-ebad-4ef0-bffe-5f85a2794a05\" (UID: \"ed247ba4-ebad-4ef0-bffe-5f85a2794a05\") "
Oct 30 01:23:44.149957 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:44.149821    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed247ba4-ebad-4ef0-bffe-5f85a2794a05-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed247ba4-ebad-4ef0-bffe-5f85a2794a05-istio-data\") pod \"ed247ba4-ebad-4ef0-bffe-5f85a2794a05\" (UID: \"ed247ba4-ebad-4ef0-bffe-5f85a2794a05\") "
Oct 30 01:23:44.149957 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:44.149859    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ed247ba4-ebad-4ef0-bffe-5f85a2794a05-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed247ba4-ebad-4ef0-bffe-5f85a2794a05-istio-podinfo\") pod \"ed247ba4-ebad-4ef0-bffe-5f85a2794a05\" (UID: \"ed247ba4-ebad-4ef0-bffe-5f85a2794a05\") "
Oct 30 01:23:44.149957 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:44.149928    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed247ba4-ebad-4ef0-bffe-5f85a2794a05-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed247ba4-ebad-4ef0-bffe-5f85a2794a05-credential-socket\") pod \"ed247ba4-ebad-4ef0-bffe-5f85a2794a05\" (UID: \"ed247ba4-ebad-4ef0-bffe-5f85a2794a05\") "
Oct 30 01:23:44.150251 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:44.150015    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed247ba4-ebad-4ef0-bffe-5f85a2794a05-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed247ba4-ebad-4ef0-bffe-5f85a2794a05-workload-certs\") pod \"ed247ba4-ebad-4ef0-bffe-5f85a2794a05\" (UID: \"ed247ba4-ebad-4ef0-bffe-5f85a2794a05\") "
Oct 30 01:23:44.150251 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:44.150066    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed247ba4-ebad-4ef0-bffe-5f85a2794a05-kube-api-access-7hrbr\" (UniqueName: \"kubernetes.io/projected/ed247ba4-ebad-4ef0-bffe-5f85a2794a05-kube-api-access-7hrbr\") pod \"ed247ba4-ebad-4ef0-bffe-5f85a2794a05\" (UID: \"ed247ba4-ebad-4ef0-bffe-5f85a2794a05\") "
Oct 30 01:23:44.150362 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:44.150251    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed247ba4-ebad-4ef0-bffe-5f85a2794a05-istio-data" pod "ed247ba4-ebad-4ef0-bffe-5f85a2794a05" (UID: "ed247ba4-ebad-4ef0-bffe-5f85a2794a05"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:23:44.150362 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:44.150335    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed247ba4-ebad-4ef0-bffe-5f85a2794a05-workload-certs" pod "ed247ba4-ebad-4ef0-bffe-5f85a2794a05" (UID: "ed247ba4-ebad-4ef0-bffe-5f85a2794a05"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:23:44.150550 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:44.150529    5002 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed247ba4-ebad-4ef0-bffe-5f85a2794a05-istio-data\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:23:44.150611 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:44.150538    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed247ba4-ebad-4ef0-bffe-5f85a2794a05-istiod-ca-cert" pod "ed247ba4-ebad-4ef0-bffe-5f85a2794a05" (UID: "ed247ba4-ebad-4ef0-bffe-5f85a2794a05"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:23:44.150611 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:44.150556    5002 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed247ba4-ebad-4ef0-bffe-5f85a2794a05-workload-certs\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:23:44.150735 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:44.150624    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed247ba4-ebad-4ef0-bffe-5f85a2794a05-credential-socket" pod "ed247ba4-ebad-4ef0-bffe-5f85a2794a05" (UID: "ed247ba4-ebad-4ef0-bffe-5f85a2794a05"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:23:44.150735 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:44.150179    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed247ba4-ebad-4ef0-bffe-5f85a2794a05-workload-socket" pod "ed247ba4-ebad-4ef0-bffe-5f85a2794a05" (UID: "ed247ba4-ebad-4ef0-bffe-5f85a2794a05"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:23:44.152585 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:44.152551    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed247ba4-ebad-4ef0-bffe-5f85a2794a05-istio-envoy" pod "ed247ba4-ebad-4ef0-bffe-5f85a2794a05" (UID: "ed247ba4-ebad-4ef0-bffe-5f85a2794a05"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:23:44.152742 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:44.152617    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed247ba4-ebad-4ef0-bffe-5f85a2794a05-istio-token" pod "ed247ba4-ebad-4ef0-bffe-5f85a2794a05" (UID: "ed247ba4-ebad-4ef0-bffe-5f85a2794a05"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:23:44.152824 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:44.152765    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ed247ba4-ebad-4ef0-bffe-5f85a2794a05-istio-podinfo" pod "ed247ba4-ebad-4ef0-bffe-5f85a2794a05" (UID: "ed247ba4-ebad-4ef0-bffe-5f85a2794a05"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:23:44.152902 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:44.152855    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed247ba4-ebad-4ef0-bffe-5f85a2794a05-kube-api-access-7hrbr" pod "ed247ba4-ebad-4ef0-bffe-5f85a2794a05" (UID: "ed247ba4-ebad-4ef0-bffe-5f85a2794a05"). InnerVolumeSpecName "kube-api-access-7hrbr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:23:44.251340 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:44.251304    5002 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed247ba4-ebad-4ef0-bffe-5f85a2794a05-workload-socket\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:23:44.251340 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:44.251339    5002 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed247ba4-ebad-4ef0-bffe-5f85a2794a05-istiod-ca-cert\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:23:44.251582 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:44.251351    5002 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed247ba4-ebad-4ef0-bffe-5f85a2794a05-istio-podinfo\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:23:44.251582 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:44.251361    5002 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed247ba4-ebad-4ef0-bffe-5f85a2794a05-credential-socket\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:23:44.251582 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:44.251370    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7hrbr\" (UniqueName: \"kubernetes.io/projected/ed247ba4-ebad-4ef0-bffe-5f85a2794a05-kube-api-access-7hrbr\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:23:44.251582 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:44.251406    5002 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed247ba4-ebad-4ef0-bffe-5f85a2794a05-istio-envoy\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:23:44.251582 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:44.251419    5002 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed247ba4-ebad-4ef0-bffe-5f85a2794a05-istio-token\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:23:44.401523 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:44.401433    5002 generic.go:356] "Generic (PLEG): container finished" podID="ed247ba4-ebad-4ef0-bffe-5f85a2794a05" containerID="07890073533e0669fe179142ae54a40b9c0c01eb2025c91f7e78c8098ff6c414" exitCode=0
Oct 30 01:23:44.401523 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:44.401476    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-009e78-openshift-default-5674f58c7-9fkcq" event={"ID":"ed247ba4-ebad-4ef0-bffe-5f85a2794a05","Type":"ContainerDied","Data":"07890073533e0669fe179142ae54a40b9c0c01eb2025c91f7e78c8098ff6c414"}
Oct 30 01:23:44.401523 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:44.401513    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-009e78-openshift-default-5674f58c7-9fkcq"
Oct 30 01:23:44.401795 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:44.401545    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-009e78-openshift-default-5674f58c7-9fkcq" event={"ID":"ed247ba4-ebad-4ef0-bffe-5f85a2794a05","Type":"ContainerDied","Data":"7032a5d9c27cdc753448ebd7f9d851562d8d31f45f14afc8dbc29178bdabd388"}
Oct 30 01:23:44.401795 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:44.401576    5002 scope.go:122] "RemoveContainer" containerID="07890073533e0669fe179142ae54a40b9c0c01eb2025c91f7e78c8098ff6c414"
Oct 30 01:23:44.412120 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:44.412095    5002 scope.go:122] "RemoveContainer" containerID="07890073533e0669fe179142ae54a40b9c0c01eb2025c91f7e78c8098ff6c414"
Oct 30 01:23:44.412414 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:23:44.412374    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07890073533e0669fe179142ae54a40b9c0c01eb2025c91f7e78c8098ff6c414\": container with ID starting with 07890073533e0669fe179142ae54a40b9c0c01eb2025c91f7e78c8098ff6c414 not found: ID does not exist" containerID="07890073533e0669fe179142ae54a40b9c0c01eb2025c91f7e78c8098ff6c414"
Oct 30 01:23:44.412510 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:44.412424    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07890073533e0669fe179142ae54a40b9c0c01eb2025c91f7e78c8098ff6c414"} err="failed to get container status \"07890073533e0669fe179142ae54a40b9c0c01eb2025c91f7e78c8098ff6c414\": rpc error: code = NotFound desc = could not find container \"07890073533e0669fe179142ae54a40b9c0c01eb2025c91f7e78c8098ff6c414\": container with ID starting with 07890073533e0669fe179142ae54a40b9c0c01eb2025c91f7e78c8098ff6c414 not found: ID does not exist"
Oct 30 01:23:44.457769 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:44.457738    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-009e78-openshift-default-5674f58c7-9fkcq"]
Oct 30 01:23:44.471887 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:44.471851    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-009e78-openshift-default-5674f58c7-9fkcq"]
Oct 30 01:23:44.493773 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:44.493743    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed247ba4-ebad-4ef0-bffe-5f85a2794a05" path="/var/lib/kubelet/pods/ed247ba4-ebad-4ef0-bffe-5f85a2794a05/volumes"
Oct 30 01:23:49.682655 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:49.682619    5002 scope.go:122] "RemoveContainer" containerID="7f91e196f6bf40bcafeb373523735a8df5da484e31b442f6abc950d7ba876573"
Oct 30 01:23:49.692395 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:49.692359    5002 scope.go:122] "RemoveContainer" containerID="3b26ec4ec28249a5da79daf51da509b69c91a086c81e539262b86f492209d527"
Oct 30 01:23:49.701449 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:49.701361    5002 scope.go:122] "RemoveContainer" containerID="627ca91397436ec12df8ab9e25c36229328b229be58cff13224b8234796915c3"
Oct 30 01:23:51.027524 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:51.027481    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed4e8e7a-2xvx7"]
Oct 30 01:23:51.041485 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:51.041455    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed4e8e7a-2xvx7"]
Oct 30 01:23:52.511661 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:52.511627    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05abbb95-d56a-4abb-a43f-c043b2266219" path="/var/lib/kubelet/pods/05abbb95-d56a-4abb-a43f-c043b2266219/volumes"
Oct 30 01:23:52.999652 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:52.999614    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-009e78-54d78f6cf9-sjgcl" podUID="a0a93655-85f4-4cf7-9e80-2ea8ce25bcb5" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:23:52.999652 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:52.999630    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-009e78-847cbfbf7d-vgf6m" podUID="5a0523a2-66bc-4ec0-a5bf-ac5cbafcdbfd" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:23:53.613123 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:53.613006    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7d8z4"]
Oct 30 01:23:54.802026 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:54.801993    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-009e78-847cbfbf7d-vgf6m"
Oct 30 01:23:54.831520 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:54.831497    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-009e78-54d78f6cf9-sjgcl"
Oct 30 01:23:54.877110 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:54.877079    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0a93655-85f4-4cf7-9e80-2ea8ce25bcb5-kube-api-access-rml98\" (UniqueName: \"kubernetes.io/projected/a0a93655-85f4-4cf7-9e80-2ea8ce25bcb5-kube-api-access-rml98\") pod \"a0a93655-85f4-4cf7-9e80-2ea8ce25bcb5\" (UID: \"a0a93655-85f4-4cf7-9e80-2ea8ce25bcb5\") "
Oct 30 01:23:54.877318 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:54.877162    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a0523a2-66bc-4ec0-a5bf-ac5cbafcdbfd-kube-api-access-tv8xm\" (UniqueName: \"kubernetes.io/projected/5a0523a2-66bc-4ec0-a5bf-ac5cbafcdbfd-kube-api-access-tv8xm\") pod \"5a0523a2-66bc-4ec0-a5bf-ac5cbafcdbfd\" (UID: \"5a0523a2-66bc-4ec0-a5bf-ac5cbafcdbfd\") "
Oct 30 01:23:54.877318 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:54.877230    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a0523a2-66bc-4ec0-a5bf-ac5cbafcdbfd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a0523a2-66bc-4ec0-a5bf-ac5cbafcdbfd-plugins-config-volume\") pod \"5a0523a2-66bc-4ec0-a5bf-ac5cbafcdbfd\" (UID: \"5a0523a2-66bc-4ec0-a5bf-ac5cbafcdbfd\") "
Oct 30 01:23:54.877318 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:54.877281    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0a93655-85f4-4cf7-9e80-2ea8ce25bcb5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0a93655-85f4-4cf7-9e80-2ea8ce25bcb5-plugins-config-volume\") pod \"a0a93655-85f4-4cf7-9e80-2ea8ce25bcb5\" (UID: \"a0a93655-85f4-4cf7-9e80-2ea8ce25bcb5\") "
Oct 30 01:23:54.878017 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:54.877987    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0a93655-85f4-4cf7-9e80-2ea8ce25bcb5-plugins-config-volume" pod "a0a93655-85f4-4cf7-9e80-2ea8ce25bcb5" (UID: "a0a93655-85f4-4cf7-9e80-2ea8ce25bcb5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:23:54.878143 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:54.878122    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a0523a2-66bc-4ec0-a5bf-ac5cbafcdbfd-plugins-config-volume" pod "5a0523a2-66bc-4ec0-a5bf-ac5cbafcdbfd" (UID: "5a0523a2-66bc-4ec0-a5bf-ac5cbafcdbfd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:23:54.879907 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:54.879877    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0a93655-85f4-4cf7-9e80-2ea8ce25bcb5-kube-api-access-rml98" pod "a0a93655-85f4-4cf7-9e80-2ea8ce25bcb5" (UID: "a0a93655-85f4-4cf7-9e80-2ea8ce25bcb5"). InnerVolumeSpecName "kube-api-access-rml98". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:23:54.880202 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:54.880179    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a0523a2-66bc-4ec0-a5bf-ac5cbafcdbfd-kube-api-access-tv8xm" pod "5a0523a2-66bc-4ec0-a5bf-ac5cbafcdbfd" (UID: "5a0523a2-66bc-4ec0-a5bf-ac5cbafcdbfd"). InnerVolumeSpecName "kube-api-access-tv8xm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:23:54.978950 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:54.978913    5002 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0a93655-85f4-4cf7-9e80-2ea8ce25bcb5-plugins-config-volume\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:23:54.978950 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:54.978945    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rml98\" (UniqueName: \"kubernetes.io/projected/a0a93655-85f4-4cf7-9e80-2ea8ce25bcb5-kube-api-access-rml98\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:23:54.978950 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:54.978955    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tv8xm\" (UniqueName: \"kubernetes.io/projected/5a0523a2-66bc-4ec0-a5bf-ac5cbafcdbfd-kube-api-access-tv8xm\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:23:54.979191 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:54.978963    5002 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a0523a2-66bc-4ec0-a5bf-ac5cbafcdbfd-plugins-config-volume\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:23:55.473981 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:55.473939    5002 generic.go:356] "Generic (PLEG): container finished" podID="a0a93655-85f4-4cf7-9e80-2ea8ce25bcb5" containerID="70e6127b29dc6420849daa9cd497a262f43584dacb52352a76584bcfbd32c3aa" exitCode=0
Oct 30 01:23:55.474202 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:55.474000    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-009e78-54d78f6cf9-sjgcl"
Oct 30 01:23:55.474202 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:55.474017    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-009e78-54d78f6cf9-sjgcl" event={"ID":"a0a93655-85f4-4cf7-9e80-2ea8ce25bcb5","Type":"ContainerDied","Data":"70e6127b29dc6420849daa9cd497a262f43584dacb52352a76584bcfbd32c3aa"}
Oct 30 01:23:55.474202 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:55.474095    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-009e78-54d78f6cf9-sjgcl" event={"ID":"a0a93655-85f4-4cf7-9e80-2ea8ce25bcb5","Type":"ContainerDied","Data":"f605b853ec1bd0e754e74a585d9087e1a5d33a2561199027fae067e8cf8616b3"}
Oct 30 01:23:55.474202 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:55.474148    5002 scope.go:122] "RemoveContainer" containerID="70e6127b29dc6420849daa9cd497a262f43584dacb52352a76584bcfbd32c3aa"
Oct 30 01:23:55.475813 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:55.475783    5002 generic.go:356] "Generic (PLEG): container finished" podID="5a0523a2-66bc-4ec0-a5bf-ac5cbafcdbfd" containerID="de20524b1394285bbc1d024f323d1fee8647693e0395af6a079b4befd317d87a" exitCode=0
Oct 30 01:23:55.475956 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:55.475840    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-009e78-847cbfbf7d-vgf6m"
Oct 30 01:23:55.475956 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:55.475885    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-009e78-847cbfbf7d-vgf6m" event={"ID":"5a0523a2-66bc-4ec0-a5bf-ac5cbafcdbfd","Type":"ContainerDied","Data":"de20524b1394285bbc1d024f323d1fee8647693e0395af6a079b4befd317d87a"}
Oct 30 01:23:55.475956 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:55.475944    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-009e78-847cbfbf7d-vgf6m" event={"ID":"5a0523a2-66bc-4ec0-a5bf-ac5cbafcdbfd","Type":"ContainerDied","Data":"c2020329b9e8c2fde20084b7282dfcb7bd61a461d8990e00a29f0ec01a833442"}
Oct 30 01:23:55.485928 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:55.485829    5002 scope.go:122] "RemoveContainer" containerID="70e6127b29dc6420849daa9cd497a262f43584dacb52352a76584bcfbd32c3aa"
Oct 30 01:23:55.486240 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:23:55.486193    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70e6127b29dc6420849daa9cd497a262f43584dacb52352a76584bcfbd32c3aa\": container with ID starting with 70e6127b29dc6420849daa9cd497a262f43584dacb52352a76584bcfbd32c3aa not found: ID does not exist" containerID="70e6127b29dc6420849daa9cd497a262f43584dacb52352a76584bcfbd32c3aa"
Oct 30 01:23:55.486240 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:55.486226    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70e6127b29dc6420849daa9cd497a262f43584dacb52352a76584bcfbd32c3aa"} err="failed to get container status \"70e6127b29dc6420849daa9cd497a262f43584dacb52352a76584bcfbd32c3aa\": rpc error: code = NotFound desc = could not find container \"70e6127b29dc6420849daa9cd497a262f43584dacb52352a76584bcfbd32c3aa\": container with ID starting with 70e6127b29dc6420849daa9cd497a262f43584dacb52352a76584bcfbd32c3aa not found: ID does not exist"
Oct 30 01:23:55.486240 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:55.486249    5002 scope.go:122] "RemoveContainer" containerID="de20524b1394285bbc1d024f323d1fee8647693e0395af6a079b4befd317d87a"
Oct 30 01:23:55.496166 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:55.496142    5002 scope.go:122] "RemoveContainer" containerID="de20524b1394285bbc1d024f323d1fee8647693e0395af6a079b4befd317d87a"
Oct 30 01:23:55.496441 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:23:55.496419    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de20524b1394285bbc1d024f323d1fee8647693e0395af6a079b4befd317d87a\": container with ID starting with de20524b1394285bbc1d024f323d1fee8647693e0395af6a079b4befd317d87a not found: ID does not exist" containerID="de20524b1394285bbc1d024f323d1fee8647693e0395af6a079b4befd317d87a"
Oct 30 01:23:55.496522 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:55.496449    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de20524b1394285bbc1d024f323d1fee8647693e0395af6a079b4befd317d87a"} err="failed to get container status \"de20524b1394285bbc1d024f323d1fee8647693e0395af6a079b4befd317d87a\": rpc error: code = NotFound desc = could not find container \"de20524b1394285bbc1d024f323d1fee8647693e0395af6a079b4befd317d87a\": container with ID starting with de20524b1394285bbc1d024f323d1fee8647693e0395af6a079b4befd317d87a not found: ID does not exist"
Oct 30 01:23:55.546591 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:55.546565    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-009e78-54d78f6cf9-sjgcl"]
Oct 30 01:23:55.551035 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:55.551013    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-009e78-54d78f6cf9-sjgcl"]
Oct 30 01:23:55.587834 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:55.587805    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-009e78-847cbfbf7d-vgf6m"]
Oct 30 01:23:55.591132 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:55.591104    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-009e78-847cbfbf7d-vgf6m"]
Oct 30 01:23:55.773204 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:55.773110    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-1a11cf/distinct-2-w2-1a11cf-kserve-758bb6dd57-j5f2k"]
Oct 30 01:23:55.778315 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:55.778285    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-1a11cf/distinct-w2-1a11cf-kserve-854cf4cdf8-cddw6"]
Oct 30 01:23:56.493538 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:56.493502    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a0523a2-66bc-4ec0-a5bf-ac5cbafcdbfd" path="/var/lib/kubelet/pods/5a0523a2-66bc-4ec0-a5bf-ac5cbafcdbfd/volumes"
Oct 30 01:23:56.493965 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:56.493905    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0a93655-85f4-4cf7-9e80-2ea8ce25bcb5" path="/var/lib/kubelet/pods/a0a93655-85f4-4cf7-9e80-2ea8ce25bcb5/volumes"
Oct 30 01:23:58.276109 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:58.276070    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-rnd66"]
Oct 30 01:23:58.276736 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:23:58.276368    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-rnd66" podUID="4df0b3e0-04ca-440d-9b4c-01635ee412e7" containerName="istio-proxy" containerID="cri-o://26d3b1ec3ce483ad4c1644f36ce78ed7f2d22ecdfa2fbe11d3306267a482b6f7" gracePeriod=30
Oct 30 01:24:02.804182 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:02.804138    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b864cda-97k9j"]
Oct 30 01:24:02.804768 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:02.804750    5002 cpu_manager.go:414] "containerMap: removing container" podUID="7077b8b8-1f35-49c4-9e58-762297c046cf" containerName="istio-proxy"
Oct 30 01:24:02.804811 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:02.804771    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7077b8b8-1f35-49c4-9e58-762297c046cf" containerName="istio-proxy"
Oct 30 01:24:02.804811 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:02.804797    5002 cpu_manager.go:414] "containerMap: removing container" podUID="7077b8b8-1f35-49c4-9e58-762297c046cf" containerName="istio-proxy"
Oct 30 01:24:02.804811 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:02.804805    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7077b8b8-1f35-49c4-9e58-762297c046cf" containerName="istio-proxy"
Oct 30 01:24:02.804932 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:02.804819    5002 cpu_manager.go:414] "containerMap: removing container" podUID="60bf7933-a5cb-447a-96cc-e28389894a20" containerName="payload-processing"
Oct 30 01:24:02.804932 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:02.804828    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60bf7933-a5cb-447a-96cc-e28389894a20" containerName="payload-processing"
Oct 30 01:24:02.804932 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:02.804840    5002 cpu_manager.go:414] "containerMap: removing container" podUID="e09703cf-6ee3-45e9-893d-5a626ca636c9" containerName="istio-proxy"
Oct 30 01:24:02.804932 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:02.804847    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e09703cf-6ee3-45e9-893d-5a626ca636c9" containerName="istio-proxy"
Oct 30 01:24:02.804932 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:02.804858    5002 cpu_manager.go:414] "containerMap: removing container" podUID="a02e27db-2cc5-404f-8881-6efd27b6f48a" containerName="payload-pre-processing"
Oct 30 01:24:02.804932 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:02.804866    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a02e27db-2cc5-404f-8881-6efd27b6f48a" containerName="payload-pre-processing"
Oct 30 01:24:02.804932 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:02.804883    5002 cpu_manager.go:414] "containerMap: removing container" podUID="ed247ba4-ebad-4ef0-bffe-5f85a2794a05" containerName="istio-proxy"
Oct 30 01:24:02.804932 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:02.804890    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed247ba4-ebad-4ef0-bffe-5f85a2794a05" containerName="istio-proxy"
Oct 30 01:24:02.804932 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:02.804902    5002 cpu_manager.go:414] "containerMap: removing container" podUID="b9058f9e-a7e7-4f30-87be-e037e5378b3c" containerName="istio-proxy"
Oct 30 01:24:02.804932 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:02.804909    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9058f9e-a7e7-4f30-87be-e037e5378b3c" containerName="istio-proxy"
Oct 30 01:24:02.804932 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:02.804923    5002 cpu_manager.go:414] "containerMap: removing container" podUID="f86c006d-5087-4443-b907-c2a30815b206" containerName="istio-proxy"
Oct 30 01:24:02.804932 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:02.804931    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f86c006d-5087-4443-b907-c2a30815b206" containerName="istio-proxy"
Oct 30 01:24:02.805310 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:02.804943    5002 cpu_manager.go:414] "containerMap: removing container" podUID="5a0523a2-66bc-4ec0-a5bf-ac5cbafcdbfd" containerName="payload-processing"
Oct 30 01:24:02.805310 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:02.804950    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a0523a2-66bc-4ec0-a5bf-ac5cbafcdbfd" containerName="payload-processing"
Oct 30 01:24:02.805310 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:02.804974    5002 cpu_manager.go:414] "containerMap: removing container" podUID="0ff20380-75e5-4089-9f54-50d1917eff55" containerName="istio-proxy"
Oct 30 01:24:02.805310 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:02.804983    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ff20380-75e5-4089-9f54-50d1917eff55" containerName="istio-proxy"
Oct 30 01:24:02.805310 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:02.804994    5002 cpu_manager.go:414] "containerMap: removing container" podUID="5de8bebf-cfad-49da-9a98-08aedde4442c" containerName="main"
Oct 30 01:24:02.805310 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:02.805001    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5de8bebf-cfad-49da-9a98-08aedde4442c" containerName="main"
Oct 30 01:24:02.805310 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:02.805014    5002 cpu_manager.go:414] "containerMap: removing container" podUID="e09703cf-6ee3-45e9-893d-5a626ca636c9" containerName="istio-proxy"
Oct 30 01:24:02.805310 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:02.805021    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e09703cf-6ee3-45e9-893d-5a626ca636c9" containerName="istio-proxy"
Oct 30 01:24:02.805310 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:02.805030    5002 cpu_manager.go:414] "containerMap: removing container" podUID="b38dbda1-9a06-461d-8d0a-a9413b76d333" containerName="istio-proxy"
Oct 30 01:24:02.805310 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:02.805038    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b38dbda1-9a06-461d-8d0a-a9413b76d333" containerName="istio-proxy"
Oct 30 01:24:02.805310 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:02.805050    5002 cpu_manager.go:414] "containerMap: removing container" podUID="b02a73f0-b8fd-4b73-8437-c940fc676a67" containerName="payload-pre-processing"
Oct 30 01:24:02.805310 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:02.805058    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b02a73f0-b8fd-4b73-8437-c940fc676a67" containerName="payload-pre-processing"
Oct 30 01:24:02.805310 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:02.805070    5002 cpu_manager.go:414] "containerMap: removing container" podUID="7121e81e-f903-42c0-a7a4-bc18d49194e9" containerName="main"
Oct 30 01:24:02.805310 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:02.805077    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7121e81e-f903-42c0-a7a4-bc18d49194e9" containerName="main"
Oct 30 01:24:02.805310 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:02.805089    5002 cpu_manager.go:414] "containerMap: removing container" podUID="b38dbda1-9a06-461d-8d0a-a9413b76d333" containerName="istio-proxy"
Oct 30 01:24:02.805310 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:02.805096    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b38dbda1-9a06-461d-8d0a-a9413b76d333" containerName="istio-proxy"
Oct 30 01:24:02.805310 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:02.805112    5002 cpu_manager.go:414] "containerMap: removing container" podUID="8c2a594b-3bbf-4c94-8a9f-7b79c663f135" containerName="istio-proxy"
Oct 30 01:24:02.805310 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:02.805119    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c2a594b-3bbf-4c94-8a9f-7b79c663f135" containerName="istio-proxy"
Oct 30 01:24:02.805310 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:02.805129    5002 cpu_manager.go:414] "containerMap: removing container" podUID="d3af29b6-2cf9-451e-bf6b-7009c5f20f26" containerName="revoke-keys"
Oct 30 01:24:02.805310 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:02.805137    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3af29b6-2cf9-451e-bf6b-7009c5f20f26" containerName="revoke-keys"
Oct 30 01:24:02.805310 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:02.805152    5002 cpu_manager.go:414] "containerMap: removing container" podUID="a0a93655-85f4-4cf7-9e80-2ea8ce25bcb5" containerName="payload-pre-processing"
Oct 30 01:24:02.805310 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:02.805159    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0a93655-85f4-4cf7-9e80-2ea8ce25bcb5" containerName="payload-pre-processing"
Oct 30 01:24:02.805310 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:02.805169    5002 cpu_manager.go:414] "containerMap: removing container" podUID="735d76ff-0add-4acc-b5e9-4ebeb27c0e9a" containerName="istio-proxy"
Oct 30 01:24:02.805310 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:02.805176    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="735d76ff-0add-4acc-b5e9-4ebeb27c0e9a" containerName="istio-proxy"
Oct 30 01:24:02.805310 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:02.805263    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="7077b8b8-1f35-49c4-9e58-762297c046cf" containerName="istio-proxy"
Oct 30 01:24:02.805310 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:02.805278    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="7077b8b8-1f35-49c4-9e58-762297c046cf" containerName="istio-proxy"
Oct 30 01:24:02.805310 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:02.805288    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="60bf7933-a5cb-447a-96cc-e28389894a20" containerName="payload-processing"
Oct 30 01:24:02.805310 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:02.805296    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="e09703cf-6ee3-45e9-893d-5a626ca636c9" containerName="istio-proxy"
Oct 30 01:24:02.805310 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:02.805306    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="a02e27db-2cc5-404f-8881-6efd27b6f48a" containerName="payload-pre-processing"
Oct 30 01:24:02.805310 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:02.805315    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed247ba4-ebad-4ef0-bffe-5f85a2794a05" containerName="istio-proxy"
Oct 30 01:24:02.806286 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:02.805324    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9058f9e-a7e7-4f30-87be-e037e5378b3c" containerName="istio-proxy"
Oct 30 01:24:02.806286 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:02.805332    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="f86c006d-5087-4443-b907-c2a30815b206" containerName="istio-proxy"
Oct 30 01:24:02.806286 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:02.805341    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a0523a2-66bc-4ec0-a5bf-ac5cbafcdbfd" containerName="payload-processing"
Oct 30 01:24:02.806286 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:02.805353    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ff20380-75e5-4089-9f54-50d1917eff55" containerName="istio-proxy"
Oct 30 01:24:02.806286 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:02.805362    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="5de8bebf-cfad-49da-9a98-08aedde4442c" containerName="main"
Oct 30 01:24:02.806286 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:02.805369    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="e09703cf-6ee3-45e9-893d-5a626ca636c9" containerName="istio-proxy"
Oct 30 01:24:02.806286 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:02.805393    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="b38dbda1-9a06-461d-8d0a-a9413b76d333" containerName="istio-proxy"
Oct 30 01:24:02.806286 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:02.805402    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="b02a73f0-b8fd-4b73-8437-c940fc676a67" containerName="payload-pre-processing"
Oct 30 01:24:02.806286 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:02.805411    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="7121e81e-f903-42c0-a7a4-bc18d49194e9" containerName="main"
Oct 30 01:24:02.806286 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:02.805419    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="b38dbda1-9a06-461d-8d0a-a9413b76d333" containerName="istio-proxy"
Oct 30 01:24:02.806286 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:02.805428    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c2a594b-3bbf-4c94-8a9f-7b79c663f135" containerName="istio-proxy"
Oct 30 01:24:02.806286 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:02.805438    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3af29b6-2cf9-451e-bf6b-7009c5f20f26" containerName="revoke-keys"
Oct 30 01:24:02.806286 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:02.805448    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0a93655-85f4-4cf7-9e80-2ea8ce25bcb5" containerName="payload-pre-processing"
Oct 30 01:24:02.806286 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:02.805458    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c2a594b-3bbf-4c94-8a9f-7b79c663f135" containerName="istio-proxy"
Oct 30 01:24:02.806286 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:02.805466    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="735d76ff-0add-4acc-b5e9-4ebeb27c0e9a" containerName="istio-proxy"
Oct 30 01:24:02.810184 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:02.810162    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b864cda-97k9j"
Oct 30 01:24:02.826373 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:02.826343    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b864cda-97k9j"]
Oct 30 01:24:02.838200 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:02.838169    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:24:02.838359 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:02.838337    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:24:02.840724 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:02.840706    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:24:02.840946 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:02.840929    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5phsh\""
Oct 30 01:24:02.859893 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:02.859864    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/921cb648-96f3-483f-8984-0359a6208ce0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1b864cda-97k9j\" (UID: \"921cb648-96f3-483f-8984-0359a6208ce0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b864cda-97k9j"
Oct 30 01:24:02.860040 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:02.859928    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/921cb648-96f3-483f-8984-0359a6208ce0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1b864cda-97k9j\" (UID: \"921cb648-96f3-483f-8984-0359a6208ce0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b864cda-97k9j"
Oct 30 01:24:02.860101 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:02.860079    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmljv\" (UniqueName: \"kubernetes.io/projected/921cb648-96f3-483f-8984-0359a6208ce0-kube-api-access-wmljv\") pod \"maas-api-revoke-sub-1b864cda-97k9j\" (UID: \"921cb648-96f3-483f-8984-0359a6208ce0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b864cda-97k9j"
Oct 30 01:24:02.961600 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:02.961565    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/921cb648-96f3-483f-8984-0359a6208ce0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1b864cda-97k9j\" (UID: \"921cb648-96f3-483f-8984-0359a6208ce0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b864cda-97k9j"
Oct 30 01:24:02.961786 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:02.961627    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmljv\" (UniqueName: \"kubernetes.io/projected/921cb648-96f3-483f-8984-0359a6208ce0-kube-api-access-wmljv\") pod \"maas-api-revoke-sub-1b864cda-97k9j\" (UID: \"921cb648-96f3-483f-8984-0359a6208ce0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b864cda-97k9j"
Oct 30 01:24:02.961786 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:02.961714    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/921cb648-96f3-483f-8984-0359a6208ce0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1b864cda-97k9j\" (UID: \"921cb648-96f3-483f-8984-0359a6208ce0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b864cda-97k9j"
Oct 30 01:24:02.962466 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:02.962438    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/921cb648-96f3-483f-8984-0359a6208ce0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1b864cda-97k9j\" (UID: \"921cb648-96f3-483f-8984-0359a6208ce0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b864cda-97k9j"
Oct 30 01:24:02.964209 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:02.964179    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/921cb648-96f3-483f-8984-0359a6208ce0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1b864cda-97k9j\" (UID: \"921cb648-96f3-483f-8984-0359a6208ce0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b864cda-97k9j"
Oct 30 01:24:02.986736 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:02.986707    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmljv\" (UniqueName: \"kubernetes.io/projected/921cb648-96f3-483f-8984-0359a6208ce0-kube-api-access-wmljv\") pod \"maas-api-revoke-sub-1b864cda-97k9j\" (UID: \"921cb648-96f3-483f-8984-0359a6208ce0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b864cda-97k9j"
Oct 30 01:24:03.123828 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:03.123738    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b864cda-97k9j"
Oct 30 01:24:03.426250 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:03.426204    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b864cda-97k9j"]
Oct 30 01:24:03.427229 ip-10-0-128-7 kubenswrapper[5002]: W1030 01:24:03.427198    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod921cb648_96f3_483f_8984_0359a6208ce0.slice/crio-983511c1a09bd109b4670a4854edbe166ff3598608fa317de74130e24b40b608 WatchSource:0}: Error finding container 983511c1a09bd109b4670a4854edbe166ff3598608fa317de74130e24b40b608: Status 404 returned error can't find the container with id 983511c1a09bd109b4670a4854edbe166ff3598608fa317de74130e24b40b608
Oct 30 01:24:03.521996 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:03.521956    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b864cda-97k9j" event={"ID":"921cb648-96f3-483f-8984-0359a6208ce0","Type":"ContainerStarted","Data":"983511c1a09bd109b4670a4854edbe166ff3598608fa317de74130e24b40b608"}
Oct 30 01:24:03.524073 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:03.524039    5002 generic.go:356] "Generic (PLEG): container finished" podID="4df0b3e0-04ca-440d-9b4c-01635ee412e7" containerID="26d3b1ec3ce483ad4c1644f36ce78ed7f2d22ecdfa2fbe11d3306267a482b6f7" exitCode=0
Oct 30 01:24:03.524224 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:03.524097    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-rnd66" event={"ID":"4df0b3e0-04ca-440d-9b4c-01635ee412e7","Type":"ContainerDied","Data":"26d3b1ec3ce483ad4c1644f36ce78ed7f2d22ecdfa2fbe11d3306267a482b6f7"}
Oct 30 01:24:03.561073 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:03.561049    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-rnd66"
Oct 30 01:24:03.668411 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:03.668364    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4df0b3e0-04ca-440d-9b4c-01635ee412e7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4df0b3e0-04ca-440d-9b4c-01635ee412e7-credential-socket\") pod \"4df0b3e0-04ca-440d-9b4c-01635ee412e7\" (UID: \"4df0b3e0-04ca-440d-9b4c-01635ee412e7\") "
Oct 30 01:24:03.668577 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:03.668432    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4df0b3e0-04ca-440d-9b4c-01635ee412e7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4df0b3e0-04ca-440d-9b4c-01635ee412e7-workload-socket\") pod \"4df0b3e0-04ca-440d-9b4c-01635ee412e7\" (UID: \"4df0b3e0-04ca-440d-9b4c-01635ee412e7\") "
Oct 30 01:24:03.668577 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:03.668456    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4df0b3e0-04ca-440d-9b4c-01635ee412e7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4df0b3e0-04ca-440d-9b4c-01635ee412e7-istio-envoy\") pod \"4df0b3e0-04ca-440d-9b4c-01635ee412e7\" (UID: \"4df0b3e0-04ca-440d-9b4c-01635ee412e7\") "
Oct 30 01:24:03.668577 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:03.668498    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4df0b3e0-04ca-440d-9b4c-01635ee412e7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4df0b3e0-04ca-440d-9b4c-01635ee412e7-istiod-ca-cert\") pod \"4df0b3e0-04ca-440d-9b4c-01635ee412e7\" (UID: \"4df0b3e0-04ca-440d-9b4c-01635ee412e7\") "
Oct 30 01:24:03.668577 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:03.668525    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4df0b3e0-04ca-440d-9b4c-01635ee412e7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4df0b3e0-04ca-440d-9b4c-01635ee412e7-workload-certs\") pod \"4df0b3e0-04ca-440d-9b4c-01635ee412e7\" (UID: \"4df0b3e0-04ca-440d-9b4c-01635ee412e7\") "
Oct 30 01:24:03.668887 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:03.668600    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4df0b3e0-04ca-440d-9b4c-01635ee412e7-kube-api-access-dwd2v\" (UniqueName: \"kubernetes.io/projected/4df0b3e0-04ca-440d-9b4c-01635ee412e7-kube-api-access-dwd2v\") pod \"4df0b3e0-04ca-440d-9b4c-01635ee412e7\" (UID: \"4df0b3e0-04ca-440d-9b4c-01635ee412e7\") "
Oct 30 01:24:03.668887 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:03.668671    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4df0b3e0-04ca-440d-9b4c-01635ee412e7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4df0b3e0-04ca-440d-9b4c-01635ee412e7-istio-data\") pod \"4df0b3e0-04ca-440d-9b4c-01635ee412e7\" (UID: \"4df0b3e0-04ca-440d-9b4c-01635ee412e7\") "
Oct 30 01:24:03.668887 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:03.668720    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4df0b3e0-04ca-440d-9b4c-01635ee412e7-credential-socket" pod "4df0b3e0-04ca-440d-9b4c-01635ee412e7" (UID: "4df0b3e0-04ca-440d-9b4c-01635ee412e7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:24:03.668887 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:03.668730    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4df0b3e0-04ca-440d-9b4c-01635ee412e7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4df0b3e0-04ca-440d-9b4c-01635ee412e7-istio-podinfo\") pod \"4df0b3e0-04ca-440d-9b4c-01635ee412e7\" (UID: \"4df0b3e0-04ca-440d-9b4c-01635ee412e7\") "
Oct 30 01:24:03.668887 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:03.668749    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4df0b3e0-04ca-440d-9b4c-01635ee412e7-workload-socket" pod "4df0b3e0-04ca-440d-9b4c-01635ee412e7" (UID: "4df0b3e0-04ca-440d-9b4c-01635ee412e7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:24:03.668887 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:03.668798    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4df0b3e0-04ca-440d-9b4c-01635ee412e7-istio-token\" (UniqueName: \"kubernetes.io/projected/4df0b3e0-04ca-440d-9b4c-01635ee412e7-istio-token\") pod \"4df0b3e0-04ca-440d-9b4c-01635ee412e7\" (UID: \"4df0b3e0-04ca-440d-9b4c-01635ee412e7\") "
Oct 30 01:24:03.668887 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:03.668810    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4df0b3e0-04ca-440d-9b4c-01635ee412e7-workload-certs" pod "4df0b3e0-04ca-440d-9b4c-01635ee412e7" (UID: "4df0b3e0-04ca-440d-9b4c-01635ee412e7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:24:03.669224 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:03.668997    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4df0b3e0-04ca-440d-9b4c-01635ee412e7-istio-data" pod "4df0b3e0-04ca-440d-9b4c-01635ee412e7" (UID: "4df0b3e0-04ca-440d-9b4c-01635ee412e7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:24:03.669224 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:03.669204    5002 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4df0b3e0-04ca-440d-9b4c-01635ee412e7-credential-socket\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:24:03.669347 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:03.669224    5002 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4df0b3e0-04ca-440d-9b4c-01635ee412e7-workload-socket\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:24:03.669347 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:03.669238    5002 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4df0b3e0-04ca-440d-9b4c-01635ee412e7-workload-certs\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:24:03.669347 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:03.669251    5002 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4df0b3e0-04ca-440d-9b4c-01635ee412e7-istio-data\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:24:03.669571 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:03.669397    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4df0b3e0-04ca-440d-9b4c-01635ee412e7-istiod-ca-cert" pod "4df0b3e0-04ca-440d-9b4c-01635ee412e7" (UID: "4df0b3e0-04ca-440d-9b4c-01635ee412e7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:24:03.671246 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:03.671222    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4df0b3e0-04ca-440d-9b4c-01635ee412e7-istio-token" pod "4df0b3e0-04ca-440d-9b4c-01635ee412e7" (UID: "4df0b3e0-04ca-440d-9b4c-01635ee412e7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:24:03.671363 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:03.671298    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4df0b3e0-04ca-440d-9b4c-01635ee412e7-istio-envoy" pod "4df0b3e0-04ca-440d-9b4c-01635ee412e7" (UID: "4df0b3e0-04ca-440d-9b4c-01635ee412e7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:24:03.671363 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:03.671338    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4df0b3e0-04ca-440d-9b4c-01635ee412e7-kube-api-access-dwd2v" pod "4df0b3e0-04ca-440d-9b4c-01635ee412e7" (UID: "4df0b3e0-04ca-440d-9b4c-01635ee412e7"). InnerVolumeSpecName "kube-api-access-dwd2v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:24:03.671468 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:03.671360    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4df0b3e0-04ca-440d-9b4c-01635ee412e7-istio-podinfo" pod "4df0b3e0-04ca-440d-9b4c-01635ee412e7" (UID: "4df0b3e0-04ca-440d-9b4c-01635ee412e7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:24:03.743701 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:03.743663    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-021d6d/simulator-w6-021d6d-kserve-6cc97fdf86-dqtpw"]
Oct 30 01:24:03.770335 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:03.770299    5002 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4df0b3e0-04ca-440d-9b4c-01635ee412e7-istio-podinfo\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:24:03.770335 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:03.770326    5002 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4df0b3e0-04ca-440d-9b4c-01635ee412e7-istio-token\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:24:03.770335 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:03.770335    5002 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4df0b3e0-04ca-440d-9b4c-01635ee412e7-istio-envoy\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:24:03.770645 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:03.770346    5002 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4df0b3e0-04ca-440d-9b4c-01635ee412e7-istiod-ca-cert\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:24:03.770645 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:03.770360    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwd2v\" (UniqueName: \"kubernetes.io/projected/4df0b3e0-04ca-440d-9b4c-01635ee412e7-kube-api-access-dwd2v\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:24:04.530975 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:04.530933    5002 generic.go:356] "Generic (PLEG): container finished" podID="921cb648-96f3-483f-8984-0359a6208ce0" containerID="428ca539e50fc07913a9ae98b9d613185c460779b293d9d92fd3da8d764eedc5" exitCode=0
Oct 30 01:24:04.531439 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:04.531012    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b864cda-97k9j" event={"ID":"921cb648-96f3-483f-8984-0359a6208ce0","Type":"ContainerDied","Data":"428ca539e50fc07913a9ae98b9d613185c460779b293d9d92fd3da8d764eedc5"}
Oct 30 01:24:04.532490 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:04.532460    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-rnd66"
Oct 30 01:24:04.532600 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:04.532508    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-rnd66" event={"ID":"4df0b3e0-04ca-440d-9b4c-01635ee412e7","Type":"ContainerDied","Data":"1236175ca7da2a4fb9c7c52c8492f25255f7cdb00ba88e07bbce87a2f5efea00"}
Oct 30 01:24:04.532600 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:04.532566    5002 scope.go:122] "RemoveContainer" containerID="26d3b1ec3ce483ad4c1644f36ce78ed7f2d22ecdfa2fbe11d3306267a482b6f7"
Oct 30 01:24:04.718210 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:04.718176    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-rnd66"]
Oct 30 01:24:04.733211 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:04.733176    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-rnd66"]
Oct 30 01:24:05.690688 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:05.690659    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b864cda-97k9j"
Oct 30 01:24:05.790007 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:05.789914    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/921cb648-96f3-483f-8984-0359a6208ce0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/921cb648-96f3-483f-8984-0359a6208ce0-maas-api-openshift-ca\") pod \"921cb648-96f3-483f-8984-0359a6208ce0\" (UID: \"921cb648-96f3-483f-8984-0359a6208ce0\") "
Oct 30 01:24:05.790007 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:05.789965    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/921cb648-96f3-483f-8984-0359a6208ce0-kube-api-access-wmljv\" (UniqueName: \"kubernetes.io/projected/921cb648-96f3-483f-8984-0359a6208ce0-kube-api-access-wmljv\") pod \"921cb648-96f3-483f-8984-0359a6208ce0\" (UID: \"921cb648-96f3-483f-8984-0359a6208ce0\") "
Oct 30 01:24:05.790224 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:05.790158    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/921cb648-96f3-483f-8984-0359a6208ce0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/921cb648-96f3-483f-8984-0359a6208ce0-maas-api-xks-ca\") pod \"921cb648-96f3-483f-8984-0359a6208ce0\" (UID: \"921cb648-96f3-483f-8984-0359a6208ce0\") "
Oct 30 01:24:05.790437 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:05.790372    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/921cb648-96f3-483f-8984-0359a6208ce0-maas-api-openshift-ca" pod "921cb648-96f3-483f-8984-0359a6208ce0" (UID: "921cb648-96f3-483f-8984-0359a6208ce0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:24:05.790582 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:05.790557    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/921cb648-96f3-483f-8984-0359a6208ce0-maas-api-openshift-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:24:05.792194 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:05.792167    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/921cb648-96f3-483f-8984-0359a6208ce0-maas-api-xks-ca" pod "921cb648-96f3-483f-8984-0359a6208ce0" (UID: "921cb648-96f3-483f-8984-0359a6208ce0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:24:05.792304 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:05.792249    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/921cb648-96f3-483f-8984-0359a6208ce0-kube-api-access-wmljv" pod "921cb648-96f3-483f-8984-0359a6208ce0" (UID: "921cb648-96f3-483f-8984-0359a6208ce0"). InnerVolumeSpecName "kube-api-access-wmljv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:24:05.891829 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:05.891789    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wmljv\" (UniqueName: \"kubernetes.io/projected/921cb648-96f3-483f-8984-0359a6208ce0-kube-api-access-wmljv\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:24:05.891829 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:05.891826    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/921cb648-96f3-483f-8984-0359a6208ce0-maas-api-xks-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:24:06.510715 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:06.510673    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4df0b3e0-04ca-440d-9b4c-01635ee412e7" path="/var/lib/kubelet/pods/4df0b3e0-04ca-440d-9b4c-01635ee412e7/volumes"
Oct 30 01:24:06.546830 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:06.546795    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b864cda-97k9j" event={"ID":"921cb648-96f3-483f-8984-0359a6208ce0","Type":"ContainerDied","Data":"983511c1a09bd109b4670a4854edbe166ff3598608fa317de74130e24b40b608"}
Oct 30 01:24:06.546985 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:06.546837    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b864cda-97k9j"
Oct 30 01:24:06.546985 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:06.546853    5002 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="983511c1a09bd109b4670a4854edbe166ff3598608fa317de74130e24b40b608"
Oct 30 01:24:07.197141 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:07.197110    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7d8z4"]
Oct 30 01:24:10.796321 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:10.796255    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-1a11cf/distinct-2-w2-1a11cf-kserve-758bb6dd57-j5f2k" podUID="f2d66dea-9f0b-43f3-a774-bb065e88f378" containerName="main" containerID="cri-o://b601a0f189cbbee81fcc9087c896e022ebe09ab1a82eee2857b16e53c827c75c" gracePeriod=45
Oct 30 01:24:10.799460 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:10.799425    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-1a11cf/distinct-w2-1a11cf-kserve-854cf4cdf8-cddw6" podUID="32a03478-46eb-4363-97b6-51a32c4099c7" containerName="main" containerID="cri-o://dcb0a41337b62c29232f8b4261c4e6d0adf60e2e9d0dd0073c743932690095a8" gracePeriod=45
Oct 30 01:24:11.084556 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:11.084525    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-1a11cf/distinct-2-w2-1a11cf-kserve-758bb6dd57-j5f2k"
Oct 30 01:24:11.087577 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:11.087551    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-1a11cf/distinct-w2-1a11cf-kserve-854cf4cdf8-cddw6"
Oct 30 01:24:11.243426 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:11.243355    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f2d66dea-9f0b-43f3-a774-bb065e88f378-tls-certs\" (UniqueName: \"kubernetes.io/secret/f2d66dea-9f0b-43f3-a774-bb065e88f378-tls-certs\") pod \"f2d66dea-9f0b-43f3-a774-bb065e88f378\" (UID: \"f2d66dea-9f0b-43f3-a774-bb065e88f378\") "
Oct 30 01:24:11.243651 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:11.243450    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/32a03478-46eb-4363-97b6-51a32c4099c7-tls-certs\" (UniqueName: \"kubernetes.io/secret/32a03478-46eb-4363-97b6-51a32c4099c7-tls-certs\") pod \"32a03478-46eb-4363-97b6-51a32c4099c7\" (UID: \"32a03478-46eb-4363-97b6-51a32c4099c7\") "
Oct 30 01:24:11.243651 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:11.243486    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32a03478-46eb-4363-97b6-51a32c4099c7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32a03478-46eb-4363-97b6-51a32c4099c7-model-cache\") pod \"32a03478-46eb-4363-97b6-51a32c4099c7\" (UID: \"32a03478-46eb-4363-97b6-51a32c4099c7\") "
Oct 30 01:24:11.243651 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:11.243526    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32a03478-46eb-4363-97b6-51a32c4099c7-home\" (UniqueName: \"kubernetes.io/empty-dir/32a03478-46eb-4363-97b6-51a32c4099c7-home\") pod \"32a03478-46eb-4363-97b6-51a32c4099c7\" (UID: \"32a03478-46eb-4363-97b6-51a32c4099c7\") "
Oct 30 01:24:11.243651 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:11.243543    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2d66dea-9f0b-43f3-a774-bb065e88f378-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2d66dea-9f0b-43f3-a774-bb065e88f378-dshm\") pod \"f2d66dea-9f0b-43f3-a774-bb065e88f378\" (UID: \"f2d66dea-9f0b-43f3-a774-bb065e88f378\") "
Oct 30 01:24:11.243651 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:11.243564    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2d66dea-9f0b-43f3-a774-bb065e88f378-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2d66dea-9f0b-43f3-a774-bb065e88f378-model-cache\") pod \"f2d66dea-9f0b-43f3-a774-bb065e88f378\" (UID: \"f2d66dea-9f0b-43f3-a774-bb065e88f378\") "
Oct 30 01:24:11.243651 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:11.243596    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2d66dea-9f0b-43f3-a774-bb065e88f378-kube-api-access-7dklb\" (UniqueName: \"kubernetes.io/projected/f2d66dea-9f0b-43f3-a774-bb065e88f378-kube-api-access-7dklb\") pod \"f2d66dea-9f0b-43f3-a774-bb065e88f378\" (UID: \"f2d66dea-9f0b-43f3-a774-bb065e88f378\") "
Oct 30 01:24:11.243651 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:11.243631    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32a03478-46eb-4363-97b6-51a32c4099c7-kube-api-access-8t8n9\" (UniqueName: \"kubernetes.io/projected/32a03478-46eb-4363-97b6-51a32c4099c7-kube-api-access-8t8n9\") pod \"32a03478-46eb-4363-97b6-51a32c4099c7\" (UID: \"32a03478-46eb-4363-97b6-51a32c4099c7\") "
Oct 30 01:24:11.244070 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:11.243663    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32a03478-46eb-4363-97b6-51a32c4099c7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32a03478-46eb-4363-97b6-51a32c4099c7-tmp-dir\") pod \"32a03478-46eb-4363-97b6-51a32c4099c7\" (UID: \"32a03478-46eb-4363-97b6-51a32c4099c7\") "
Oct 30 01:24:11.244070 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:11.243753    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32a03478-46eb-4363-97b6-51a32c4099c7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/32a03478-46eb-4363-97b6-51a32c4099c7-dshm\") pod \"32a03478-46eb-4363-97b6-51a32c4099c7\" (UID: \"32a03478-46eb-4363-97b6-51a32c4099c7\") "
Oct 30 01:24:11.244070 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:11.243799    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2d66dea-9f0b-43f3-a774-bb065e88f378-home\" (UniqueName: \"kubernetes.io/empty-dir/f2d66dea-9f0b-43f3-a774-bb065e88f378-home\") pod \"f2d66dea-9f0b-43f3-a774-bb065e88f378\" (UID: \"f2d66dea-9f0b-43f3-a774-bb065e88f378\") "
Oct 30 01:24:11.244070 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:11.243806    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32a03478-46eb-4363-97b6-51a32c4099c7-home" pod "32a03478-46eb-4363-97b6-51a32c4099c7" (UID: "32a03478-46eb-4363-97b6-51a32c4099c7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:24:11.244070 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:11.243846    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32a03478-46eb-4363-97b6-51a32c4099c7-model-cache" pod "32a03478-46eb-4363-97b6-51a32c4099c7" (UID: "32a03478-46eb-4363-97b6-51a32c4099c7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:24:11.244070 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:11.243869    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2d66dea-9f0b-43f3-a774-bb065e88f378-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2d66dea-9f0b-43f3-a774-bb065e88f378-tmp-dir\") pod \"f2d66dea-9f0b-43f3-a774-bb065e88f378\" (UID: \"f2d66dea-9f0b-43f3-a774-bb065e88f378\") "
Oct 30 01:24:11.244438 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:11.244152    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2d66dea-9f0b-43f3-a774-bb065e88f378-model-cache" pod "f2d66dea-9f0b-43f3-a774-bb065e88f378" (UID: "f2d66dea-9f0b-43f3-a774-bb065e88f378"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:24:11.244438 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:11.244220    5002 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32a03478-46eb-4363-97b6-51a32c4099c7-model-cache\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:24:11.244438 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:11.244238    5002 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32a03478-46eb-4363-97b6-51a32c4099c7-home\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:24:11.244438 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:11.244364    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2d66dea-9f0b-43f3-a774-bb065e88f378-home" pod "f2d66dea-9f0b-43f3-a774-bb065e88f378" (UID: "f2d66dea-9f0b-43f3-a774-bb065e88f378"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:24:11.244438 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:11.244425    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2d66dea-9f0b-43f3-a774-bb065e88f378-tmp-dir" pod "f2d66dea-9f0b-43f3-a774-bb065e88f378" (UID: "f2d66dea-9f0b-43f3-a774-bb065e88f378"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:24:11.244734 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:11.244571    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32a03478-46eb-4363-97b6-51a32c4099c7-tmp-dir" pod "32a03478-46eb-4363-97b6-51a32c4099c7" (UID: "32a03478-46eb-4363-97b6-51a32c4099c7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:24:11.246019 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:11.245987    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32a03478-46eb-4363-97b6-51a32c4099c7-tls-certs" pod "32a03478-46eb-4363-97b6-51a32c4099c7" (UID: "32a03478-46eb-4363-97b6-51a32c4099c7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:24:11.246125 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:11.246091    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2d66dea-9f0b-43f3-a774-bb065e88f378-tls-certs" pod "f2d66dea-9f0b-43f3-a774-bb065e88f378" (UID: "f2d66dea-9f0b-43f3-a774-bb065e88f378"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:24:11.246268 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:11.246250    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32a03478-46eb-4363-97b6-51a32c4099c7-dshm" pod "32a03478-46eb-4363-97b6-51a32c4099c7" (UID: "32a03478-46eb-4363-97b6-51a32c4099c7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:24:11.246348 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:11.246307    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2d66dea-9f0b-43f3-a774-bb065e88f378-dshm" pod "f2d66dea-9f0b-43f3-a774-bb065e88f378" (UID: "f2d66dea-9f0b-43f3-a774-bb065e88f378"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:24:11.246348 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:11.246332    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32a03478-46eb-4363-97b6-51a32c4099c7-kube-api-access-8t8n9" pod "32a03478-46eb-4363-97b6-51a32c4099c7" (UID: "32a03478-46eb-4363-97b6-51a32c4099c7"). InnerVolumeSpecName "kube-api-access-8t8n9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:24:11.246466 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:11.246345    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2d66dea-9f0b-43f3-a774-bb065e88f378-kube-api-access-7dklb" pod "f2d66dea-9f0b-43f3-a774-bb065e88f378" (UID: "f2d66dea-9f0b-43f3-a774-bb065e88f378"). InnerVolumeSpecName "kube-api-access-7dklb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:24:11.345635 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:11.345547    5002 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2d66dea-9f0b-43f3-a774-bb065e88f378-model-cache\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:24:11.345635 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:11.345577    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7dklb\" (UniqueName: \"kubernetes.io/projected/f2d66dea-9f0b-43f3-a774-bb065e88f378-kube-api-access-7dklb\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:24:11.345635 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:11.345591    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8t8n9\" (UniqueName: \"kubernetes.io/projected/32a03478-46eb-4363-97b6-51a32c4099c7-kube-api-access-8t8n9\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:24:11.345635 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:11.345603    5002 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32a03478-46eb-4363-97b6-51a32c4099c7-tmp-dir\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:24:11.345635 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:11.345614    5002 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32a03478-46eb-4363-97b6-51a32c4099c7-dshm\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:24:11.345635 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:11.345625    5002 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2d66dea-9f0b-43f3-a774-bb065e88f378-home\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:24:11.345635 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:11.345636    5002 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2d66dea-9f0b-43f3-a774-bb065e88f378-tmp-dir\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:24:11.345997 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:11.345649    5002 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2d66dea-9f0b-43f3-a774-bb065e88f378-tls-certs\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:24:11.345997 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:11.345660    5002 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32a03478-46eb-4363-97b6-51a32c4099c7-tls-certs\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:24:11.345997 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:11.345672    5002 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2d66dea-9f0b-43f3-a774-bb065e88f378-dshm\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:24:11.495267 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:11.495232    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-baf2df27-gbnsl"]
Oct 30 01:24:11.495728 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:11.495709    5002 cpu_manager.go:414] "containerMap: removing container" podUID="32a03478-46eb-4363-97b6-51a32c4099c7" containerName="main"
Oct 30 01:24:11.495853 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:11.495729    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32a03478-46eb-4363-97b6-51a32c4099c7" containerName="main"
Oct 30 01:24:11.495853 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:11.495776    5002 cpu_manager.go:414] "containerMap: removing container" podUID="f2d66dea-9f0b-43f3-a774-bb065e88f378" containerName="main"
Oct 30 01:24:11.495853 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:11.495784    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2d66dea-9f0b-43f3-a774-bb065e88f378" containerName="main"
Oct 30 01:24:11.495853 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:11.495799    5002 cpu_manager.go:414] "containerMap: removing container" podUID="921cb648-96f3-483f-8984-0359a6208ce0" containerName="revoke-keys"
Oct 30 01:24:11.495853 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:11.495808    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="921cb648-96f3-483f-8984-0359a6208ce0" containerName="revoke-keys"
Oct 30 01:24:11.495853 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:11.495825    5002 cpu_manager.go:414] "containerMap: removing container" podUID="4df0b3e0-04ca-440d-9b4c-01635ee412e7" containerName="istio-proxy"
Oct 30 01:24:11.495853 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:11.495832    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4df0b3e0-04ca-440d-9b4c-01635ee412e7" containerName="istio-proxy"
Oct 30 01:24:11.495853 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:11.495846    5002 cpu_manager.go:414] "containerMap: removing container" podUID="8c2a594b-3bbf-4c94-8a9f-7b79c663f135" containerName="istio-proxy"
Oct 30 01:24:11.495853 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:11.495854    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c2a594b-3bbf-4c94-8a9f-7b79c663f135" containerName="istio-proxy"
Oct 30 01:24:11.496326 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:11.495957    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2d66dea-9f0b-43f3-a774-bb065e88f378" containerName="main"
Oct 30 01:24:11.496326 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:11.495970    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="921cb648-96f3-483f-8984-0359a6208ce0" containerName="revoke-keys"
Oct 30 01:24:11.496326 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:11.495983    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="4df0b3e0-04ca-440d-9b4c-01635ee412e7" containerName="istio-proxy"
Oct 30 01:24:11.496326 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:11.495995    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="32a03478-46eb-4363-97b6-51a32c4099c7" containerName="main"
Oct 30 01:24:11.500623 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:11.500597    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-baf2df27-gbnsl"
Oct 30 01:24:11.518665 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:11.518636    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-baf2df27-gbnsl"]
Oct 30 01:24:11.578112 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:11.578067    5002 generic.go:356] "Generic (PLEG): container finished" podID="f2d66dea-9f0b-43f3-a774-bb065e88f378" containerID="b601a0f189cbbee81fcc9087c896e022ebe09ab1a82eee2857b16e53c827c75c" exitCode=0
Oct 30 01:24:11.578326 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:11.578136    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-1a11cf/distinct-2-w2-1a11cf-kserve-758bb6dd57-j5f2k"
Oct 30 01:24:11.578326 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:11.578151    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-1a11cf/distinct-2-w2-1a11cf-kserve-758bb6dd57-j5f2k" event={"ID":"f2d66dea-9f0b-43f3-a774-bb065e88f378","Type":"ContainerDied","Data":"b601a0f189cbbee81fcc9087c896e022ebe09ab1a82eee2857b16e53c827c75c"}
Oct 30 01:24:11.578326 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:11.578194    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-1a11cf/distinct-2-w2-1a11cf-kserve-758bb6dd57-j5f2k" event={"ID":"f2d66dea-9f0b-43f3-a774-bb065e88f378","Type":"ContainerDied","Data":"7cbb4f5bff2ac65c0d1c45834e666fa2ebb5b8499382af1454c7223c42ff432a"}
Oct 30 01:24:11.578326 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:11.578215    5002 scope.go:122] "RemoveContainer" containerID="b601a0f189cbbee81fcc9087c896e022ebe09ab1a82eee2857b16e53c827c75c"
Oct 30 01:24:11.579893 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:11.579873    5002 generic.go:356] "Generic (PLEG): container finished" podID="32a03478-46eb-4363-97b6-51a32c4099c7" containerID="dcb0a41337b62c29232f8b4261c4e6d0adf60e2e9d0dd0073c743932690095a8" exitCode=0
Oct 30 01:24:11.580037 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:11.580013    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-1a11cf/distinct-w2-1a11cf-kserve-854cf4cdf8-cddw6" event={"ID":"32a03478-46eb-4363-97b6-51a32c4099c7","Type":"ContainerDied","Data":"dcb0a41337b62c29232f8b4261c4e6d0adf60e2e9d0dd0073c743932690095a8"}
Oct 30 01:24:11.580294 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:11.580263    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-1a11cf/distinct-w2-1a11cf-kserve-854cf4cdf8-cddw6" event={"ID":"32a03478-46eb-4363-97b6-51a32c4099c7","Type":"ContainerDied","Data":"61371647b3505acee6dfc168db3f0c9fc10ec3cd0c104b317dd56c3c15ab2b1c"}
Oct 30 01:24:11.580294 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:11.580277    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-1a11cf/distinct-w2-1a11cf-kserve-854cf4cdf8-cddw6"
Oct 30 01:24:11.590630 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:11.590605    5002 scope.go:122] "RemoveContainer" containerID="b601a0f189cbbee81fcc9087c896e022ebe09ab1a82eee2857b16e53c827c75c"
Oct 30 01:24:11.591151 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:24:11.591128    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b601a0f189cbbee81fcc9087c896e022ebe09ab1a82eee2857b16e53c827c75c\": container with ID starting with b601a0f189cbbee81fcc9087c896e022ebe09ab1a82eee2857b16e53c827c75c not found: ID does not exist" containerID="b601a0f189cbbee81fcc9087c896e022ebe09ab1a82eee2857b16e53c827c75c"
Oct 30 01:24:11.591270 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:11.591158    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b601a0f189cbbee81fcc9087c896e022ebe09ab1a82eee2857b16e53c827c75c"} err="failed to get container status \"b601a0f189cbbee81fcc9087c896e022ebe09ab1a82eee2857b16e53c827c75c\": rpc error: code = NotFound desc = could not find container \"b601a0f189cbbee81fcc9087c896e022ebe09ab1a82eee2857b16e53c827c75c\": container with ID starting with b601a0f189cbbee81fcc9087c896e022ebe09ab1a82eee2857b16e53c827c75c not found: ID does not exist"
Oct 30 01:24:11.591270 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:11.591176    5002 scope.go:122] "RemoveContainer" containerID="dcb0a41337b62c29232f8b4261c4e6d0adf60e2e9d0dd0073c743932690095a8"
Oct 30 01:24:11.593157 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:11.593137    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:24:11.593263 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:11.593197    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5phsh\""
Oct 30 01:24:11.593830 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:11.593756    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:24:11.593884 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:11.593834    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:24:11.601806 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:11.601787    5002 scope.go:122] "RemoveContainer" containerID="dcb0a41337b62c29232f8b4261c4e6d0adf60e2e9d0dd0073c743932690095a8"
Oct 30 01:24:11.602039 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:24:11.602021    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dcb0a41337b62c29232f8b4261c4e6d0adf60e2e9d0dd0073c743932690095a8\": container with ID starting with dcb0a41337b62c29232f8b4261c4e6d0adf60e2e9d0dd0073c743932690095a8 not found: ID does not exist" containerID="dcb0a41337b62c29232f8b4261c4e6d0adf60e2e9d0dd0073c743932690095a8"
Oct 30 01:24:11.602102 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:11.602045    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dcb0a41337b62c29232f8b4261c4e6d0adf60e2e9d0dd0073c743932690095a8"} err="failed to get container status \"dcb0a41337b62c29232f8b4261c4e6d0adf60e2e9d0dd0073c743932690095a8\": rpc error: code = NotFound desc = could not find container \"dcb0a41337b62c29232f8b4261c4e6d0adf60e2e9d0dd0073c743932690095a8\": container with ID starting with dcb0a41337b62c29232f8b4261c4e6d0adf60e2e9d0dd0073c743932690095a8 not found: ID does not exist"
Oct 30 01:24:11.634248 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:11.634206    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-1a11cf/distinct-2-w2-1a11cf-kserve-758bb6dd57-j5f2k"]
Oct 30 01:24:11.637870 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:11.637842    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-1a11cf/distinct-2-w2-1a11cf-kserve-758bb6dd57-j5f2k"]
Oct 30 01:24:11.652231 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:11.652192    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91d2a0e6-d589-4747-b9f8-2ec905ce7780-maas-api-xks-ca\") pod \"maas-api-revoke-sub-baf2df27-gbnsl\" (UID: \"91d2a0e6-d589-4747-b9f8-2ec905ce7780\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-baf2df27-gbnsl"
Oct 30 01:24:11.652421 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:11.652237    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91d2a0e6-d589-4747-b9f8-2ec905ce7780-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-baf2df27-gbnsl\" (UID: \"91d2a0e6-d589-4747-b9f8-2ec905ce7780\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-baf2df27-gbnsl"
Oct 30 01:24:11.652490 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:11.652429    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqxn9\" (UniqueName: \"kubernetes.io/projected/91d2a0e6-d589-4747-b9f8-2ec905ce7780-kube-api-access-hqxn9\") pod \"maas-api-revoke-sub-baf2df27-gbnsl\" (UID: \"91d2a0e6-d589-4747-b9f8-2ec905ce7780\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-baf2df27-gbnsl"
Oct 30 01:24:11.674948 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:11.674909    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-1a11cf/distinct-w2-1a11cf-kserve-854cf4cdf8-cddw6"]
Oct 30 01:24:11.678658 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:11.678628    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-1a11cf/distinct-w2-1a11cf-kserve-854cf4cdf8-cddw6"]
Oct 30 01:24:11.753141 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:11.753093    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91d2a0e6-d589-4747-b9f8-2ec905ce7780-maas-api-xks-ca\") pod \"maas-api-revoke-sub-baf2df27-gbnsl\" (UID: \"91d2a0e6-d589-4747-b9f8-2ec905ce7780\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-baf2df27-gbnsl"
Oct 30 01:24:11.753141 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:11.753139    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91d2a0e6-d589-4747-b9f8-2ec905ce7780-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-baf2df27-gbnsl\" (UID: \"91d2a0e6-d589-4747-b9f8-2ec905ce7780\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-baf2df27-gbnsl"
Oct 30 01:24:11.753481 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:11.753219    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqxn9\" (UniqueName: \"kubernetes.io/projected/91d2a0e6-d589-4747-b9f8-2ec905ce7780-kube-api-access-hqxn9\") pod \"maas-api-revoke-sub-baf2df27-gbnsl\" (UID: \"91d2a0e6-d589-4747-b9f8-2ec905ce7780\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-baf2df27-gbnsl"
Oct 30 01:24:11.753947 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:11.753921    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91d2a0e6-d589-4747-b9f8-2ec905ce7780-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-baf2df27-gbnsl\" (UID: \"91d2a0e6-d589-4747-b9f8-2ec905ce7780\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-baf2df27-gbnsl"
Oct 30 01:24:11.755572 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:11.755549    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91d2a0e6-d589-4747-b9f8-2ec905ce7780-maas-api-xks-ca\") pod \"maas-api-revoke-sub-baf2df27-gbnsl\" (UID: \"91d2a0e6-d589-4747-b9f8-2ec905ce7780\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-baf2df27-gbnsl"
Oct 30 01:24:11.778438 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:11.778398    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqxn9\" (UniqueName: \"kubernetes.io/projected/91d2a0e6-d589-4747-b9f8-2ec905ce7780-kube-api-access-hqxn9\") pod \"maas-api-revoke-sub-baf2df27-gbnsl\" (UID: \"91d2a0e6-d589-4747-b9f8-2ec905ce7780\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-baf2df27-gbnsl"
Oct 30 01:24:11.818955 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:11.818926    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-baf2df27-gbnsl"
Oct 30 01:24:12.120898 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:12.120842    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-baf2df27-gbnsl"]
Oct 30 01:24:12.121616 ip-10-0-128-7 kubenswrapper[5002]: W1030 01:24:12.121583    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91d2a0e6_d589_4747_b9f8_2ec905ce7780.slice/crio-aa8341697e74cf715b2885d492800807e02e0bbf005dd1c3635d97747ccd377b WatchSource:0}: Error finding container aa8341697e74cf715b2885d492800807e02e0bbf005dd1c3635d97747ccd377b: Status 404 returned error can't find the container with id aa8341697e74cf715b2885d492800807e02e0bbf005dd1c3635d97747ccd377b
Oct 30 01:24:12.492728 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:12.492697    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32a03478-46eb-4363-97b6-51a32c4099c7" path="/var/lib/kubelet/pods/32a03478-46eb-4363-97b6-51a32c4099c7/volumes"
Oct 30 01:24:12.493137 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:12.493125    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2d66dea-9f0b-43f3-a774-bb065e88f378" path="/var/lib/kubelet/pods/f2d66dea-9f0b-43f3-a774-bb065e88f378/volumes"
Oct 30 01:24:12.589427 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:12.589366    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-baf2df27-gbnsl" event={"ID":"91d2a0e6-d589-4747-b9f8-2ec905ce7780","Type":"ContainerStarted","Data":"b5ebba3928ec50f83223f15e9af5acbd34426a1ac0c308e9fc7278c0bda2afa4"}
Oct 30 01:24:12.589639 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:12.589442    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-baf2df27-gbnsl" event={"ID":"91d2a0e6-d589-4747-b9f8-2ec905ce7780","Type":"ContainerStarted","Data":"aa8341697e74cf715b2885d492800807e02e0bbf005dd1c3635d97747ccd377b"}
Oct 30 01:24:12.632541 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:12.632486    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-baf2df27-gbnsl" podStartSLOduration=1.632471664 podStartE2EDuration="1.632471664s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:24:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:24:12.631920565 +0000 UTC m=+3148.248353445" watchObservedRunningTime="2026-10-30 01:24:12.632471664 +0000 UTC m=+3148.248904543"
Oct 30 01:24:12.861042 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:24:12.860959    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbjdq" containerName="extractor"
Oct 30 01:24:13.596683 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:13.596648    5002 generic.go:356] "Generic (PLEG): container finished" podID="91d2a0e6-d589-4747-b9f8-2ec905ce7780" containerID="b5ebba3928ec50f83223f15e9af5acbd34426a1ac0c308e9fc7278c0bda2afa4" exitCode=0
Oct 30 01:24:13.596879 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:13.596728    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-baf2df27-gbnsl" event={"ID":"91d2a0e6-d589-4747-b9f8-2ec905ce7780","Type":"ContainerDied","Data":"b5ebba3928ec50f83223f15e9af5acbd34426a1ac0c308e9fc7278c0bda2afa4"}
Oct 30 01:24:14.746687 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:14.746650    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-baf2df27-gbnsl"
Oct 30 01:24:14.885507 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:14.885423    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91d2a0e6-d589-4747-b9f8-2ec905ce7780-kube-api-access-hqxn9\" (UniqueName: \"kubernetes.io/projected/91d2a0e6-d589-4747-b9f8-2ec905ce7780-kube-api-access-hqxn9\") pod \"91d2a0e6-d589-4747-b9f8-2ec905ce7780\" (UID: \"91d2a0e6-d589-4747-b9f8-2ec905ce7780\") "
Oct 30 01:24:14.885670 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:14.885539    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/91d2a0e6-d589-4747-b9f8-2ec905ce7780-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91d2a0e6-d589-4747-b9f8-2ec905ce7780-maas-api-xks-ca\") pod \"91d2a0e6-d589-4747-b9f8-2ec905ce7780\" (UID: \"91d2a0e6-d589-4747-b9f8-2ec905ce7780\") "
Oct 30 01:24:14.885670 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:14.885563    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91d2a0e6-d589-4747-b9f8-2ec905ce7780-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91d2a0e6-d589-4747-b9f8-2ec905ce7780-maas-api-openshift-ca\") pod \"91d2a0e6-d589-4747-b9f8-2ec905ce7780\" (UID: \"91d2a0e6-d589-4747-b9f8-2ec905ce7780\") "
Oct 30 01:24:14.885976 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:14.885947    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91d2a0e6-d589-4747-b9f8-2ec905ce7780-maas-api-openshift-ca" pod "91d2a0e6-d589-4747-b9f8-2ec905ce7780" (UID: "91d2a0e6-d589-4747-b9f8-2ec905ce7780"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:24:14.887607 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:14.887583    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91d2a0e6-d589-4747-b9f8-2ec905ce7780-kube-api-access-hqxn9" pod "91d2a0e6-d589-4747-b9f8-2ec905ce7780" (UID: "91d2a0e6-d589-4747-b9f8-2ec905ce7780"). InnerVolumeSpecName "kube-api-access-hqxn9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:24:14.887713 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:14.887627    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91d2a0e6-d589-4747-b9f8-2ec905ce7780-maas-api-xks-ca" pod "91d2a0e6-d589-4747-b9f8-2ec905ce7780" (UID: "91d2a0e6-d589-4747-b9f8-2ec905ce7780"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:24:14.987355 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:14.987324    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91d2a0e6-d589-4747-b9f8-2ec905ce7780-maas-api-xks-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:24:14.987355 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:14.987352    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91d2a0e6-d589-4747-b9f8-2ec905ce7780-maas-api-openshift-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:24:14.987591 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:14.987363    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hqxn9\" (UniqueName: \"kubernetes.io/projected/91d2a0e6-d589-4747-b9f8-2ec905ce7780-kube-api-access-hqxn9\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:24:15.608473 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:15.608442    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-baf2df27-gbnsl"
Oct 30 01:24:15.608649 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:15.608470    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-baf2df27-gbnsl" event={"ID":"91d2a0e6-d589-4747-b9f8-2ec905ce7780","Type":"ContainerDied","Data":"aa8341697e74cf715b2885d492800807e02e0bbf005dd1c3635d97747ccd377b"}
Oct 30 01:24:15.608649 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:15.608526    5002 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aa8341697e74cf715b2885d492800807e02e0bbf005dd1c3635d97747ccd377b"
Oct 30 01:24:16.306938 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:16.306888    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7d8z4"]
Oct 30 01:24:18.767173 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:18.767128    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-021d6d/simulator-w6-021d6d-kserve-6cc97fdf86-dqtpw" podUID="50586f76-4f3b-4e7a-83ac-6c573faee415" containerName="main" containerID="cri-o://56454254bb774ad52b1f7fc34bf1f2e5fe8215deb7a28707d8a280186ca001d0" gracePeriod=45
Oct 30 01:24:19.035411 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:19.035371    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-021d6d/simulator-w6-021d6d-kserve-6cc97fdf86-dqtpw"
Oct 30 01:24:19.126174 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:19.126141    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50586f76-4f3b-4e7a-83ac-6c573faee415-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50586f76-4f3b-4e7a-83ac-6c573faee415-tmp-dir\") pod \"50586f76-4f3b-4e7a-83ac-6c573faee415\" (UID: \"50586f76-4f3b-4e7a-83ac-6c573faee415\") "
Oct 30 01:24:19.126364 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:19.126197    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50586f76-4f3b-4e7a-83ac-6c573faee415-home\" (UniqueName: \"kubernetes.io/empty-dir/50586f76-4f3b-4e7a-83ac-6c573faee415-home\") pod \"50586f76-4f3b-4e7a-83ac-6c573faee415\" (UID: \"50586f76-4f3b-4e7a-83ac-6c573faee415\") "
Oct 30 01:24:19.126364 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:19.126215    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/50586f76-4f3b-4e7a-83ac-6c573faee415-tls-certs\" (UniqueName: \"kubernetes.io/secret/50586f76-4f3b-4e7a-83ac-6c573faee415-tls-certs\") pod \"50586f76-4f3b-4e7a-83ac-6c573faee415\" (UID: \"50586f76-4f3b-4e7a-83ac-6c573faee415\") "
Oct 30 01:24:19.126364 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:19.126234    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50586f76-4f3b-4e7a-83ac-6c573faee415-kube-api-access-fslcn\" (UniqueName: \"kubernetes.io/projected/50586f76-4f3b-4e7a-83ac-6c573faee415-kube-api-access-fslcn\") pod \"50586f76-4f3b-4e7a-83ac-6c573faee415\" (UID: \"50586f76-4f3b-4e7a-83ac-6c573faee415\") "
Oct 30 01:24:19.126364 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:19.126268    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50586f76-4f3b-4e7a-83ac-6c573faee415-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50586f76-4f3b-4e7a-83ac-6c573faee415-model-cache\") pod \"50586f76-4f3b-4e7a-83ac-6c573faee415\" (UID: \"50586f76-4f3b-4e7a-83ac-6c573faee415\") "
Oct 30 01:24:19.126364 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:19.126298    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50586f76-4f3b-4e7a-83ac-6c573faee415-dshm\" (UniqueName: \"kubernetes.io/empty-dir/50586f76-4f3b-4e7a-83ac-6c573faee415-dshm\") pod \"50586f76-4f3b-4e7a-83ac-6c573faee415\" (UID: \"50586f76-4f3b-4e7a-83ac-6c573faee415\") "
Oct 30 01:24:19.126705 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:19.126667    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50586f76-4f3b-4e7a-83ac-6c573faee415-home" pod "50586f76-4f3b-4e7a-83ac-6c573faee415" (UID: "50586f76-4f3b-4e7a-83ac-6c573faee415"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:24:19.126860 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:19.126837    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50586f76-4f3b-4e7a-83ac-6c573faee415-tmp-dir" pod "50586f76-4f3b-4e7a-83ac-6c573faee415" (UID: "50586f76-4f3b-4e7a-83ac-6c573faee415"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:24:19.126919 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:19.126859    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50586f76-4f3b-4e7a-83ac-6c573faee415-model-cache" pod "50586f76-4f3b-4e7a-83ac-6c573faee415" (UID: "50586f76-4f3b-4e7a-83ac-6c573faee415"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:24:19.127082 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:19.127060    5002 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50586f76-4f3b-4e7a-83ac-6c573faee415-tmp-dir\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:24:19.127167 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:19.127088    5002 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/50586f76-4f3b-4e7a-83ac-6c573faee415-home\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:24:19.127167 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:19.127102    5002 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50586f76-4f3b-4e7a-83ac-6c573faee415-model-cache\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:24:19.128803 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:19.128771    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50586f76-4f3b-4e7a-83ac-6c573faee415-tls-certs" pod "50586f76-4f3b-4e7a-83ac-6c573faee415" (UID: "50586f76-4f3b-4e7a-83ac-6c573faee415"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:24:19.128921 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:19.128891    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50586f76-4f3b-4e7a-83ac-6c573faee415-kube-api-access-fslcn" pod "50586f76-4f3b-4e7a-83ac-6c573faee415" (UID: "50586f76-4f3b-4e7a-83ac-6c573faee415"). InnerVolumeSpecName "kube-api-access-fslcn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:24:19.130393 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:19.130362    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50586f76-4f3b-4e7a-83ac-6c573faee415-dshm" pod "50586f76-4f3b-4e7a-83ac-6c573faee415" (UID: "50586f76-4f3b-4e7a-83ac-6c573faee415"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:24:19.228542 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:19.228499    5002 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50586f76-4f3b-4e7a-83ac-6c573faee415-tls-certs\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:24:19.228542 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:19.228533    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fslcn\" (UniqueName: \"kubernetes.io/projected/50586f76-4f3b-4e7a-83ac-6c573faee415-kube-api-access-fslcn\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:24:19.228542 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:19.228544    5002 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/50586f76-4f3b-4e7a-83ac-6c573faee415-dshm\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:24:19.630778 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:19.630745    5002 generic.go:356] "Generic (PLEG): container finished" podID="50586f76-4f3b-4e7a-83ac-6c573faee415" containerID="56454254bb774ad52b1f7fc34bf1f2e5fe8215deb7a28707d8a280186ca001d0" exitCode=0
Oct 30 01:24:19.630944 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:19.630792    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-021d6d/simulator-w6-021d6d-kserve-6cc97fdf86-dqtpw"
Oct 30 01:24:19.630944 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:19.630834    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-021d6d/simulator-w6-021d6d-kserve-6cc97fdf86-dqtpw" event={"ID":"50586f76-4f3b-4e7a-83ac-6c573faee415","Type":"ContainerDied","Data":"56454254bb774ad52b1f7fc34bf1f2e5fe8215deb7a28707d8a280186ca001d0"}
Oct 30 01:24:19.630944 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:19.630879    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-021d6d/simulator-w6-021d6d-kserve-6cc97fdf86-dqtpw" event={"ID":"50586f76-4f3b-4e7a-83ac-6c573faee415","Type":"ContainerDied","Data":"0b328238f902a468aafb5896fea00bc84ca0dbf2641606f30109008020ffe2ab"}
Oct 30 01:24:19.630944 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:19.630903    5002 scope.go:122] "RemoveContainer" containerID="56454254bb774ad52b1f7fc34bf1f2e5fe8215deb7a28707d8a280186ca001d0"
Oct 30 01:24:19.640363 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:19.640343    5002 scope.go:122] "RemoveContainer" containerID="56454254bb774ad52b1f7fc34bf1f2e5fe8215deb7a28707d8a280186ca001d0"
Oct 30 01:24:19.640749 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:24:19.640731    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56454254bb774ad52b1f7fc34bf1f2e5fe8215deb7a28707d8a280186ca001d0\": container with ID starting with 56454254bb774ad52b1f7fc34bf1f2e5fe8215deb7a28707d8a280186ca001d0 not found: ID does not exist" containerID="56454254bb774ad52b1f7fc34bf1f2e5fe8215deb7a28707d8a280186ca001d0"
Oct 30 01:24:19.640797 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:19.640758    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56454254bb774ad52b1f7fc34bf1f2e5fe8215deb7a28707d8a280186ca001d0"} err="failed to get container status \"56454254bb774ad52b1f7fc34bf1f2e5fe8215deb7a28707d8a280186ca001d0\": rpc error: code = NotFound desc = could not find container \"56454254bb774ad52b1f7fc34bf1f2e5fe8215deb7a28707d8a280186ca001d0\": container with ID starting with 56454254bb774ad52b1f7fc34bf1f2e5fe8215deb7a28707d8a280186ca001d0 not found: ID does not exist"
Oct 30 01:24:19.688462 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:19.688425    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-021d6d/simulator-w6-021d6d-kserve-6cc97fdf86-dqtpw"]
Oct 30 01:24:19.701360 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:19.701325    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-021d6d/simulator-w6-021d6d-kserve-6cc97fdf86-dqtpw"]
Oct 30 01:24:20.494872 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:20.494841    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50586f76-4f3b-4e7a-83ac-6c573faee415" path="/var/lib/kubelet/pods/50586f76-4f3b-4e7a-83ac-6c573faee415/volumes"
Oct 30 01:24:23.877862 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:23.877822    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-947lv"]
Oct 30 01:24:23.878364 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:23.878063    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-947lv" podUID="ee192122-5e45-439e-93b4-eb2a0bf03260" containerName="istio-proxy" containerID="cri-o://7730166cee01474ea4a323a0a9ced68c13e88ead18eb88f6d8ebc207fc3c3ff1" gracePeriod=30
Oct 30 01:24:26.521709 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:26.521667    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7d8z4"]
Oct 30 01:24:27.673464 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:27.673424    5002 patch_prober.go:29] interesting pod/e2e-worker-w2-1a11cf-openshift-default-748ddd859-947lv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 01:24:27.673876 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:27.673492    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-947lv" podUID="ee192122-5e45-439e-93b4-eb2a0bf03260" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 01:24:29.138901 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:29.138874    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-947lv"
Oct 30 01:24:29.221053 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:29.221019    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee192122-5e45-439e-93b4-eb2a0bf03260-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee192122-5e45-439e-93b4-eb2a0bf03260-istio-envoy\") pod \"ee192122-5e45-439e-93b4-eb2a0bf03260\" (UID: \"ee192122-5e45-439e-93b4-eb2a0bf03260\") "
Oct 30 01:24:29.221259 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:29.221082    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee192122-5e45-439e-93b4-eb2a0bf03260-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee192122-5e45-439e-93b4-eb2a0bf03260-workload-certs\") pod \"ee192122-5e45-439e-93b4-eb2a0bf03260\" (UID: \"ee192122-5e45-439e-93b4-eb2a0bf03260\") "
Oct 30 01:24:29.221259 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:29.221115    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee192122-5e45-439e-93b4-eb2a0bf03260-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee192122-5e45-439e-93b4-eb2a0bf03260-istio-data\") pod \"ee192122-5e45-439e-93b4-eb2a0bf03260\" (UID: \"ee192122-5e45-439e-93b4-eb2a0bf03260\") "
Oct 30 01:24:29.221259 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:29.221144    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee192122-5e45-439e-93b4-eb2a0bf03260-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee192122-5e45-439e-93b4-eb2a0bf03260-workload-socket\") pod \"ee192122-5e45-439e-93b4-eb2a0bf03260\" (UID: \"ee192122-5e45-439e-93b4-eb2a0bf03260\") "
Oct 30 01:24:29.221259 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:29.221178    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee192122-5e45-439e-93b4-eb2a0bf03260-kube-api-access-htz6l\" (UniqueName: \"kubernetes.io/projected/ee192122-5e45-439e-93b4-eb2a0bf03260-kube-api-access-htz6l\") pod \"ee192122-5e45-439e-93b4-eb2a0bf03260\" (UID: \"ee192122-5e45-439e-93b4-eb2a0bf03260\") "
Oct 30 01:24:29.221259 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:29.221205    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee192122-5e45-439e-93b4-eb2a0bf03260-istio-token\" (UniqueName: \"kubernetes.io/projected/ee192122-5e45-439e-93b4-eb2a0bf03260-istio-token\") pod \"ee192122-5e45-439e-93b4-eb2a0bf03260\" (UID: \"ee192122-5e45-439e-93b4-eb2a0bf03260\") "
Oct 30 01:24:29.221259 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:29.221233    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ee192122-5e45-439e-93b4-eb2a0bf03260-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee192122-5e45-439e-93b4-eb2a0bf03260-istio-podinfo\") pod \"ee192122-5e45-439e-93b4-eb2a0bf03260\" (UID: \"ee192122-5e45-439e-93b4-eb2a0bf03260\") "
Oct 30 01:24:29.221675 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:29.221261    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee192122-5e45-439e-93b4-eb2a0bf03260-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee192122-5e45-439e-93b4-eb2a0bf03260-istiod-ca-cert\") pod \"ee192122-5e45-439e-93b4-eb2a0bf03260\" (UID: \"ee192122-5e45-439e-93b4-eb2a0bf03260\") "
Oct 30 01:24:29.221675 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:29.221419    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee192122-5e45-439e-93b4-eb2a0bf03260-workload-certs" pod "ee192122-5e45-439e-93b4-eb2a0bf03260" (UID: "ee192122-5e45-439e-93b4-eb2a0bf03260"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:24:29.221675 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:29.221506    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee192122-5e45-439e-93b4-eb2a0bf03260-workload-socket" pod "ee192122-5e45-439e-93b4-eb2a0bf03260" (UID: "ee192122-5e45-439e-93b4-eb2a0bf03260"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:24:29.221675 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:29.221519    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee192122-5e45-439e-93b4-eb2a0bf03260-istio-data" pod "ee192122-5e45-439e-93b4-eb2a0bf03260" (UID: "ee192122-5e45-439e-93b4-eb2a0bf03260"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:24:29.221675 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:29.221670    5002 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee192122-5e45-439e-93b4-eb2a0bf03260-workload-certs\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:24:29.221908 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:29.221690    5002 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee192122-5e45-439e-93b4-eb2a0bf03260-istio-data\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:24:29.221908 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:29.221705    5002 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee192122-5e45-439e-93b4-eb2a0bf03260-workload-socket\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:24:29.221908 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:29.221793    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee192122-5e45-439e-93b4-eb2a0bf03260-istiod-ca-cert" pod "ee192122-5e45-439e-93b4-eb2a0bf03260" (UID: "ee192122-5e45-439e-93b4-eb2a0bf03260"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:24:29.223570 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:29.223542    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee192122-5e45-439e-93b4-eb2a0bf03260-istio-envoy" pod "ee192122-5e45-439e-93b4-eb2a0bf03260" (UID: "ee192122-5e45-439e-93b4-eb2a0bf03260"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:24:29.224029 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:29.223997    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ee192122-5e45-439e-93b4-eb2a0bf03260-istio-podinfo" pod "ee192122-5e45-439e-93b4-eb2a0bf03260" (UID: "ee192122-5e45-439e-93b4-eb2a0bf03260"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:24:29.224029 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:29.224015    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee192122-5e45-439e-93b4-eb2a0bf03260-istio-token" pod "ee192122-5e45-439e-93b4-eb2a0bf03260" (UID: "ee192122-5e45-439e-93b4-eb2a0bf03260"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:24:29.224159 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:29.224082    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee192122-5e45-439e-93b4-eb2a0bf03260-kube-api-access-htz6l" pod "ee192122-5e45-439e-93b4-eb2a0bf03260" (UID: "ee192122-5e45-439e-93b4-eb2a0bf03260"). InnerVolumeSpecName "kube-api-access-htz6l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:24:29.322246 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:29.322209    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee192122-5e45-439e-93b4-eb2a0bf03260-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee192122-5e45-439e-93b4-eb2a0bf03260-credential-socket\") pod \"ee192122-5e45-439e-93b4-eb2a0bf03260\" (UID: \"ee192122-5e45-439e-93b4-eb2a0bf03260\") "
Oct 30 01:24:29.322453 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:29.322440    5002 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee192122-5e45-439e-93b4-eb2a0bf03260-istio-envoy\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:24:29.322498 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:29.322456    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-htz6l\" (UniqueName: \"kubernetes.io/projected/ee192122-5e45-439e-93b4-eb2a0bf03260-kube-api-access-htz6l\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:24:29.322498 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:29.322465    5002 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee192122-5e45-439e-93b4-eb2a0bf03260-istio-token\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:24:29.322498 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:29.322474    5002 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee192122-5e45-439e-93b4-eb2a0bf03260-istio-podinfo\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:24:29.322498 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:29.322481    5002 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee192122-5e45-439e-93b4-eb2a0bf03260-istiod-ca-cert\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:24:29.322644 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:29.322491    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee192122-5e45-439e-93b4-eb2a0bf03260-credential-socket" pod "ee192122-5e45-439e-93b4-eb2a0bf03260" (UID: "ee192122-5e45-439e-93b4-eb2a0bf03260"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:24:29.423269 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:29.423225    5002 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee192122-5e45-439e-93b4-eb2a0bf03260-credential-socket\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:24:29.696065 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:29.696029    5002 generic.go:356] "Generic (PLEG): container finished" podID="ee192122-5e45-439e-93b4-eb2a0bf03260" containerID="7730166cee01474ea4a323a0a9ced68c13e88ead18eb88f6d8ebc207fc3c3ff1" exitCode=0
Oct 30 01:24:29.696272 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:29.696083    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-947lv"
Oct 30 01:24:29.696272 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:29.696113    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-947lv" event={"ID":"ee192122-5e45-439e-93b4-eb2a0bf03260","Type":"ContainerDied","Data":"7730166cee01474ea4a323a0a9ced68c13e88ead18eb88f6d8ebc207fc3c3ff1"}
Oct 30 01:24:29.696272 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:29.696168    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-947lv" event={"ID":"ee192122-5e45-439e-93b4-eb2a0bf03260","Type":"ContainerDied","Data":"f1783e53b99a5291479a6c8dc84912812c677c7a3fab9b3c3ac62f86a43a5036"}
Oct 30 01:24:29.696272 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:29.696190    5002 scope.go:122] "RemoveContainer" containerID="7730166cee01474ea4a323a0a9ced68c13e88ead18eb88f6d8ebc207fc3c3ff1"
Oct 30 01:24:29.706635 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:29.706612    5002 scope.go:122] "RemoveContainer" containerID="7730166cee01474ea4a323a0a9ced68c13e88ead18eb88f6d8ebc207fc3c3ff1"
Oct 30 01:24:29.706913 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:24:29.706891    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7730166cee01474ea4a323a0a9ced68c13e88ead18eb88f6d8ebc207fc3c3ff1\": container with ID starting with 7730166cee01474ea4a323a0a9ced68c13e88ead18eb88f6d8ebc207fc3c3ff1 not found: ID does not exist" containerID="7730166cee01474ea4a323a0a9ced68c13e88ead18eb88f6d8ebc207fc3c3ff1"
Oct 30 01:24:29.706964 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:29.706923    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7730166cee01474ea4a323a0a9ced68c13e88ead18eb88f6d8ebc207fc3c3ff1"} err="failed to get container status \"7730166cee01474ea4a323a0a9ced68c13e88ead18eb88f6d8ebc207fc3c3ff1\": rpc error: code = NotFound desc = could not find container \"7730166cee01474ea4a323a0a9ced68c13e88ead18eb88f6d8ebc207fc3c3ff1\": container with ID starting with 7730166cee01474ea4a323a0a9ced68c13e88ead18eb88f6d8ebc207fc3c3ff1 not found: ID does not exist"
Oct 30 01:24:29.752190 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:29.752157    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-947lv"]
Oct 30 01:24:29.766106 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:29.766078    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-947lv"]
Oct 30 01:24:30.493527 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:30.493491    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee192122-5e45-439e-93b4-eb2a0bf03260" path="/var/lib/kubelet/pods/ee192122-5e45-439e-93b4-eb2a0bf03260/volumes"
Oct 30 01:24:32.103165 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:32.103125    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-lrlct"]
Oct 30 01:24:32.103649 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:32.103401    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-lrlct" podUID="099508e4-19ee-4e9b-93f3-0ef7a0d5b8b7" containerName="istio-proxy" containerID="cri-o://db3fd84c86046aa65d14b2238cf927ef241c031ac33aef5a027e6c3afbafed7b" gracePeriod=30
Oct 30 01:24:37.364871 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:37.364836    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-lrlct"
Oct 30 01:24:37.387480 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:37.387366    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/099508e4-19ee-4e9b-93f3-0ef7a0d5b8b7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/099508e4-19ee-4e9b-93f3-0ef7a0d5b8b7-workload-socket\") pod \"099508e4-19ee-4e9b-93f3-0ef7a0d5b8b7\" (UID: \"099508e4-19ee-4e9b-93f3-0ef7a0d5b8b7\") "
Oct 30 01:24:37.387480 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:37.387436    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/099508e4-19ee-4e9b-93f3-0ef7a0d5b8b7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/099508e4-19ee-4e9b-93f3-0ef7a0d5b8b7-istio-data\") pod \"099508e4-19ee-4e9b-93f3-0ef7a0d5b8b7\" (UID: \"099508e4-19ee-4e9b-93f3-0ef7a0d5b8b7\") "
Oct 30 01:24:37.387741 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:37.387508    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/099508e4-19ee-4e9b-93f3-0ef7a0d5b8b7-kube-api-access-pvgzp\" (UniqueName: \"kubernetes.io/projected/099508e4-19ee-4e9b-93f3-0ef7a0d5b8b7-kube-api-access-pvgzp\") pod \"099508e4-19ee-4e9b-93f3-0ef7a0d5b8b7\" (UID: \"099508e4-19ee-4e9b-93f3-0ef7a0d5b8b7\") "
Oct 30 01:24:37.387741 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:37.387538    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/099508e4-19ee-4e9b-93f3-0ef7a0d5b8b7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/099508e4-19ee-4e9b-93f3-0ef7a0d5b8b7-istio-podinfo\") pod \"099508e4-19ee-4e9b-93f3-0ef7a0d5b8b7\" (UID: \"099508e4-19ee-4e9b-93f3-0ef7a0d5b8b7\") "
Oct 30 01:24:37.387741 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:37.387570    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/099508e4-19ee-4e9b-93f3-0ef7a0d5b8b7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/099508e4-19ee-4e9b-93f3-0ef7a0d5b8b7-istio-envoy\") pod \"099508e4-19ee-4e9b-93f3-0ef7a0d5b8b7\" (UID: \"099508e4-19ee-4e9b-93f3-0ef7a0d5b8b7\") "
Oct 30 01:24:37.387741 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:37.387633    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/099508e4-19ee-4e9b-93f3-0ef7a0d5b8b7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/099508e4-19ee-4e9b-93f3-0ef7a0d5b8b7-istiod-ca-cert\") pod \"099508e4-19ee-4e9b-93f3-0ef7a0d5b8b7\" (UID: \"099508e4-19ee-4e9b-93f3-0ef7a0d5b8b7\") "
Oct 30 01:24:37.387741 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:37.387671    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/099508e4-19ee-4e9b-93f3-0ef7a0d5b8b7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/099508e4-19ee-4e9b-93f3-0ef7a0d5b8b7-credential-socket\") pod \"099508e4-19ee-4e9b-93f3-0ef7a0d5b8b7\" (UID: \"099508e4-19ee-4e9b-93f3-0ef7a0d5b8b7\") "
Oct 30 01:24:37.387741 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:37.387704    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/099508e4-19ee-4e9b-93f3-0ef7a0d5b8b7-workload-socket" pod "099508e4-19ee-4e9b-93f3-0ef7a0d5b8b7" (UID: "099508e4-19ee-4e9b-93f3-0ef7a0d5b8b7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:24:37.387741 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:37.387715    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/099508e4-19ee-4e9b-93f3-0ef7a0d5b8b7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/099508e4-19ee-4e9b-93f3-0ef7a0d5b8b7-workload-certs\") pod \"099508e4-19ee-4e9b-93f3-0ef7a0d5b8b7\" (UID: \"099508e4-19ee-4e9b-93f3-0ef7a0d5b8b7\") "
Oct 30 01:24:37.388174 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:37.387759    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/099508e4-19ee-4e9b-93f3-0ef7a0d5b8b7-istio-token\" (UniqueName: \"kubernetes.io/projected/099508e4-19ee-4e9b-93f3-0ef7a0d5b8b7-istio-token\") pod \"099508e4-19ee-4e9b-93f3-0ef7a0d5b8b7\" (UID: \"099508e4-19ee-4e9b-93f3-0ef7a0d5b8b7\") "
Oct 30 01:24:37.388174 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:37.387803    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/099508e4-19ee-4e9b-93f3-0ef7a0d5b8b7-istio-data" pod "099508e4-19ee-4e9b-93f3-0ef7a0d5b8b7" (UID: "099508e4-19ee-4e9b-93f3-0ef7a0d5b8b7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:24:37.388174 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:37.388002    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/099508e4-19ee-4e9b-93f3-0ef7a0d5b8b7-credential-socket" pod "099508e4-19ee-4e9b-93f3-0ef7a0d5b8b7" (UID: "099508e4-19ee-4e9b-93f3-0ef7a0d5b8b7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:24:37.388174 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:37.388124    5002 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/099508e4-19ee-4e9b-93f3-0ef7a0d5b8b7-istio-data\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:24:37.388174 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:37.388144    5002 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/099508e4-19ee-4e9b-93f3-0ef7a0d5b8b7-credential-socket\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:24:37.388174 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:37.388161    5002 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/099508e4-19ee-4e9b-93f3-0ef7a0d5b8b7-workload-socket\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:24:37.388620 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:37.388255    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/099508e4-19ee-4e9b-93f3-0ef7a0d5b8b7-workload-certs" pod "099508e4-19ee-4e9b-93f3-0ef7a0d5b8b7" (UID: "099508e4-19ee-4e9b-93f3-0ef7a0d5b8b7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:24:37.388805 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:37.388779    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/099508e4-19ee-4e9b-93f3-0ef7a0d5b8b7-istiod-ca-cert" pod "099508e4-19ee-4e9b-93f3-0ef7a0d5b8b7" (UID: "099508e4-19ee-4e9b-93f3-0ef7a0d5b8b7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:24:37.390291 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:37.390250    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/099508e4-19ee-4e9b-93f3-0ef7a0d5b8b7-kube-api-access-pvgzp" pod "099508e4-19ee-4e9b-93f3-0ef7a0d5b8b7" (UID: "099508e4-19ee-4e9b-93f3-0ef7a0d5b8b7"). InnerVolumeSpecName "kube-api-access-pvgzp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:24:37.391115 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:37.391076    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/099508e4-19ee-4e9b-93f3-0ef7a0d5b8b7-istio-envoy" pod "099508e4-19ee-4e9b-93f3-0ef7a0d5b8b7" (UID: "099508e4-19ee-4e9b-93f3-0ef7a0d5b8b7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:24:37.391577 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:37.391555    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/099508e4-19ee-4e9b-93f3-0ef7a0d5b8b7-istio-podinfo" pod "099508e4-19ee-4e9b-93f3-0ef7a0d5b8b7" (UID: "099508e4-19ee-4e9b-93f3-0ef7a0d5b8b7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:24:37.394817 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:37.394708    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/099508e4-19ee-4e9b-93f3-0ef7a0d5b8b7-istio-token" pod "099508e4-19ee-4e9b-93f3-0ef7a0d5b8b7" (UID: "099508e4-19ee-4e9b-93f3-0ef7a0d5b8b7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:24:37.489006 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:37.488960    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pvgzp\" (UniqueName: \"kubernetes.io/projected/099508e4-19ee-4e9b-93f3-0ef7a0d5b8b7-kube-api-access-pvgzp\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:24:37.489006 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:37.489003    5002 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/099508e4-19ee-4e9b-93f3-0ef7a0d5b8b7-istio-podinfo\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:24:37.489222 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:37.489020    5002 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/099508e4-19ee-4e9b-93f3-0ef7a0d5b8b7-istio-envoy\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:24:37.489222 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:37.489034    5002 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/099508e4-19ee-4e9b-93f3-0ef7a0d5b8b7-istiod-ca-cert\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:24:37.489222 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:37.489050    5002 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/099508e4-19ee-4e9b-93f3-0ef7a0d5b8b7-workload-certs\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:24:37.489222 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:37.489064    5002 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/099508e4-19ee-4e9b-93f3-0ef7a0d5b8b7-istio-token\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:24:37.743327 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:37.743291    5002 generic.go:356] "Generic (PLEG): container finished" podID="099508e4-19ee-4e9b-93f3-0ef7a0d5b8b7" containerID="db3fd84c86046aa65d14b2238cf927ef241c031ac33aef5a027e6c3afbafed7b" exitCode=0
Oct 30 01:24:37.743551 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:37.743349    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-lrlct"
Oct 30 01:24:37.743551 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:37.743365    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-lrlct" event={"ID":"099508e4-19ee-4e9b-93f3-0ef7a0d5b8b7","Type":"ContainerDied","Data":"db3fd84c86046aa65d14b2238cf927ef241c031ac33aef5a027e6c3afbafed7b"}
Oct 30 01:24:37.743551 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:37.743422    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-lrlct" event={"ID":"099508e4-19ee-4e9b-93f3-0ef7a0d5b8b7","Type":"ContainerDied","Data":"d46b75111b0a4ae94b0aa66ed1efc9a703e2543f928bd353e9a3a3987dcdb5a4"}
Oct 30 01:24:37.743551 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:37.743444    5002 scope.go:122] "RemoveContainer" containerID="db3fd84c86046aa65d14b2238cf927ef241c031ac33aef5a027e6c3afbafed7b"
Oct 30 01:24:37.757536 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:37.757508    5002 scope.go:122] "RemoveContainer" containerID="db3fd84c86046aa65d14b2238cf927ef241c031ac33aef5a027e6c3afbafed7b"
Oct 30 01:24:37.758142 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:24:37.758071    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db3fd84c86046aa65d14b2238cf927ef241c031ac33aef5a027e6c3afbafed7b\": container with ID starting with db3fd84c86046aa65d14b2238cf927ef241c031ac33aef5a027e6c3afbafed7b not found: ID does not exist" containerID="db3fd84c86046aa65d14b2238cf927ef241c031ac33aef5a027e6c3afbafed7b"
Oct 30 01:24:37.758142 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:37.758110    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db3fd84c86046aa65d14b2238cf927ef241c031ac33aef5a027e6c3afbafed7b"} err="failed to get container status \"db3fd84c86046aa65d14b2238cf927ef241c031ac33aef5a027e6c3afbafed7b\": rpc error: code = NotFound desc = could not find container \"db3fd84c86046aa65d14b2238cf927ef241c031ac33aef5a027e6c3afbafed7b\": container with ID starting with db3fd84c86046aa65d14b2238cf927ef241c031ac33aef5a027e6c3afbafed7b not found: ID does not exist"
Oct 30 01:24:37.816585 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:37.816544    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-lrlct"]
Oct 30 01:24:37.819464 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:37.819435    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-lrlct"]
Oct 30 01:24:38.494490 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:38.494446    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="099508e4-19ee-4e9b-93f3-0ef7a0d5b8b7" path="/var/lib/kubelet/pods/099508e4-19ee-4e9b-93f3-0ef7a0d5b8b7/volumes"
Oct 30 01:24:42.860701 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:24:42.860663    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbjdq" containerName="extractor"
Oct 30 01:24:49.863819 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:49.863782    5002 scope.go:122] "RemoveContainer" containerID="2c800e74243482d0afbd6cf7d107f8938f43eaa6c42b74c53266e03caf681283"
Oct 30 01:24:49.872788 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:49.872765    5002 scope.go:122] "RemoveContainer" containerID="54c4b08cad4414856b2f1e1a87036e6d7d9e9eeec497095fdb218bea687be400"
Oct 30 01:24:49.881029 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:49.881002    5002 scope.go:122] "RemoveContainer" containerID="e83e8ac28d10eabb02c0dd7a4a6b9613ab7c0901e5f1eeee42305112d8d87fe5"
Oct 30 01:24:49.888965 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:24:49.888942    5002 scope.go:122] "RemoveContainer" containerID="01f558c05d63c0d1920299adfb232c13fece79e41d37df9ec69c162cb77d21a1"
Oct 30 01:25:12.860059 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:25:12.859977    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbjdq" containerName="extractor"
Oct 30 01:25:20.001091 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:25:20.001042    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-2jzrh"]
Oct 30 01:25:20.001611 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:25:20.001488    5002 cpu_manager.go:414] "containerMap: removing container" podUID="ee192122-5e45-439e-93b4-eb2a0bf03260" containerName="istio-proxy"
Oct 30 01:25:20.001611 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:25:20.001502    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee192122-5e45-439e-93b4-eb2a0bf03260" containerName="istio-proxy"
Oct 30 01:25:20.001611 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:25:20.001519    5002 cpu_manager.go:414] "containerMap: removing container" podUID="50586f76-4f3b-4e7a-83ac-6c573faee415" containerName="main"
Oct 30 01:25:20.001611 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:25:20.001525    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50586f76-4f3b-4e7a-83ac-6c573faee415" containerName="main"
Oct 30 01:25:20.001611 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:25:20.001531    5002 cpu_manager.go:414] "containerMap: removing container" podUID="91d2a0e6-d589-4747-b9f8-2ec905ce7780" containerName="revoke-keys"
Oct 30 01:25:20.001611 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:25:20.001537    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91d2a0e6-d589-4747-b9f8-2ec905ce7780" containerName="revoke-keys"
Oct 30 01:25:20.001611 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:25:20.001552    5002 cpu_manager.go:414] "containerMap: removing container" podUID="099508e4-19ee-4e9b-93f3-0ef7a0d5b8b7" containerName="istio-proxy"
Oct 30 01:25:20.001611 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:25:20.001556    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="099508e4-19ee-4e9b-93f3-0ef7a0d5b8b7" containerName="istio-proxy"
Oct 30 01:25:20.001881 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:25:20.001622    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee192122-5e45-439e-93b4-eb2a0bf03260" containerName="istio-proxy"
Oct 30 01:25:20.001881 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:25:20.001632    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="50586f76-4f3b-4e7a-83ac-6c573faee415" containerName="main"
Oct 30 01:25:20.001881 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:25:20.001639    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="91d2a0e6-d589-4747-b9f8-2ec905ce7780" containerName="revoke-keys"
Oct 30 01:25:20.001881 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:25:20.001647    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="099508e4-19ee-4e9b-93f3-0ef7a0d5b8b7" containerName="istio-proxy"
Oct 30 01:25:20.004650 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:25:20.004634    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-2jzrh"
Oct 30 01:25:20.036816 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:25:20.036786    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-2jzrh"]
Oct 30 01:25:20.082561 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:25:20.082524    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49e9b6e9-fa30-4ea2-82e9-69e9e4ecab93-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-2jzrh\" (UID: \"49e9b6e9-fa30-4ea2-82e9-69e9e4ecab93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-2jzrh"
Oct 30 01:25:20.082731 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:25:20.082566    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dz7br\" (UniqueName: \"kubernetes.io/projected/49e9b6e9-fa30-4ea2-82e9-69e9e4ecab93-kube-api-access-dz7br\") pod \"maas-default-gateway-openshift-default-7d9897b54c-2jzrh\" (UID: \"49e9b6e9-fa30-4ea2-82e9-69e9e4ecab93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-2jzrh"
Oct 30 01:25:20.082731 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:25:20.082635    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49e9b6e9-fa30-4ea2-82e9-69e9e4ecab93-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-2jzrh\" (UID: \"49e9b6e9-fa30-4ea2-82e9-69e9e4ecab93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-2jzrh"
Oct 30 01:25:20.082731 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:25:20.082656    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49e9b6e9-fa30-4ea2-82e9-69e9e4ecab93-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-2jzrh\" (UID: \"49e9b6e9-fa30-4ea2-82e9-69e9e4ecab93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-2jzrh"
Oct 30 01:25:20.082849 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:25:20.082735    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49e9b6e9-fa30-4ea2-82e9-69e9e4ecab93-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-2jzrh\" (UID: \"49e9b6e9-fa30-4ea2-82e9-69e9e4ecab93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-2jzrh"
Oct 30 01:25:20.082849 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:25:20.082767    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49e9b6e9-fa30-4ea2-82e9-69e9e4ecab93-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-2jzrh\" (UID: \"49e9b6e9-fa30-4ea2-82e9-69e9e4ecab93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-2jzrh"
Oct 30 01:25:20.082849 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:25:20.082831    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49e9b6e9-fa30-4ea2-82e9-69e9e4ecab93-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-2jzrh\" (UID: \"49e9b6e9-fa30-4ea2-82e9-69e9e4ecab93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-2jzrh"
Oct 30 01:25:20.082951 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:25:20.082870    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49e9b6e9-fa30-4ea2-82e9-69e9e4ecab93-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-2jzrh\" (UID: \"49e9b6e9-fa30-4ea2-82e9-69e9e4ecab93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-2jzrh"
Oct 30 01:25:20.082951 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:25:20.082944    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49e9b6e9-fa30-4ea2-82e9-69e9e4ecab93-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-2jzrh\" (UID: \"49e9b6e9-fa30-4ea2-82e9-69e9e4ecab93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-2jzrh"
Oct 30 01:25:20.184173 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:25:20.184130    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49e9b6e9-fa30-4ea2-82e9-69e9e4ecab93-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-2jzrh\" (UID: \"49e9b6e9-fa30-4ea2-82e9-69e9e4ecab93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-2jzrh"
Oct 30 01:25:20.184416 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:25:20.184182    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49e9b6e9-fa30-4ea2-82e9-69e9e4ecab93-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-2jzrh\" (UID: \"49e9b6e9-fa30-4ea2-82e9-69e9e4ecab93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-2jzrh"
Oct 30 01:25:20.184416 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:25:20.184228    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49e9b6e9-fa30-4ea2-82e9-69e9e4ecab93-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-2jzrh\" (UID: \"49e9b6e9-fa30-4ea2-82e9-69e9e4ecab93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-2jzrh"
Oct 30 01:25:20.184416 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:25:20.184270    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49e9b6e9-fa30-4ea2-82e9-69e9e4ecab93-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-2jzrh\" (UID: \"49e9b6e9-fa30-4ea2-82e9-69e9e4ecab93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-2jzrh"
Oct 30 01:25:20.184416 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:25:20.184300    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dz7br\" (UniqueName: \"kubernetes.io/projected/49e9b6e9-fa30-4ea2-82e9-69e9e4ecab93-kube-api-access-dz7br\") pod \"maas-default-gateway-openshift-default-7d9897b54c-2jzrh\" (UID: \"49e9b6e9-fa30-4ea2-82e9-69e9e4ecab93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-2jzrh"
Oct 30 01:25:20.184416 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:25:20.184339    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49e9b6e9-fa30-4ea2-82e9-69e9e4ecab93-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-2jzrh\" (UID: \"49e9b6e9-fa30-4ea2-82e9-69e9e4ecab93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-2jzrh"
Oct 30 01:25:20.184416 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:25:20.184364    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49e9b6e9-fa30-4ea2-82e9-69e9e4ecab93-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-2jzrh\" (UID: \"49e9b6e9-fa30-4ea2-82e9-69e9e4ecab93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-2jzrh"
Oct 30 01:25:20.184809 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:25:20.184431    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49e9b6e9-fa30-4ea2-82e9-69e9e4ecab93-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-2jzrh\" (UID: \"49e9b6e9-fa30-4ea2-82e9-69e9e4ecab93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-2jzrh"
Oct 30 01:25:20.184809 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:25:20.184458    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49e9b6e9-fa30-4ea2-82e9-69e9e4ecab93-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-2jzrh\" (UID: \"49e9b6e9-fa30-4ea2-82e9-69e9e4ecab93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-2jzrh"
Oct 30 01:25:20.184809 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:25:20.184724    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49e9b6e9-fa30-4ea2-82e9-69e9e4ecab93-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-2jzrh\" (UID: \"49e9b6e9-fa30-4ea2-82e9-69e9e4ecab93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-2jzrh"
Oct 30 01:25:20.184809 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:25:20.184793    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49e9b6e9-fa30-4ea2-82e9-69e9e4ecab93-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-2jzrh\" (UID: \"49e9b6e9-fa30-4ea2-82e9-69e9e4ecab93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-2jzrh"
Oct 30 01:25:20.185034 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:25:20.184836    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49e9b6e9-fa30-4ea2-82e9-69e9e4ecab93-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-2jzrh\" (UID: \"49e9b6e9-fa30-4ea2-82e9-69e9e4ecab93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-2jzrh"
Oct 30 01:25:20.185034 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:25:20.184963    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49e9b6e9-fa30-4ea2-82e9-69e9e4ecab93-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-2jzrh\" (UID: \"49e9b6e9-fa30-4ea2-82e9-69e9e4ecab93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-2jzrh"
Oct 30 01:25:20.185165 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:25:20.185137    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49e9b6e9-fa30-4ea2-82e9-69e9e4ecab93-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-2jzrh\" (UID: \"49e9b6e9-fa30-4ea2-82e9-69e9e4ecab93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-2jzrh"
Oct 30 01:25:20.186696 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:25:20.186670    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49e9b6e9-fa30-4ea2-82e9-69e9e4ecab93-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-2jzrh\" (UID: \"49e9b6e9-fa30-4ea2-82e9-69e9e4ecab93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-2jzrh"
Oct 30 01:25:20.186980 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:25:20.186956    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49e9b6e9-fa30-4ea2-82e9-69e9e4ecab93-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-2jzrh\" (UID: \"49e9b6e9-fa30-4ea2-82e9-69e9e4ecab93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-2jzrh"
Oct 30 01:25:20.209078 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:25:20.209052    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49e9b6e9-fa30-4ea2-82e9-69e9e4ecab93-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-2jzrh\" (UID: \"49e9b6e9-fa30-4ea2-82e9-69e9e4ecab93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-2jzrh"
Oct 30 01:25:20.209235 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:25:20.209198    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dz7br\" (UniqueName: \"kubernetes.io/projected/49e9b6e9-fa30-4ea2-82e9-69e9e4ecab93-kube-api-access-dz7br\") pod \"maas-default-gateway-openshift-default-7d9897b54c-2jzrh\" (UID: \"49e9b6e9-fa30-4ea2-82e9-69e9e4ecab93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-2jzrh"
Oct 30 01:25:20.317393 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:25:20.317288    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-2jzrh"
Oct 30 01:25:20.632355 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:25:20.632325    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-2jzrh"]
Oct 30 01:25:20.633023 ip-10-0-128-7 kubenswrapper[5002]: W1030 01:25:20.632996    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49e9b6e9_fa30_4ea2_82e9_69e9e4ecab93.slice/crio-dc1c3f3d954017e5e527dac2e97af1375351b51e3e986de424aae0e9898d49e1 WatchSource:0}: Error finding container dc1c3f3d954017e5e527dac2e97af1375351b51e3e986de424aae0e9898d49e1: Status 404 returned error can't find the container with id dc1c3f3d954017e5e527dac2e97af1375351b51e3e986de424aae0e9898d49e1
Oct 30 01:25:20.635033 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:25:20.635001    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:25:20.635099 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:25:20.635066    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:25:20.635099 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:25:20.635095    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:25:20.975204 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:25:20.975160    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-2jzrh" event={"ID":"49e9b6e9-fa30-4ea2-82e9-69e9e4ecab93","Type":"ContainerStarted","Data":"8f4e458f83fcad73ff8351e135b84a71b40dea23ade3fa4674cea02d0be5130e"}
Oct 30 01:25:20.975364 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:25:20.975227    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-2jzrh" event={"ID":"49e9b6e9-fa30-4ea2-82e9-69e9e4ecab93","Type":"ContainerStarted","Data":"dc1c3f3d954017e5e527dac2e97af1375351b51e3e986de424aae0e9898d49e1"}
Oct 30 01:25:21.040227 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:25:21.039931    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-2jzrh" podStartSLOduration=2.039911317 podStartE2EDuration="2.039911317s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:25:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:25:21.037028184 +0000 UTC m=+3216.653461062" watchObservedRunningTime="2026-10-30 01:25:21.039911317 +0000 UTC m=+3216.656344197"
Oct 30 01:25:21.317830 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:25:21.317747    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-2jzrh"
Oct 30 01:25:21.319452 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:25:21.319424    5002 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-2jzrh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.25:15021/healthz/ready\": dial tcp 10.132.1.25:15021: connect: connection refused" start-of-body=
Oct 30 01:25:21.319603 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:25:21.319482    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-2jzrh" podUID="49e9b6e9-fa30-4ea2-82e9-69e9e4ecab93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.25:15021/healthz/ready\": dial tcp 10.132.1.25:15021: connect: connection refused"
Oct 30 01:25:22.318234 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:25:22.318197    5002 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-2jzrh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.25:15021/healthz/ready\": dial tcp 10.132.1.25:15021: connect: connection refused" start-of-body=
Oct 30 01:25:22.318697 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:25:22.318251    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-2jzrh" podUID="49e9b6e9-fa30-4ea2-82e9-69e9e4ecab93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.25:15021/healthz/ready\": dial tcp 10.132.1.25:15021: connect: connection refused"
Oct 30 01:25:23.318807 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:25:23.318755    5002 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-2jzrh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.25:15021/healthz/ready\": dial tcp 10.132.1.25:15021: connect: connection refused" start-of-body=
Oct 30 01:25:23.319312 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:25:23.318828    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-2jzrh" podUID="49e9b6e9-fa30-4ea2-82e9-69e9e4ecab93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.25:15021/healthz/ready\": dial tcp 10.132.1.25:15021: connect: connection refused"
Oct 30 01:25:24.318394 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:25:24.318351    5002 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-2jzrh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.25:15021/healthz/ready\": dial tcp 10.132.1.25:15021: connect: connection refused" start-of-body=
Oct 30 01:25:24.318581 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:25:24.318441    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-2jzrh" podUID="49e9b6e9-fa30-4ea2-82e9-69e9e4ecab93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.25:15021/healthz/ready\": dial tcp 10.132.1.25:15021: connect: connection refused"
Oct 30 01:25:25.317935 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:25:25.317898    5002 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-2jzrh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.25:15021/healthz/ready\": dial tcp 10.132.1.25:15021: connect: connection refused" start-of-body=
Oct 30 01:25:25.318449 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:25:25.317956    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-2jzrh" podUID="49e9b6e9-fa30-4ea2-82e9-69e9e4ecab93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.25:15021/healthz/ready\": dial tcp 10.132.1.25:15021: connect: connection refused"
Oct 30 01:25:26.318390 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:25:26.318344    5002 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-2jzrh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.25:15021/healthz/ready\": dial tcp 10.132.1.25:15021: connect: connection refused" start-of-body=
Oct 30 01:25:26.318788 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:25:26.318429    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-2jzrh" podUID="49e9b6e9-fa30-4ea2-82e9-69e9e4ecab93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.25:15021/healthz/ready\": dial tcp 10.132.1.25:15021: connect: connection refused"
Oct 30 01:25:27.318437 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:25:27.318356    5002 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-2jzrh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.25:15021/healthz/ready\": dial tcp 10.132.1.25:15021: connect: connection refused" start-of-body=
Oct 30 01:25:27.318988 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:25:27.318444    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-2jzrh" podUID="49e9b6e9-fa30-4ea2-82e9-69e9e4ecab93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.25:15021/healthz/ready\": dial tcp 10.132.1.25:15021: connect: connection refused"
Oct 30 01:25:28.894211 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:25:28.894164    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-2jzrh"
Oct 30 01:25:28.894750 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:25:28.894723    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-2jzrh"
Oct 30 01:25:28.895612 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:25:28.895584    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-2jzrh"
Oct 30 01:25:42.859893 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:25:42.859854    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbjdq" containerName="extractor"
Oct 30 01:25:45.412303 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:25:45.412256    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7d8z4"]
Oct 30 01:25:49.952844 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:25:49.952815    5002 scope.go:122] "RemoveContainer" containerID="d91c918077d8323d5ae7581e02ce29afc5f1d956547b9196d73aa6930b3369bf"
Oct 30 01:26:02.508638 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:26:02.508598    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7d8z4"]
Oct 30 01:26:05.137263 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:26:05.137215    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-hb9g9"]
Oct 30 01:26:05.142209 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:26:05.142176    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-hb9g9"
Oct 30 01:26:05.172643 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:26:05.172593    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9224697d-3f44-4128-82de-642f6cdda3e3-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-hb9g9\" (UID: \"9224697d-3f44-4128-82de-642f6cdda3e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-hb9g9"
Oct 30 01:26:05.172825 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:26:05.172727    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9224697d-3f44-4128-82de-642f6cdda3e3-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-hb9g9\" (UID: \"9224697d-3f44-4128-82de-642f6cdda3e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-hb9g9"
Oct 30 01:26:05.172891 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:26:05.172850    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9224697d-3f44-4128-82de-642f6cdda3e3-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-hb9g9\" (UID: \"9224697d-3f44-4128-82de-642f6cdda3e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-hb9g9"
Oct 30 01:26:05.172948 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:26:05.172897    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9224697d-3f44-4128-82de-642f6cdda3e3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-hb9g9\" (UID: \"9224697d-3f44-4128-82de-642f6cdda3e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-hb9g9"
Oct 30 01:26:05.172948 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:26:05.172935    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9224697d-3f44-4128-82de-642f6cdda3e3-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-hb9g9\" (UID: \"9224697d-3f44-4128-82de-642f6cdda3e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-hb9g9"
Oct 30 01:26:05.173076 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:26:05.172964    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9224697d-3f44-4128-82de-642f6cdda3e3-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-hb9g9\" (UID: \"9224697d-3f44-4128-82de-642f6cdda3e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-hb9g9"
Oct 30 01:26:05.173076 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:26:05.173021    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9224697d-3f44-4128-82de-642f6cdda3e3-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-hb9g9\" (UID: \"9224697d-3f44-4128-82de-642f6cdda3e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-hb9g9"
Oct 30 01:26:05.173076 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:26:05.173040    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9224697d-3f44-4128-82de-642f6cdda3e3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-hb9g9\" (UID: \"9224697d-3f44-4128-82de-642f6cdda3e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-hb9g9"
Oct 30 01:26:05.173076 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:26:05.173070    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cq89q\" (UniqueName: \"kubernetes.io/projected/9224697d-3f44-4128-82de-642f6cdda3e3-kube-api-access-cq89q\") pod \"maas-default-gateway-openshift-default-7d9897b54c-hb9g9\" (UID: \"9224697d-3f44-4128-82de-642f6cdda3e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-hb9g9"
Oct 30 01:26:05.179589 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:26:05.179562    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-hb9g9"]
Oct 30 01:26:05.274011 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:26:05.273975    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cq89q\" (UniqueName: \"kubernetes.io/projected/9224697d-3f44-4128-82de-642f6cdda3e3-kube-api-access-cq89q\") pod \"maas-default-gateway-openshift-default-7d9897b54c-hb9g9\" (UID: \"9224697d-3f44-4128-82de-642f6cdda3e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-hb9g9"
Oct 30 01:26:05.274193 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:26:05.274040    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9224697d-3f44-4128-82de-642f6cdda3e3-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-hb9g9\" (UID: \"9224697d-3f44-4128-82de-642f6cdda3e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-hb9g9"
Oct 30 01:26:05.274193 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:26:05.274082    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9224697d-3f44-4128-82de-642f6cdda3e3-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-hb9g9\" (UID: \"9224697d-3f44-4128-82de-642f6cdda3e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-hb9g9"
Oct 30 01:26:05.274193 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:26:05.274123    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9224697d-3f44-4128-82de-642f6cdda3e3-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-hb9g9\" (UID: \"9224697d-3f44-4128-82de-642f6cdda3e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-hb9g9"
Oct 30 01:26:05.274193 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:26:05.274153    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9224697d-3f44-4128-82de-642f6cdda3e3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-hb9g9\" (UID: \"9224697d-3f44-4128-82de-642f6cdda3e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-hb9g9"
Oct 30 01:26:05.274193 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:26:05.274184    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9224697d-3f44-4128-82de-642f6cdda3e3-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-hb9g9\" (UID: \"9224697d-3f44-4128-82de-642f6cdda3e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-hb9g9"
Oct 30 01:26:05.274521 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:26:05.274212    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9224697d-3f44-4128-82de-642f6cdda3e3-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-hb9g9\" (UID: \"9224697d-3f44-4128-82de-642f6cdda3e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-hb9g9"
Oct 30 01:26:05.274521 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:26:05.274234    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9224697d-3f44-4128-82de-642f6cdda3e3-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-hb9g9\" (UID: \"9224697d-3f44-4128-82de-642f6cdda3e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-hb9g9"
Oct 30 01:26:05.274521 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:26:05.274259    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9224697d-3f44-4128-82de-642f6cdda3e3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-hb9g9\" (UID: \"9224697d-3f44-4128-82de-642f6cdda3e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-hb9g9"
Oct 30 01:26:05.274638 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:26:05.274569    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9224697d-3f44-4128-82de-642f6cdda3e3-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-hb9g9\" (UID: \"9224697d-3f44-4128-82de-642f6cdda3e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-hb9g9"
Oct 30 01:26:05.274675 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:26:05.274629    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9224697d-3f44-4128-82de-642f6cdda3e3-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-hb9g9\" (UID: \"9224697d-3f44-4128-82de-642f6cdda3e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-hb9g9"
Oct 30 01:26:05.274715 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:26:05.274683    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9224697d-3f44-4128-82de-642f6cdda3e3-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-hb9g9\" (UID: \"9224697d-3f44-4128-82de-642f6cdda3e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-hb9g9"
Oct 30 01:26:05.274773 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:26:05.274750    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9224697d-3f44-4128-82de-642f6cdda3e3-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-hb9g9\" (UID: \"9224697d-3f44-4128-82de-642f6cdda3e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-hb9g9"
Oct 30 01:26:05.274939 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:26:05.274915    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9224697d-3f44-4128-82de-642f6cdda3e3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-hb9g9\" (UID: \"9224697d-3f44-4128-82de-642f6cdda3e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-hb9g9"
Oct 30 01:26:05.276734 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:26:05.276710    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9224697d-3f44-4128-82de-642f6cdda3e3-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-hb9g9\" (UID: \"9224697d-3f44-4128-82de-642f6cdda3e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-hb9g9"
Oct 30 01:26:05.276850 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:26:05.276800    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9224697d-3f44-4128-82de-642f6cdda3e3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-hb9g9\" (UID: \"9224697d-3f44-4128-82de-642f6cdda3e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-hb9g9"
Oct 30 01:26:05.301497 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:26:05.301467    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9224697d-3f44-4128-82de-642f6cdda3e3-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-hb9g9\" (UID: \"9224697d-3f44-4128-82de-642f6cdda3e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-hb9g9"
Oct 30 01:26:05.301659 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:26:05.301569    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cq89q\" (UniqueName: \"kubernetes.io/projected/9224697d-3f44-4128-82de-642f6cdda3e3-kube-api-access-cq89q\") pod \"maas-default-gateway-openshift-default-7d9897b54c-hb9g9\" (UID: \"9224697d-3f44-4128-82de-642f6cdda3e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-hb9g9"
Oct 30 01:26:05.457058 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:26:05.457018    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-hb9g9"
Oct 30 01:26:05.772368 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:26:05.772338    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-hb9g9"]
Oct 30 01:26:05.773087 ip-10-0-128-7 kubenswrapper[5002]: W1030 01:26:05.773058    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9224697d_3f44_4128_82de_642f6cdda3e3.slice/crio-61a21b87c15e4e93873693a37de28b11f35eff794db8037f2ada9c7506067bbd WatchSource:0}: Error finding container 61a21b87c15e4e93873693a37de28b11f35eff794db8037f2ada9c7506067bbd: Status 404 returned error can't find the container with id 61a21b87c15e4e93873693a37de28b11f35eff794db8037f2ada9c7506067bbd
Oct 30 01:26:05.775214 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:26:05.775181    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:26:05.775305 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:26:05.775246    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:26:05.775305 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:26:05.775276    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:26:06.228301 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:26:06.228256    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-hb9g9" event={"ID":"9224697d-3f44-4128-82de-642f6cdda3e3","Type":"ContainerStarted","Data":"0c7fa84d18e015303deb967d48a3e99e7b443e40a888b9c1abb34b93269d44a6"}
Oct 30 01:26:06.228742 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:26:06.228322    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-hb9g9" event={"ID":"9224697d-3f44-4128-82de-642f6cdda3e3","Type":"ContainerStarted","Data":"61a21b87c15e4e93873693a37de28b11f35eff794db8037f2ada9c7506067bbd"}
Oct 30 01:26:06.279607 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:26:06.279548    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-hb9g9" podStartSLOduration=1.279533807 podStartE2EDuration="1.279533807s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:26:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:26:06.278761401 +0000 UTC m=+3261.895194280" watchObservedRunningTime="2026-10-30 01:26:06.279533807 +0000 UTC m=+3261.895966686"
Oct 30 01:26:06.468541 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:26:06.468452    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-hb9g9"
Oct 30 01:26:06.474404 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:26:06.472521    5002 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-hb9g9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.26:15021/healthz/ready\": dial tcp 10.132.1.26:15021: connect: connection refused" start-of-body=
Oct 30 01:26:06.474404 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:26:06.472594    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-hb9g9" podUID="9224697d-3f44-4128-82de-642f6cdda3e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.26:15021/healthz/ready\": dial tcp 10.132.1.26:15021: connect: connection refused"
Oct 30 01:26:07.306330 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:26:07.306294    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7d8z4"]
Oct 30 01:26:07.465864 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:26:07.465821    5002 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-hb9g9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.26:15021/healthz/ready\": dial tcp 10.132.1.26:15021: connect: connection refused" start-of-body=
Oct 30 01:26:07.466156 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:26:07.466123    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-hb9g9" podUID="9224697d-3f44-4128-82de-642f6cdda3e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.26:15021/healthz/ready\": dial tcp 10.132.1.26:15021: connect: connection refused"
Oct 30 01:26:08.461406 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:26:08.460840    5002 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-hb9g9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.26:15021/healthz/ready\": dial tcp 10.132.1.26:15021: connect: connection refused" start-of-body=
Oct 30 01:26:08.461406 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:26:08.460950    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-hb9g9" podUID="9224697d-3f44-4128-82de-642f6cdda3e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.26:15021/healthz/ready\": dial tcp 10.132.1.26:15021: connect: connection refused"
Oct 30 01:26:09.462403 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:26:09.461862    5002 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-hb9g9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.26:15021/healthz/ready\": dial tcp 10.132.1.26:15021: connect: connection refused" start-of-body=
Oct 30 01:26:09.462403 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:26:09.461931    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-hb9g9" podUID="9224697d-3f44-4128-82de-642f6cdda3e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.26:15021/healthz/ready\": dial tcp 10.132.1.26:15021: connect: connection refused"
Oct 30 01:26:10.461404 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:26:10.460768    5002 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-hb9g9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.26:15021/healthz/ready\": dial tcp 10.132.1.26:15021: connect: connection refused" start-of-body=
Oct 30 01:26:10.461404 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:26:10.460839    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-hb9g9" podUID="9224697d-3f44-4128-82de-642f6cdda3e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.26:15021/healthz/ready\": dial tcp 10.132.1.26:15021: connect: connection refused"
Oct 30 01:26:11.466405 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:26:11.462782    5002 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-hb9g9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.26:15021/healthz/ready\": dial tcp 10.132.1.26:15021: connect: connection refused" start-of-body=
Oct 30 01:26:11.466405 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:26:11.462846    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-hb9g9" podUID="9224697d-3f44-4128-82de-642f6cdda3e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.26:15021/healthz/ready\": dial tcp 10.132.1.26:15021: connect: connection refused"
Oct 30 01:26:12.458105 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:26:12.458072    5002 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-hb9g9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.26:15021/healthz/ready\": dial tcp 10.132.1.26:15021: connect: connection refused" start-of-body=
Oct 30 01:26:12.458289 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:26:12.458124    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-hb9g9" podUID="9224697d-3f44-4128-82de-642f6cdda3e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.26:15021/healthz/ready\": dial tcp 10.132.1.26:15021: connect: connection refused"
Oct 30 01:26:12.861201 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:26:12.861106    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbjdq" containerName="extractor"
Oct 30 01:26:14.458370 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:26:14.458321    5002 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-hb9g9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.26:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 01:26:14.458924 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:26:14.458373    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-hb9g9" podUID="9224697d-3f44-4128-82de-642f6cdda3e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.26:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:26:15.499588 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:26:15.499536    5002 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-hb9g9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.26:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 01:26:15.500139 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:26:15.499602    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-hb9g9" podUID="9224697d-3f44-4128-82de-642f6cdda3e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.26:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:26:16.129224 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:26:16.129187    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-hb9g9"
Oct 30 01:26:16.129745 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:26:16.129722    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-hb9g9"
Oct 30 01:26:16.130498 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:26:16.130481    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-hb9g9"
Oct 30 01:26:25.789943 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:26:25.789853    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7d8z4"]
Oct 30 01:26:41.322986 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:26:41.322944    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7d8z4"]
Oct 30 01:26:42.932854 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:26:42.932799    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbjdq" containerName="extractor"
Oct 30 01:26:45.283710 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:26:45.283680    5002 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-g7c2w_5dc56e7e-f3c1-43eb-a21b-8be6bb4706cf/console-operator/1.log"
Oct 30 01:26:45.313789 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:26:45.313759    5002 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-g7c2w_5dc56e7e-f3c1-43eb-a21b-8be6bb4706cf/console-operator/1.log"
Oct 30 01:26:49.973931 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:26:49.973899    5002 scope.go:122] "RemoveContainer" containerID="80d96b84010ed176848a534e6b53080569a82ff93f3323ee9ab470ea64e34a65"
Oct 30 01:26:49.982663 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:26:49.982645    5002 scope.go:122] "RemoveContainer" containerID="11d649d9fa32c403be495fce07741e2c3988d49744e247bd10a44082bbbd99bb"
Oct 30 01:26:49.990513 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:26:49.990495    5002 scope.go:122] "RemoveContainer" containerID="86ae70df480ea8c844011ee7d41692d8c9cef62ee847dfdae0e26114d8a4a995"
Oct 30 01:26:49.998149 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:26:49.998124    5002 scope.go:122] "RemoveContainer" containerID="c1cbe437652412f8cf52c5ee01bf4aa497dea6c8629ea009a1d43bbff18a9282"
Oct 30 01:27:04.888714 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:27:04.888670    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7d8z4"]
Oct 30 01:27:12.860357 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:27:12.860318    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbjdq" containerName="extractor"
Oct 30 01:27:15.421013 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:27:15.420976    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7d8z4"]
Oct 30 01:27:19.910397 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:27:19.910342    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7d8z4"]
Oct 30 01:27:23.032753 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:27:23.032713    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c3c6878-mwshd"]
Oct 30 01:27:23.046441 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:27:23.046404    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c3c6878-mwshd"]
Oct 30 01:27:24.503455 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:27:24.503405    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b878fdb-4d45-4e91-910e-a10aec5326e6" path="/var/lib/kubelet/pods/9b878fdb-4d45-4e91-910e-a10aec5326e6/volumes"
Oct 30 01:27:38.016692 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:27:38.016650    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7d8z4"]
Oct 30 01:27:42.946972 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:27:42.946918    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbjdq" containerName="extractor"
Oct 30 01:27:43.073403 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:27:43.072669    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7d8z4"]
Oct 30 01:27:50.030892 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:27:50.030854    5002 scope.go:122] "RemoveContainer" containerID="d427548acd3258707e85ee7c7ba1a0c29897dfd2c94184abf2c0fc9dc3cc9bc6"
Oct 30 01:27:50.039686 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:27:50.039660    5002 scope.go:122] "RemoveContainer" containerID="39d626c438fe1b3f09a2ee53154e67f4b28bba3e310f9d4b3f39090a03f39399"
Oct 30 01:27:59.677888 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:27:59.677800    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7d8z4"]
Oct 30 01:28:04.702049 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:28:04.702006    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7d8z4"]
Oct 30 01:28:12.860546 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:28:12.860509    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbjdq" containerName="extractor"
Oct 30 01:28:21.190197 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:28:21.190156    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7d8z4"]
Oct 30 01:28:24.231408 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:28:24.230474    5002 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-pdgh4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 01:28:24.231408 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:28:24.230558    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-pdgh4" podUID="c341e49a-5e59-414d-a432-e338bef60081" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:28:37.220277 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:28:37.220236    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7d8z4"]
Oct 30 01:28:42.860897 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:28:42.860855    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbjdq" containerName="extractor"
Oct 30 01:28:48.028856 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:28:48.028814    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7d8z4"]
Oct 30 01:28:50.068512 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:28:50.068473    5002 scope.go:122] "RemoveContainer" containerID="b254f80b2e38979762ddab4f2d84ee77460cfa08d1d56bcc8e55e04cdc01e5df"
Oct 30 01:28:50.107822 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:28:50.107794    5002 scope.go:122] "RemoveContainer" containerID="642555ab9c1737905a24a3577f3fd39390af54d2d5aebde47da78edcb98a1a6d"
Oct 30 01:28:50.130177 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:28:50.130148    5002 scope.go:122] "RemoveContainer" containerID="87e2f1653f106f3b7486b227c24256dc236733d15945bcbec2ca7cefcf4e7215"
Oct 30 01:28:53.089128 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:28:53.089093    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7d8z4"]
Oct 30 01:29:06.034548 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:29:06.034511    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b864cda-97k9j"]
Oct 30 01:29:06.048305 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:29:06.048271    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b864cda-97k9j"]
Oct 30 01:29:06.493410 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:29:06.493360    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="921cb648-96f3-483f-8984-0359a6208ce0" path="/var/lib/kubelet/pods/921cb648-96f3-483f-8984-0359a6208ce0/volumes"
Oct 30 01:29:10.330986 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:29:10.330950    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7d8z4"]
Oct 30 01:29:12.860918 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:29:12.860873    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbjdq" containerName="extractor"
Oct 30 01:29:15.028560 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:29:15.028518    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-baf2df27-gbnsl"]
Oct 30 01:29:15.042436 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:29:15.042399    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-baf2df27-gbnsl"]
Oct 30 01:29:15.509481 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:29:15.509447    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7d8z4"]
Oct 30 01:29:16.494751 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:29:16.494718    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91d2a0e6-d589-4747-b9f8-2ec905ce7780" path="/var/lib/kubelet/pods/91d2a0e6-d589-4747-b9f8-2ec905ce7780/volumes"
Oct 30 01:29:34.409638 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:29:34.409557    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7d8z4"]
Oct 30 01:29:39.405570 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:29:39.405529    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7d8z4"]
Oct 30 01:29:42.860143 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:29:42.860103    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbjdq" containerName="extractor"
Oct 30 01:29:52.720080 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:29:52.720032    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7d8z4"]
Oct 30 01:30:00.151242 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:30:00.151203    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-3ab7f1-2988873rtv4h"]
Oct 30 01:30:00.155008 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:30:00.154988    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-3ab7f1-2988873rtv4h"
Oct 30 01:30:00.176038 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:30:00.176005    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-3ab7f1-2988873rtv4h"]
Oct 30 01:30:00.269332 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:30:00.269289    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:30:00.269557 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:30:00.269355    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:30:00.269557 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:30:00.269409    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5phsh\""
Oct 30 01:30:00.326650 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:30:00.326615    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7nqh\" (UniqueName: \"kubernetes.io/projected/89fd0c36-a17c-46a9-8a81-05f9717d48f3-kube-api-access-s7nqh\") pod \"maas-api-key-cleanup-e2e-worker-single-main-3ab7f1-2988873rtv4h\" (UID: \"89fd0c36-a17c-46a9-8a81-05f9717d48f3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-3ab7f1-2988873rtv4h"
Oct 30 01:30:00.326826 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:30:00.326657    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/89fd0c36-a17c-46a9-8a81-05f9717d48f3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-single-main-3ab7f1-2988873rtv4h\" (UID: \"89fd0c36-a17c-46a9-8a81-05f9717d48f3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-3ab7f1-2988873rtv4h"
Oct 30 01:30:00.427618 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:30:00.427541    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7nqh\" (UniqueName: \"kubernetes.io/projected/89fd0c36-a17c-46a9-8a81-05f9717d48f3-kube-api-access-s7nqh\") pod \"maas-api-key-cleanup-e2e-worker-single-main-3ab7f1-2988873rtv4h\" (UID: \"89fd0c36-a17c-46a9-8a81-05f9717d48f3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-3ab7f1-2988873rtv4h"
Oct 30 01:30:00.427618 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:30:00.427588    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/89fd0c36-a17c-46a9-8a81-05f9717d48f3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-single-main-3ab7f1-2988873rtv4h\" (UID: \"89fd0c36-a17c-46a9-8a81-05f9717d48f3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-3ab7f1-2988873rtv4h"
Oct 30 01:30:00.428252 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:30:00.428231    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/89fd0c36-a17c-46a9-8a81-05f9717d48f3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-single-main-3ab7f1-2988873rtv4h\" (UID: \"89fd0c36-a17c-46a9-8a81-05f9717d48f3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-3ab7f1-2988873rtv4h"
Oct 30 01:30:00.453085 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:30:00.453053    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7nqh\" (UniqueName: \"kubernetes.io/projected/89fd0c36-a17c-46a9-8a81-05f9717d48f3-kube-api-access-s7nqh\") pod \"maas-api-key-cleanup-e2e-worker-single-main-3ab7f1-2988873rtv4h\" (UID: \"89fd0c36-a17c-46a9-8a81-05f9717d48f3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-3ab7f1-2988873rtv4h"
Oct 30 01:30:00.469833 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:30:00.469809    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-3ab7f1-2988873rtv4h"
Oct 30 01:30:00.769404 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:30:00.769349    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-3ab7f1-2988873rtv4h"]
Oct 30 01:30:00.769897 ip-10-0-128-7 kubenswrapper[5002]: W1030 01:30:00.769871    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89fd0c36_a17c_46a9_8a81_05f9717d48f3.slice/crio-8db283a5dab94c97a0cbbd40dff2847343e91f4c3cb2898fdbcc31a36b7e4f08 WatchSource:0}: Error finding container 8db283a5dab94c97a0cbbd40dff2847343e91f4c3cb2898fdbcc31a36b7e4f08: Status 404 returned error can't find the container with id 8db283a5dab94c97a0cbbd40dff2847343e91f4c3cb2898fdbcc31a36b7e4f08
Oct 30 01:30:01.623697 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:30:01.623664    5002 generic.go:356] "Generic (PLEG): container finished" podID="89fd0c36-a17c-46a9-8a81-05f9717d48f3" containerID="4a358c0b8574cb10bfad97d1c2bc18f6aaa040bb98d40a8afccdfe86bc42c6e6" exitCode=60
Oct 30 01:30:01.624087 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:30:01.623749    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-3ab7f1-2988873rtv4h" event={"ID":"89fd0c36-a17c-46a9-8a81-05f9717d48f3","Type":"ContainerDied","Data":"4a358c0b8574cb10bfad97d1c2bc18f6aaa040bb98d40a8afccdfe86bc42c6e6"}
Oct 30 01:30:01.624087 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:30:01.623803    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-3ab7f1-2988873rtv4h" event={"ID":"89fd0c36-a17c-46a9-8a81-05f9717d48f3","Type":"ContainerStarted","Data":"8db283a5dab94c97a0cbbd40dff2847343e91f4c3cb2898fdbcc31a36b7e4f08"}
Oct 30 01:30:01.624087 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:30:01.623958    5002 scope.go:122] "RemoveContainer" containerID="4a358c0b8574cb10bfad97d1c2bc18f6aaa040bb98d40a8afccdfe86bc42c6e6"
Oct 30 01:30:02.630442 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:30:02.630368    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-3ab7f1-2988873rtv4h" event={"ID":"89fd0c36-a17c-46a9-8a81-05f9717d48f3","Type":"ContainerStarted","Data":"a01ad3c8c9294b1592cdb7995ca9b3214f4e1bbb1d2f6f1ba0813f163f240141"}
Oct 30 01:30:02.674461 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:30:02.674414    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-3ab7f1-2988873rtv4h" podStartSLOduration=2.67440187 podStartE2EDuration="2.67440187s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:30:02.673091556 +0000 UTC m=+3498.289524434" watchObservedRunningTime="2026-10-30 01:30:02.67440187 +0000 UTC m=+3498.290834745"
Oct 30 01:30:03.637397 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:30:03.637350    5002 generic.go:356] "Generic (PLEG): container finished" podID="89fd0c36-a17c-46a9-8a81-05f9717d48f3" containerID="a01ad3c8c9294b1592cdb7995ca9b3214f4e1bbb1d2f6f1ba0813f163f240141" exitCode=60
Oct 30 01:30:03.637919 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:30:03.637432    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-3ab7f1-2988873rtv4h" event={"ID":"89fd0c36-a17c-46a9-8a81-05f9717d48f3","Type":"ContainerDied","Data":"a01ad3c8c9294b1592cdb7995ca9b3214f4e1bbb1d2f6f1ba0813f163f240141"}
Oct 30 01:30:03.637919 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:30:03.637508    5002 scope.go:122] "RemoveContainer" containerID="4a358c0b8574cb10bfad97d1c2bc18f6aaa040bb98d40a8afccdfe86bc42c6e6"
Oct 30 01:30:03.637919 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:30:03.637735    5002 scope.go:122] "RemoveContainer" containerID="a01ad3c8c9294b1592cdb7995ca9b3214f4e1bbb1d2f6f1ba0813f163f240141"
Oct 30 01:30:03.638143 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:30:03.638002    5002 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-single-main-3ab7f1-2988873rtv4h_odh-ai-gateway-infra(89fd0c36-a17c-46a9-8a81-05f9717d48f3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-3ab7f1-2988873rtv4h" podUID="89fd0c36-a17c-46a9-8a81-05f9717d48f3"
Oct 30 01:30:04.644986 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:30:04.644946    5002 scope.go:122] "RemoveContainer" containerID="a01ad3c8c9294b1592cdb7995ca9b3214f4e1bbb1d2f6f1ba0813f163f240141"
Oct 30 01:30:04.645409 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:30:04.645173    5002 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-single-main-3ab7f1-2988873rtv4h_odh-ai-gateway-infra(89fd0c36-a17c-46a9-8a81-05f9717d48f3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-3ab7f1-2988873rtv4h" podUID="89fd0c36-a17c-46a9-8a81-05f9717d48f3"
Oct 30 01:30:06.299637 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:30:06.299596    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7d8z4"]
Oct 30 01:30:12.860422 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:30:12.860359    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbjdq" containerName="extractor"
Oct 30 01:30:14.489418 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:30:14.489372    5002 scope.go:122] "RemoveContainer" containerID="a01ad3c8c9294b1592cdb7995ca9b3214f4e1bbb1d2f6f1ba0813f163f240141"
Oct 30 01:30:15.701485 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:30:15.701450    5002 generic.go:356] "Generic (PLEG): container finished" podID="89fd0c36-a17c-46a9-8a81-05f9717d48f3" containerID="b0152cb91fff066ca1f72ae578a9ee4cccad081bf99b37781cb49bece846c8dc" exitCode=60
Oct 30 01:30:15.701912 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:30:15.701512    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-3ab7f1-2988873rtv4h" event={"ID":"89fd0c36-a17c-46a9-8a81-05f9717d48f3","Type":"ContainerDied","Data":"b0152cb91fff066ca1f72ae578a9ee4cccad081bf99b37781cb49bece846c8dc"}
Oct 30 01:30:15.701912 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:30:15.701559    5002 scope.go:122] "RemoveContainer" containerID="a01ad3c8c9294b1592cdb7995ca9b3214f4e1bbb1d2f6f1ba0813f163f240141"
Oct 30 01:30:15.701912 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:30:15.701889    5002 scope.go:122] "RemoveContainer" containerID="b0152cb91fff066ca1f72ae578a9ee4cccad081bf99b37781cb49bece846c8dc"
Oct 30 01:30:15.702162 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:30:15.702137    5002 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-single-main-3ab7f1-2988873rtv4h_odh-ai-gateway-infra(89fd0c36-a17c-46a9-8a81-05f9717d48f3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-3ab7f1-2988873rtv4h" podUID="89fd0c36-a17c-46a9-8a81-05f9717d48f3"
Oct 30 01:30:16.754502 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:30:16.754460    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-3ab7f1-2988873rtv4h"]
Oct 30 01:30:16.906590 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:30:16.906560    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-3ab7f1-2988873rtv4h"
Oct 30 01:30:16.990269 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:30:16.990236    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89fd0c36-a17c-46a9-8a81-05f9717d48f3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/89fd0c36-a17c-46a9-8a81-05f9717d48f3-maas-api-service-ca\") pod \"89fd0c36-a17c-46a9-8a81-05f9717d48f3\" (UID: \"89fd0c36-a17c-46a9-8a81-05f9717d48f3\") "
Oct 30 01:30:16.990269 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:30:16.990271    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89fd0c36-a17c-46a9-8a81-05f9717d48f3-kube-api-access-s7nqh\" (UniqueName: \"kubernetes.io/projected/89fd0c36-a17c-46a9-8a81-05f9717d48f3-kube-api-access-s7nqh\") pod \"89fd0c36-a17c-46a9-8a81-05f9717d48f3\" (UID: \"89fd0c36-a17c-46a9-8a81-05f9717d48f3\") "
Oct 30 01:30:16.990632 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:30:16.990605    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89fd0c36-a17c-46a9-8a81-05f9717d48f3-maas-api-service-ca" pod "89fd0c36-a17c-46a9-8a81-05f9717d48f3" (UID: "89fd0c36-a17c-46a9-8a81-05f9717d48f3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:30:16.992528 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:30:16.992503    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89fd0c36-a17c-46a9-8a81-05f9717d48f3-kube-api-access-s7nqh" pod "89fd0c36-a17c-46a9-8a81-05f9717d48f3" (UID: "89fd0c36-a17c-46a9-8a81-05f9717d48f3"). InnerVolumeSpecName "kube-api-access-s7nqh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:30:17.092063 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:30:17.091974    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/89fd0c36-a17c-46a9-8a81-05f9717d48f3-maas-api-service-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:30:17.092063 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:30:17.092010    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7nqh\" (UniqueName: \"kubernetes.io/projected/89fd0c36-a17c-46a9-8a81-05f9717d48f3-kube-api-access-s7nqh\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:30:17.714786 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:30:17.714753    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-3ab7f1-2988873rtv4h" event={"ID":"89fd0c36-a17c-46a9-8a81-05f9717d48f3","Type":"ContainerDied","Data":"8db283a5dab94c97a0cbbd40dff2847343e91f4c3cb2898fdbcc31a36b7e4f08"}
Oct 30 01:30:17.714979 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:30:17.714793    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-3ab7f1-2988873rtv4h"
Oct 30 01:30:17.714979 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:30:17.714813    5002 scope.go:122] "RemoveContainer" containerID="b0152cb91fff066ca1f72ae578a9ee4cccad081bf99b37781cb49bece846c8dc"
Oct 30 01:30:17.773516 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:30:17.773357    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-3ab7f1-2988873rtv4h"]
Oct 30 01:30:17.786935 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:30:17.786903    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-3ab7f1-2988873rtv4h"]
Oct 30 01:30:18.493116 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:30:18.493076    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89fd0c36-a17c-46a9-8a81-05f9717d48f3" path="/var/lib/kubelet/pods/89fd0c36-a17c-46a9-8a81-05f9717d48f3/volumes"
Oct 30 01:30:22.112450 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:30:22.112411    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7d8z4"]
Oct 30 01:30:39.525587 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:30:39.525549    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7d8z4"]
Oct 30 01:30:42.864243 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:30:42.864203    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbjdq" containerName="extractor"
Oct 30 01:30:50.239964 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:30:50.239928    5002 scope.go:122] "RemoveContainer" containerID="b5ebba3928ec50f83223f15e9af5acbd34426a1ac0c308e9fc7278c0bda2afa4"
Oct 30 01:30:50.248436 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:30:50.248415    5002 scope.go:122] "RemoveContainer" containerID="428ca539e50fc07913a9ae98b9d613185c460779b293d9d92fd3da8d764eedc5"
Oct 30 01:30:53.717842 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:30:53.717797    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7d8z4"]
Oct 30 01:31:12.861353 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:31:12.861315    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbjdq" containerName="extractor"
Oct 30 01:31:13.130867 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:31:13.130772    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7d8z4"]
Oct 30 01:31:14.992404 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:31:14.972627    5002 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-2jzrh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.25:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 01:31:14.992404 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:31:14.972754    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-2jzrh" podUID="49e9b6e9-fa30-4ea2-82e9-69e9e4ecab93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.25:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:31:20.201794 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:31:20.201756    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7d8z4"]
Oct 30 01:31:28.485647 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:31:28.485607    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-gm5b5"]
Oct 30 01:31:28.487989 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:31:28.485837    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-gm5b5" podUID="4d35acb0-477c-436e-bb24-7e15e827a5af" containerName="manager" containerID="cri-o://42929532b7cbea74a7a29b0eaa8225607f945ce3c87371fdb38af2c7c1bb89e0" gracePeriod=10
Oct 30 01:31:28.748819 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:31:28.748759    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-gm5b5"
Oct 30 01:31:28.846022 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:31:28.845990    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d35acb0-477c-436e-bb24-7e15e827a5af-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4d35acb0-477c-436e-bb24-7e15e827a5af-extensions-socket-volume\") pod \"4d35acb0-477c-436e-bb24-7e15e827a5af\" (UID: \"4d35acb0-477c-436e-bb24-7e15e827a5af\") "
Oct 30 01:31:28.846238 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:31:28.846129    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d35acb0-477c-436e-bb24-7e15e827a5af-kube-api-access-xc7g8\" (UniqueName: \"kubernetes.io/projected/4d35acb0-477c-436e-bb24-7e15e827a5af-kube-api-access-xc7g8\") pod \"4d35acb0-477c-436e-bb24-7e15e827a5af\" (UID: \"4d35acb0-477c-436e-bb24-7e15e827a5af\") "
Oct 30 01:31:28.846422 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:31:28.846395    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d35acb0-477c-436e-bb24-7e15e827a5af-extensions-socket-volume" pod "4d35acb0-477c-436e-bb24-7e15e827a5af" (UID: "4d35acb0-477c-436e-bb24-7e15e827a5af"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:31:28.848595 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:31:28.848569    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d35acb0-477c-436e-bb24-7e15e827a5af-kube-api-access-xc7g8" pod "4d35acb0-477c-436e-bb24-7e15e827a5af" (UID: "4d35acb0-477c-436e-bb24-7e15e827a5af"). InnerVolumeSpecName "kube-api-access-xc7g8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:31:28.946791 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:31:28.946753    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xc7g8\" (UniqueName: \"kubernetes.io/projected/4d35acb0-477c-436e-bb24-7e15e827a5af-kube-api-access-xc7g8\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:31:28.946791 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:31:28.946782    5002 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4d35acb0-477c-436e-bb24-7e15e827a5af-extensions-socket-volume\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:31:29.099926 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:31:29.099837    5002 generic.go:356] "Generic (PLEG): container finished" podID="4d35acb0-477c-436e-bb24-7e15e827a5af" containerID="42929532b7cbea74a7a29b0eaa8225607f945ce3c87371fdb38af2c7c1bb89e0" exitCode=0
Oct 30 01:31:29.099926 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:31:29.099891    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-gm5b5"
Oct 30 01:31:29.099926 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:31:29.099917    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-gm5b5" event={"ID":"4d35acb0-477c-436e-bb24-7e15e827a5af","Type":"ContainerDied","Data":"42929532b7cbea74a7a29b0eaa8225607f945ce3c87371fdb38af2c7c1bb89e0"}
Oct 30 01:31:29.100203 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:31:29.099973    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-gm5b5" event={"ID":"4d35acb0-477c-436e-bb24-7e15e827a5af","Type":"ContainerDied","Data":"17f17ecfa55702e03cf07df1521e8dc503185a8752fed0442aa36b15a41ffe52"}
Oct 30 01:31:29.100203 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:31:29.099995    5002 scope.go:122] "RemoveContainer" containerID="42929532b7cbea74a7a29b0eaa8225607f945ce3c87371fdb38af2c7c1bb89e0"
Oct 30 01:31:29.112458 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:31:29.112437    5002 scope.go:122] "RemoveContainer" containerID="42929532b7cbea74a7a29b0eaa8225607f945ce3c87371fdb38af2c7c1bb89e0"
Oct 30 01:31:29.112754 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:31:29.112735    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42929532b7cbea74a7a29b0eaa8225607f945ce3c87371fdb38af2c7c1bb89e0\": container with ID starting with 42929532b7cbea74a7a29b0eaa8225607f945ce3c87371fdb38af2c7c1bb89e0 not found: ID does not exist" containerID="42929532b7cbea74a7a29b0eaa8225607f945ce3c87371fdb38af2c7c1bb89e0"
Oct 30 01:31:29.112801 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:31:29.112767    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42929532b7cbea74a7a29b0eaa8225607f945ce3c87371fdb38af2c7c1bb89e0"} err="failed to get container status \"42929532b7cbea74a7a29b0eaa8225607f945ce3c87371fdb38af2c7c1bb89e0\": rpc error: code = NotFound desc = could not find container \"42929532b7cbea74a7a29b0eaa8225607f945ce3c87371fdb38af2c7c1bb89e0\": container with ID starting with 42929532b7cbea74a7a29b0eaa8225607f945ce3c87371fdb38af2c7c1bb89e0 not found: ID does not exist"
Oct 30 01:31:29.155364 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:31:29.155335    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-gm5b5"]
Oct 30 01:31:29.169172 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:31:29.169141    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-gm5b5"]
Oct 30 01:31:30.493342 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:31:30.493296    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d35acb0-477c-436e-bb24-7e15e827a5af" path="/var/lib/kubelet/pods/4d35acb0-477c-436e-bb24-7e15e827a5af/volumes"
Oct 30 01:31:42.860938 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:31:42.860898    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbjdq" containerName="extractor"
Oct 30 01:31:45.312712 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:31:45.312675    5002 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-g7c2w_5dc56e7e-f3c1-43eb-a21b-8be6bb4706cf/console-operator/1.log"
Oct 30 01:31:45.342504 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:31:45.342476    5002 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-g7c2w_5dc56e7e-f3c1-43eb-a21b-8be6bb4706cf/console-operator/1.log"
Oct 30 01:32:12.860823 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:32:12.860784    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbjdq" containerName="extractor"
Oct 30 01:32:42.860958 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:32:42.860915    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbjdq" containerName="extractor"
Oct 30 01:32:51.754797 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:32:51.754761    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7d8z4"]
Oct 30 01:32:54.231527 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:32:54.231488    5002 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-pdgh4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 01:32:54.231978 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:32:54.231545    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-pdgh4" podUID="c341e49a-5e59-414d-a432-e338bef60081" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:33:06.970218 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:33:06.970180    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7d8z4"]
Oct 30 01:33:12.860213 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:33:12.860168    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbjdq" containerName="extractor"
Oct 30 01:33:33.489283 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:33:33.489245    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7d8z4"]
Oct 30 01:33:42.861999 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:33:42.861950    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbjdq" containerName="extractor"
Oct 30 01:33:47.464148 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:33:47.464108    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7d8z4"]
Oct 30 01:34:02.653293 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:34:02.653209    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7d8z4"]
Oct 30 01:34:12.859791 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:34:12.859754    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbjdq" containerName="extractor"
Oct 30 01:34:16.044035 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:34:16.043995    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7d8z4"]
Oct 30 01:34:30.889650 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:34:30.889607    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7d8z4"]
Oct 30 01:34:42.860178 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:34:42.860136    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbjdq" containerName="extractor"
Oct 30 01:34:43.370969 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:34:43.370930    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7d8z4"]
Oct 30 01:34:59.284441 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:34:59.284399    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7d8z4"]
Oct 30 01:35:02.172520 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:35:02.172479    5002 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-hb9g9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.26:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 01:35:02.173554 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:35:02.173516    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-hb9g9" podUID="9224697d-3f44-4128-82de-642f6cdda3e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.26:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:35:12.861309 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:35:12.861264    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbjdq" containerName="extractor"
Oct 30 01:35:19.879486 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:35:19.879427    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7d8z4"]
Oct 30 01:35:33.934347 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:35:33.934256    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7d8z4"]
Oct 30 01:35:40.071513 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:35:40.071448    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7d8z4"]
Oct 30 01:35:42.925176 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:35:42.925130    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbjdq" containerName="extractor"
Oct 30 01:35:52.635607 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:35:52.635564    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7d8z4"]
Oct 30 01:35:59.445574 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:35:59.445523    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7d8z4"]
Oct 30 01:36:12.860244 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:36:12.860206    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbjdq" containerName="extractor"
Oct 30 01:36:16.230988 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:36:16.230951    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7d8z4"]
Oct 30 01:36:25.929055 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:36:25.929018    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7d8z4"]
Oct 30 01:36:34.037731 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:36:34.037685    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7d8z4"]
Oct 30 01:36:39.506578 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:36:39.506540    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-c8tcz"]
Oct 30 01:36:39.507018 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:36:39.506910    5002 cpu_manager.go:414] "containerMap: removing container" podUID="89fd0c36-a17c-46a9-8a81-05f9717d48f3" containerName="cleanup"
Oct 30 01:36:39.507018 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:36:39.506920    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89fd0c36-a17c-46a9-8a81-05f9717d48f3" containerName="cleanup"
Oct 30 01:36:39.507018 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:36:39.506933    5002 cpu_manager.go:414] "containerMap: removing container" podUID="89fd0c36-a17c-46a9-8a81-05f9717d48f3" containerName="cleanup"
Oct 30 01:36:39.507018 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:36:39.506942    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89fd0c36-a17c-46a9-8a81-05f9717d48f3" containerName="cleanup"
Oct 30 01:36:39.507018 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:36:39.506954    5002 cpu_manager.go:414] "containerMap: removing container" podUID="4d35acb0-477c-436e-bb24-7e15e827a5af" containerName="manager"
Oct 30 01:36:39.507018 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:36:39.506959    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d35acb0-477c-436e-bb24-7e15e827a5af" containerName="manager"
Oct 30 01:36:39.507253 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:36:39.507049    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="89fd0c36-a17c-46a9-8a81-05f9717d48f3" containerName="cleanup"
Oct 30 01:36:39.507253 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:36:39.507060    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="89fd0c36-a17c-46a9-8a81-05f9717d48f3" containerName="cleanup"
Oct 30 01:36:39.507253 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:36:39.507067    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="89fd0c36-a17c-46a9-8a81-05f9717d48f3" containerName="cleanup"
Oct 30 01:36:39.507253 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:36:39.507073    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d35acb0-477c-436e-bb24-7e15e827a5af" containerName="manager"
Oct 30 01:36:39.510075 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:36:39.510058    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-c8tcz"
Oct 30 01:36:39.541330 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:36:39.541299    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:36:39.541574 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:36:39.541547    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5phsh\""
Oct 30 01:36:39.541741 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:36:39.541612    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:36:39.541741 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:36:39.541653    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:36:39.543238 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:36:39.543215    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-c8tcz"]
Oct 30 01:36:39.549183 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:36:39.549162    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b9a563b-7e8d-4965-8372-7b5c6306905d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-c8tcz\" (UID: \"5b9a563b-7e8d-4965-8372-7b5c6306905d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-c8tcz"
Oct 30 01:36:39.549293 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:36:39.549205    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rk4pb\" (UniqueName: \"kubernetes.io/projected/5b9a563b-7e8d-4965-8372-7b5c6306905d-kube-api-access-rk4pb\") pod \"maas-api-revoke-sub-f03629db-c8tcz\" (UID: \"5b9a563b-7e8d-4965-8372-7b5c6306905d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-c8tcz"
Oct 30 01:36:39.549293 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:36:39.549234    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b9a563b-7e8d-4965-8372-7b5c6306905d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-c8tcz\" (UID: \"5b9a563b-7e8d-4965-8372-7b5c6306905d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-c8tcz"
Oct 30 01:36:39.650787 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:36:39.650733    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b9a563b-7e8d-4965-8372-7b5c6306905d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-c8tcz\" (UID: \"5b9a563b-7e8d-4965-8372-7b5c6306905d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-c8tcz"
Oct 30 01:36:39.651008 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:36:39.650822    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rk4pb\" (UniqueName: \"kubernetes.io/projected/5b9a563b-7e8d-4965-8372-7b5c6306905d-kube-api-access-rk4pb\") pod \"maas-api-revoke-sub-f03629db-c8tcz\" (UID: \"5b9a563b-7e8d-4965-8372-7b5c6306905d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-c8tcz"
Oct 30 01:36:39.651008 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:36:39.650851    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b9a563b-7e8d-4965-8372-7b5c6306905d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-c8tcz\" (UID: \"5b9a563b-7e8d-4965-8372-7b5c6306905d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-c8tcz"
Oct 30 01:36:39.651630 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:36:39.651606    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b9a563b-7e8d-4965-8372-7b5c6306905d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-c8tcz\" (UID: \"5b9a563b-7e8d-4965-8372-7b5c6306905d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-c8tcz"
Oct 30 01:36:39.653109 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:36:39.653089    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b9a563b-7e8d-4965-8372-7b5c6306905d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-c8tcz\" (UID: \"5b9a563b-7e8d-4965-8372-7b5c6306905d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-c8tcz"
Oct 30 01:36:39.676537 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:36:39.676507    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rk4pb\" (UniqueName: \"kubernetes.io/projected/5b9a563b-7e8d-4965-8372-7b5c6306905d-kube-api-access-rk4pb\") pod \"maas-api-revoke-sub-f03629db-c8tcz\" (UID: \"5b9a563b-7e8d-4965-8372-7b5c6306905d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-c8tcz"
Oct 30 01:36:39.822666 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:36:39.822573    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-c8tcz"
Oct 30 01:36:40.118757 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:36:40.118727    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-c8tcz"]
Oct 30 01:36:40.120017 ip-10-0-128-7 kubenswrapper[5002]: W1030 01:36:40.119989    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b9a563b_7e8d_4965_8372_7b5c6306905d.slice/crio-37aba4e5bd95bf696c77d47ad073af681194aa4479f01ba17deed0c8ff9b504d WatchSource:0}: Error finding container 37aba4e5bd95bf696c77d47ad073af681194aa4479f01ba17deed0c8ff9b504d: Status 404 returned error can't find the container with id 37aba4e5bd95bf696c77d47ad073af681194aa4479f01ba17deed0c8ff9b504d
Oct 30 01:36:40.774419 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:36:40.774308    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-c8tcz" event={"ID":"5b9a563b-7e8d-4965-8372-7b5c6306905d","Type":"ContainerStarted","Data":"04b2aff9eb620e6e6c32100b130a84451fe8fc645ea3459bec10a42a0e37a506"}
Oct 30 01:36:40.774419 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:36:40.774365    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-c8tcz" event={"ID":"5b9a563b-7e8d-4965-8372-7b5c6306905d","Type":"ContainerStarted","Data":"37aba4e5bd95bf696c77d47ad073af681194aa4479f01ba17deed0c8ff9b504d"}
Oct 30 01:36:40.821499 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:36:40.821443    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-c8tcz" podStartSLOduration=1.821429582 podStartE2EDuration="1.821429582s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:36:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:36:40.820987194 +0000 UTC m=+3896.437420072" watchObservedRunningTime="2026-10-30 01:36:40.821429582 +0000 UTC m=+3896.437862457"
Oct 30 01:36:41.780262 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:36:41.780226    5002 generic.go:356] "Generic (PLEG): container finished" podID="5b9a563b-7e8d-4965-8372-7b5c6306905d" containerID="04b2aff9eb620e6e6c32100b130a84451fe8fc645ea3459bec10a42a0e37a506" exitCode=0
Oct 30 01:36:41.780728 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:36:41.780316    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-c8tcz" event={"ID":"5b9a563b-7e8d-4965-8372-7b5c6306905d","Type":"ContainerDied","Data":"04b2aff9eb620e6e6c32100b130a84451fe8fc645ea3459bec10a42a0e37a506"}
Oct 30 01:36:42.862785 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:36:42.862742    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbjdq" containerName="extractor"
Oct 30 01:36:42.922830 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:36:42.922806    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-c8tcz"
Oct 30 01:36:42.983334 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:36:42.983305    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b9a563b-7e8d-4965-8372-7b5c6306905d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b9a563b-7e8d-4965-8372-7b5c6306905d-maas-api-openshift-ca\") pod \"5b9a563b-7e8d-4965-8372-7b5c6306905d\" (UID: \"5b9a563b-7e8d-4965-8372-7b5c6306905d\") "
Oct 30 01:36:42.983533 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:36:42.983408    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b9a563b-7e8d-4965-8372-7b5c6306905d-kube-api-access-rk4pb\" (UniqueName: \"kubernetes.io/projected/5b9a563b-7e8d-4965-8372-7b5c6306905d-kube-api-access-rk4pb\") pod \"5b9a563b-7e8d-4965-8372-7b5c6306905d\" (UID: \"5b9a563b-7e8d-4965-8372-7b5c6306905d\") "
Oct 30 01:36:42.983533 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:36:42.983445    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5b9a563b-7e8d-4965-8372-7b5c6306905d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b9a563b-7e8d-4965-8372-7b5c6306905d-maas-api-xks-ca\") pod \"5b9a563b-7e8d-4965-8372-7b5c6306905d\" (UID: \"5b9a563b-7e8d-4965-8372-7b5c6306905d\") "
Oct 30 01:36:42.983760 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:36:42.983734    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b9a563b-7e8d-4965-8372-7b5c6306905d-maas-api-openshift-ca" pod "5b9a563b-7e8d-4965-8372-7b5c6306905d" (UID: "5b9a563b-7e8d-4965-8372-7b5c6306905d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:36:42.985541 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:36:42.985513    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b9a563b-7e8d-4965-8372-7b5c6306905d-maas-api-xks-ca" pod "5b9a563b-7e8d-4965-8372-7b5c6306905d" (UID: "5b9a563b-7e8d-4965-8372-7b5c6306905d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:36:42.985670 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:36:42.985571    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b9a563b-7e8d-4965-8372-7b5c6306905d-kube-api-access-rk4pb" pod "5b9a563b-7e8d-4965-8372-7b5c6306905d" (UID: "5b9a563b-7e8d-4965-8372-7b5c6306905d"). InnerVolumeSpecName "kube-api-access-rk4pb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:36:43.085028 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:36:43.084946    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b9a563b-7e8d-4965-8372-7b5c6306905d-maas-api-openshift-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:36:43.085028 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:36:43.084978    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rk4pb\" (UniqueName: \"kubernetes.io/projected/5b9a563b-7e8d-4965-8372-7b5c6306905d-kube-api-access-rk4pb\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:36:43.085028 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:36:43.084988    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b9a563b-7e8d-4965-8372-7b5c6306905d-maas-api-xks-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:36:43.790775 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:36:43.790730    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-c8tcz" event={"ID":"5b9a563b-7e8d-4965-8372-7b5c6306905d","Type":"ContainerDied","Data":"37aba4e5bd95bf696c77d47ad073af681194aa4479f01ba17deed0c8ff9b504d"}
Oct 30 01:36:43.790775 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:36:43.790757    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-c8tcz"
Oct 30 01:36:43.791000 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:36:43.790788    5002 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="37aba4e5bd95bf696c77d47ad073af681194aa4479f01ba17deed0c8ff9b504d"
Oct 30 01:36:45.339688 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:36:45.339659    5002 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-g7c2w_5dc56e7e-f3c1-43eb-a21b-8be6bb4706cf/console-operator/1.log"
Oct 30 01:36:45.371474 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:36:45.371435    5002 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-g7c2w_5dc56e7e-f3c1-43eb-a21b-8be6bb4706cf/console-operator/1.log"
Oct 30 01:36:47.862334 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:36:47.862294    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7d8z4"]
Oct 30 01:36:51.185848 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:36:51.185807    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-gpfw7"]
Oct 30 01:36:51.186450 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:36:51.186426    5002 cpu_manager.go:414] "containerMap: removing container" podUID="5b9a563b-7e8d-4965-8372-7b5c6306905d" containerName="revoke-keys"
Oct 30 01:36:51.186521 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:36:51.186453    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b9a563b-7e8d-4965-8372-7b5c6306905d" containerName="revoke-keys"
Oct 30 01:36:51.186521 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:36:51.186479    5002 cpu_manager.go:414] "containerMap: removing container" podUID="89fd0c36-a17c-46a9-8a81-05f9717d48f3" containerName="cleanup"
Oct 30 01:36:51.186521 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:36:51.186487    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89fd0c36-a17c-46a9-8a81-05f9717d48f3" containerName="cleanup"
Oct 30 01:36:51.186694 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:36:51.186612    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b9a563b-7e8d-4965-8372-7b5c6306905d" containerName="revoke-keys"
Oct 30 01:36:51.190126 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:36:51.190100    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-gpfw7"
Oct 30 01:36:51.211750 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:36:51.211712    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-gpfw7"]
Oct 30 01:36:51.221606 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:36:51.221558    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:36:51.221861 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:36:51.221808    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:36:51.221861 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:36:51.221629    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:36:51.221861 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:36:51.221722    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5phsh\""
Oct 30 01:36:51.259731 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:36:51.259691    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfa9a094-01c2-4c82-aeb9-37389bd63c9d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-gpfw7\" (UID: \"cfa9a094-01c2-4c82-aeb9-37389bd63c9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-gpfw7"
Oct 30 01:36:51.259960 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:36:51.259765    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfa9a094-01c2-4c82-aeb9-37389bd63c9d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-gpfw7\" (UID: \"cfa9a094-01c2-4c82-aeb9-37389bd63c9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-gpfw7"
Oct 30 01:36:51.259960 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:36:51.259908    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4dlv\" (UniqueName: \"kubernetes.io/projected/cfa9a094-01c2-4c82-aeb9-37389bd63c9d-kube-api-access-r4dlv\") pod \"maas-api-revoke-sub-0165d7d2-gpfw7\" (UID: \"cfa9a094-01c2-4c82-aeb9-37389bd63c9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-gpfw7"
Oct 30 01:36:51.360799 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:36:51.360748    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4dlv\" (UniqueName: \"kubernetes.io/projected/cfa9a094-01c2-4c82-aeb9-37389bd63c9d-kube-api-access-r4dlv\") pod \"maas-api-revoke-sub-0165d7d2-gpfw7\" (UID: \"cfa9a094-01c2-4c82-aeb9-37389bd63c9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-gpfw7"
Oct 30 01:36:51.361019 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:36:51.360880    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfa9a094-01c2-4c82-aeb9-37389bd63c9d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-gpfw7\" (UID: \"cfa9a094-01c2-4c82-aeb9-37389bd63c9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-gpfw7"
Oct 30 01:36:51.361019 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:36:51.360919    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfa9a094-01c2-4c82-aeb9-37389bd63c9d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-gpfw7\" (UID: \"cfa9a094-01c2-4c82-aeb9-37389bd63c9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-gpfw7"
Oct 30 01:36:51.361735 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:36:51.361702    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfa9a094-01c2-4c82-aeb9-37389bd63c9d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-gpfw7\" (UID: \"cfa9a094-01c2-4c82-aeb9-37389bd63c9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-gpfw7"
Oct 30 01:36:51.363587 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:36:51.363553    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfa9a094-01c2-4c82-aeb9-37389bd63c9d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-gpfw7\" (UID: \"cfa9a094-01c2-4c82-aeb9-37389bd63c9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-gpfw7"
Oct 30 01:36:51.386151 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:36:51.386119    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4dlv\" (UniqueName: \"kubernetes.io/projected/cfa9a094-01c2-4c82-aeb9-37389bd63c9d-kube-api-access-r4dlv\") pod \"maas-api-revoke-sub-0165d7d2-gpfw7\" (UID: \"cfa9a094-01c2-4c82-aeb9-37389bd63c9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-gpfw7"
Oct 30 01:36:51.520643 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:36:51.520547    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-gpfw7"
Oct 30 01:36:51.879455 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:36:51.872404    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-gpfw7"]
Oct 30 01:36:51.924057 ip-10-0-128-7 kubenswrapper[5002]: W1030 01:36:51.924019    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfa9a094_01c2_4c82_aeb9_37389bd63c9d.slice/crio-607cf9502fe6ff24365d7a6271b6f42c431f09c6f4f48cfda3fd5a585bc697b2 WatchSource:0}: Error finding container 607cf9502fe6ff24365d7a6271b6f42c431f09c6f4f48cfda3fd5a585bc697b2: Status 404 returned error can't find the container with id 607cf9502fe6ff24365d7a6271b6f42c431f09c6f4f48cfda3fd5a585bc697b2
Oct 30 01:36:52.838033 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:36:52.837993    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-gpfw7" event={"ID":"cfa9a094-01c2-4c82-aeb9-37389bd63c9d","Type":"ContainerStarted","Data":"1c99da8da9ed1a2070ac26ab9aba7fcfbcf96a27b7d8c38a7502bd8655917b88"}
Oct 30 01:36:52.838554 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:36:52.838049    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-gpfw7" event={"ID":"cfa9a094-01c2-4c82-aeb9-37389bd63c9d","Type":"ContainerStarted","Data":"607cf9502fe6ff24365d7a6271b6f42c431f09c6f4f48cfda3fd5a585bc697b2"}
Oct 30 01:36:52.901404 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:36:52.900754    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-gpfw7" podStartSLOduration=1.900735235 podStartE2EDuration="1.900735235s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:36:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:36:52.898904673 +0000 UTC m=+3908.515337553" watchObservedRunningTime="2026-10-30 01:36:52.900735235 +0000 UTC m=+3908.517168115"
Oct 30 01:36:53.844475 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:36:53.844437    5002 generic.go:356] "Generic (PLEG): container finished" podID="cfa9a094-01c2-4c82-aeb9-37389bd63c9d" containerID="1c99da8da9ed1a2070ac26ab9aba7fcfbcf96a27b7d8c38a7502bd8655917b88" exitCode=0
Oct 30 01:36:53.844955 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:36:53.844523    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-gpfw7" event={"ID":"cfa9a094-01c2-4c82-aeb9-37389bd63c9d","Type":"ContainerDied","Data":"1c99da8da9ed1a2070ac26ab9aba7fcfbcf96a27b7d8c38a7502bd8655917b88"}
Oct 30 01:36:54.988949 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:36:54.988922    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-gpfw7"
Oct 30 01:36:55.148039 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:36:55.147947    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cfa9a094-01c2-4c82-aeb9-37389bd63c9d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfa9a094-01c2-4c82-aeb9-37389bd63c9d-maas-api-xks-ca\") pod \"cfa9a094-01c2-4c82-aeb9-37389bd63c9d\" (UID: \"cfa9a094-01c2-4c82-aeb9-37389bd63c9d\") "
Oct 30 01:36:55.148039 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:36:55.147993    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cfa9a094-01c2-4c82-aeb9-37389bd63c9d-kube-api-access-r4dlv\" (UniqueName: \"kubernetes.io/projected/cfa9a094-01c2-4c82-aeb9-37389bd63c9d-kube-api-access-r4dlv\") pod \"cfa9a094-01c2-4c82-aeb9-37389bd63c9d\" (UID: \"cfa9a094-01c2-4c82-aeb9-37389bd63c9d\") "
Oct 30 01:36:55.148238 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:36:55.148070    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cfa9a094-01c2-4c82-aeb9-37389bd63c9d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfa9a094-01c2-4c82-aeb9-37389bd63c9d-maas-api-openshift-ca\") pod \"cfa9a094-01c2-4c82-aeb9-37389bd63c9d\" (UID: \"cfa9a094-01c2-4c82-aeb9-37389bd63c9d\") "
Oct 30 01:36:55.148502 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:36:55.148478    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfa9a094-01c2-4c82-aeb9-37389bd63c9d-maas-api-openshift-ca" pod "cfa9a094-01c2-4c82-aeb9-37389bd63c9d" (UID: "cfa9a094-01c2-4c82-aeb9-37389bd63c9d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:36:55.150240 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:36:55.150206    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfa9a094-01c2-4c82-aeb9-37389bd63c9d-kube-api-access-r4dlv" pod "cfa9a094-01c2-4c82-aeb9-37389bd63c9d" (UID: "cfa9a094-01c2-4c82-aeb9-37389bd63c9d"). InnerVolumeSpecName "kube-api-access-r4dlv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:36:55.150240 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:36:55.150214    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cfa9a094-01c2-4c82-aeb9-37389bd63c9d-maas-api-xks-ca" pod "cfa9a094-01c2-4c82-aeb9-37389bd63c9d" (UID: "cfa9a094-01c2-4c82-aeb9-37389bd63c9d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:36:55.249249 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:36:55.249197    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfa9a094-01c2-4c82-aeb9-37389bd63c9d-maas-api-xks-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:36:55.249249 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:36:55.249245    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r4dlv\" (UniqueName: \"kubernetes.io/projected/cfa9a094-01c2-4c82-aeb9-37389bd63c9d-kube-api-access-r4dlv\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:36:55.249249 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:36:55.249258    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfa9a094-01c2-4c82-aeb9-37389bd63c9d-maas-api-openshift-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:36:55.855286 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:36:55.855202    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-gpfw7" event={"ID":"cfa9a094-01c2-4c82-aeb9-37389bd63c9d","Type":"ContainerDied","Data":"607cf9502fe6ff24365d7a6271b6f42c431f09c6f4f48cfda3fd5a585bc697b2"}
Oct 30 01:36:55.855286 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:36:55.855232    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-gpfw7"
Oct 30 01:36:55.855286 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:36:55.855257    5002 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="607cf9502fe6ff24365d7a6271b6f42c431f09c6f4f48cfda3fd5a585bc697b2"
Oct 30 01:37:02.079162 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:37:02.079117    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xbgcp"]
Oct 30 01:37:02.079730 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:37:02.079554    5002 cpu_manager.go:414] "containerMap: removing container" podUID="cfa9a094-01c2-4c82-aeb9-37389bd63c9d" containerName="revoke-keys"
Oct 30 01:37:02.079730 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:37:02.079566    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfa9a094-01c2-4c82-aeb9-37389bd63c9d" containerName="revoke-keys"
Oct 30 01:37:02.079730 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:37:02.079638    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="cfa9a094-01c2-4c82-aeb9-37389bd63c9d" containerName="revoke-keys"
Oct 30 01:37:02.082939 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:37:02.082917    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xbgcp"
Oct 30 01:37:02.113636 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:37:02.113599    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:37:02.113636 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:37:02.113614    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:37:02.113898 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:37:02.113771    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5phsh\""
Oct 30 01:37:02.114011 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:37:02.113994    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:37:02.116650 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:37:02.116619    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xbgcp"]
Oct 30 01:37:02.208664 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:37:02.208627    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5bafadaf-051a-49ca-a0f4-9f3609fcb1b9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-xbgcp\" (UID: \"5bafadaf-051a-49ca-a0f4-9f3609fcb1b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xbgcp"
Oct 30 01:37:02.208840 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:37:02.208692    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5bafadaf-051a-49ca-a0f4-9f3609fcb1b9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-xbgcp\" (UID: \"5bafadaf-051a-49ca-a0f4-9f3609fcb1b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xbgcp"
Oct 30 01:37:02.208840 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:37:02.208789    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqjgp\" (UniqueName: \"kubernetes.io/projected/5bafadaf-051a-49ca-a0f4-9f3609fcb1b9-kube-api-access-nqjgp\") pod \"maas-api-revoke-sub-6b19338f-xbgcp\" (UID: \"5bafadaf-051a-49ca-a0f4-9f3609fcb1b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xbgcp"
Oct 30 01:37:02.309374 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:37:02.309334    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqjgp\" (UniqueName: \"kubernetes.io/projected/5bafadaf-051a-49ca-a0f4-9f3609fcb1b9-kube-api-access-nqjgp\") pod \"maas-api-revoke-sub-6b19338f-xbgcp\" (UID: \"5bafadaf-051a-49ca-a0f4-9f3609fcb1b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xbgcp"
Oct 30 01:37:02.309570 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:37:02.309429    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5bafadaf-051a-49ca-a0f4-9f3609fcb1b9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-xbgcp\" (UID: \"5bafadaf-051a-49ca-a0f4-9f3609fcb1b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xbgcp"
Oct 30 01:37:02.309570 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:37:02.309486    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5bafadaf-051a-49ca-a0f4-9f3609fcb1b9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-xbgcp\" (UID: \"5bafadaf-051a-49ca-a0f4-9f3609fcb1b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xbgcp"
Oct 30 01:37:02.310327 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:37:02.310299    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5bafadaf-051a-49ca-a0f4-9f3609fcb1b9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-xbgcp\" (UID: \"5bafadaf-051a-49ca-a0f4-9f3609fcb1b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xbgcp"
Oct 30 01:37:02.311948 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:37:02.311924    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5bafadaf-051a-49ca-a0f4-9f3609fcb1b9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-xbgcp\" (UID: \"5bafadaf-051a-49ca-a0f4-9f3609fcb1b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xbgcp"
Oct 30 01:37:02.334599 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:37:02.334538    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqjgp\" (UniqueName: \"kubernetes.io/projected/5bafadaf-051a-49ca-a0f4-9f3609fcb1b9-kube-api-access-nqjgp\") pod \"maas-api-revoke-sub-6b19338f-xbgcp\" (UID: \"5bafadaf-051a-49ca-a0f4-9f3609fcb1b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xbgcp"
Oct 30 01:37:02.396508 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:37:02.396482    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xbgcp"
Oct 30 01:37:02.700689 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:37:02.700663    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xbgcp"]
Oct 30 01:37:02.702826 ip-10-0-128-7 kubenswrapper[5002]: W1030 01:37:02.702785    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bafadaf_051a_49ca_a0f4_9f3609fcb1b9.slice/crio-528b3176024c92ee26bb496a06c24a7166f79e821cadaef8f314bd90b3482255 WatchSource:0}: Error finding container 528b3176024c92ee26bb496a06c24a7166f79e821cadaef8f314bd90b3482255: Status 404 returned error can't find the container with id 528b3176024c92ee26bb496a06c24a7166f79e821cadaef8f314bd90b3482255
Oct 30 01:37:02.896349 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:37:02.896302    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xbgcp" event={"ID":"5bafadaf-051a-49ca-a0f4-9f3609fcb1b9","Type":"ContainerStarted","Data":"528b3176024c92ee26bb496a06c24a7166f79e821cadaef8f314bd90b3482255"}
Oct 30 01:37:03.902732 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:37:03.902638    5002 generic.go:356] "Generic (PLEG): container finished" podID="5bafadaf-051a-49ca-a0f4-9f3609fcb1b9" containerID="cc27c8ea878ad311f6e96f34f0fc6e9dca23a851297c11653777231d1a44209a" exitCode=0
Oct 30 01:37:03.903110 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:37:03.902757    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xbgcp" event={"ID":"5bafadaf-051a-49ca-a0f4-9f3609fcb1b9","Type":"ContainerDied","Data":"cc27c8ea878ad311f6e96f34f0fc6e9dca23a851297c11653777231d1a44209a"}
Oct 30 01:37:05.048015 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:37:05.047985    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xbgcp"
Oct 30 01:37:05.128873 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:37:05.128839    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5bafadaf-051a-49ca-a0f4-9f3609fcb1b9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5bafadaf-051a-49ca-a0f4-9f3609fcb1b9-maas-api-xks-ca\") pod \"5bafadaf-051a-49ca-a0f4-9f3609fcb1b9\" (UID: \"5bafadaf-051a-49ca-a0f4-9f3609fcb1b9\") "
Oct 30 01:37:05.129043 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:37:05.128882    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5bafadaf-051a-49ca-a0f4-9f3609fcb1b9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5bafadaf-051a-49ca-a0f4-9f3609fcb1b9-maas-api-openshift-ca\") pod \"5bafadaf-051a-49ca-a0f4-9f3609fcb1b9\" (UID: \"5bafadaf-051a-49ca-a0f4-9f3609fcb1b9\") "
Oct 30 01:37:05.129043 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:37:05.128959    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5bafadaf-051a-49ca-a0f4-9f3609fcb1b9-kube-api-access-nqjgp\" (UniqueName: \"kubernetes.io/projected/5bafadaf-051a-49ca-a0f4-9f3609fcb1b9-kube-api-access-nqjgp\") pod \"5bafadaf-051a-49ca-a0f4-9f3609fcb1b9\" (UID: \"5bafadaf-051a-49ca-a0f4-9f3609fcb1b9\") "
Oct 30 01:37:05.129319 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:37:05.129293    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5bafadaf-051a-49ca-a0f4-9f3609fcb1b9-maas-api-openshift-ca" pod "5bafadaf-051a-49ca-a0f4-9f3609fcb1b9" (UID: "5bafadaf-051a-49ca-a0f4-9f3609fcb1b9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:37:05.131063 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:37:05.131034    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bafadaf-051a-49ca-a0f4-9f3609fcb1b9-kube-api-access-nqjgp" pod "5bafadaf-051a-49ca-a0f4-9f3609fcb1b9" (UID: "5bafadaf-051a-49ca-a0f4-9f3609fcb1b9"). InnerVolumeSpecName "kube-api-access-nqjgp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:37:05.131155 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:37:05.131088    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5bafadaf-051a-49ca-a0f4-9f3609fcb1b9-maas-api-xks-ca" pod "5bafadaf-051a-49ca-a0f4-9f3609fcb1b9" (UID: "5bafadaf-051a-49ca-a0f4-9f3609fcb1b9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:37:05.230254 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:37:05.230214    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5bafadaf-051a-49ca-a0f4-9f3609fcb1b9-maas-api-xks-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:37:05.230254 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:37:05.230249    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5bafadaf-051a-49ca-a0f4-9f3609fcb1b9-maas-api-openshift-ca\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:37:05.230254 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:37:05.230259    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nqjgp\" (UniqueName: \"kubernetes.io/projected/5bafadaf-051a-49ca-a0f4-9f3609fcb1b9-kube-api-access-nqjgp\") on node \"ip-10-0-128-7.ec2.internal\" DevicePath \"\""
Oct 30 01:37:05.913836 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:37:05.913806    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xbgcp"
Oct 30 01:37:05.914021 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:37:05.913831    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xbgcp" event={"ID":"5bafadaf-051a-49ca-a0f4-9f3609fcb1b9","Type":"ContainerDied","Data":"528b3176024c92ee26bb496a06c24a7166f79e821cadaef8f314bd90b3482255"}
Oct 30 01:37:05.914021 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:37:05.913879    5002 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="528b3176024c92ee26bb496a06c24a7166f79e821cadaef8f314bd90b3482255"
Oct 30 01:37:12.860729 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:37:12.860688    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbjdq" containerName="extractor"
Oct 30 01:37:13.236079 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:37:13.236039    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7d8z4"]
Oct 30 01:37:14.941535 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:37:14.941498    5002 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-2jzrh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.25:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 01:37:14.942062 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:37:14.941600    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-2jzrh" podUID="49e9b6e9-fa30-4ea2-82e9-69e9e4ecab93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.25:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:37:23.984956 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:37:23.984925    5002 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-k2k4j_3f63d6f0-ed39-4ac0-90e7-74fd716b58f8/manager/0.log"
Oct 30 01:37:24.186927 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:37:24.186888    5002 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-5cfb77bf8c-lm8qn_a9373505-073b-4c82-92ee-ca6365ea95dc/manager/0.log"
Oct 30 01:37:24.396015 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:37:24.395929    5002 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-cgxg8_9c8859ce-8c25-4422-a91e-d9b56618cca8/manager/0.log"
Oct 30 01:37:24.895368 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:37:24.895339    5002 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-4gc22_19afed17-6147-4955-86b1-3440be3f780a/server/0.log"
Oct 30 01:37:25.293633 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:37:25.293596    5002 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-observability-584f99c788-h48gx_a2a28b50-085d-43ec-bcd1-22d8d5a94cf9/manager/0.log"
Oct 30 01:37:25.517287 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:37:25.517256    5002 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6bcdb85766-9j9s2_e7944126-c185-43db-b8ba-04cbefec39fb/copy-manifests/0.log"
Oct 30 01:37:25.548756 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:37:25.548684    5002 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6bcdb85766-9j9s2_e7944126-c185-43db-b8ba-04cbefec39fb/manager/0.log"
Oct 30 01:37:27.414613 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:37:27.414581    5002 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-gpfw7_cfa9a094-01c2-4c82-aeb9-37389bd63c9d/revoke-keys/0.log"
Oct 30 01:37:28.151564 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:37:28.151527    5002 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-xbgcp_5bafadaf-051a-49ca-a0f4-9f3609fcb1b9/revoke-keys/0.log"
Oct 30 01:37:28.877268 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:37:28.877229    5002 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-c8tcz_5b9a563b-7e8d-4965-8372-7b5c6306905d/revoke-keys/0.log"
Oct 30 01:37:31.175075 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:37:31.175040    5002 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-jrv82_d21d8c35-34a7-42ba-89a9-b9754b6b339b/kuadrant-console-plugin/0.log"
Oct 30 01:37:31.632636 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:37:31.632598    5002 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-7d8z4_661ea448-7d8a-40ac-951b-383599d5d61f/limitador/0.log"
Oct 30 01:37:32.761369 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:37:32.761327    5002 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-j5x7c_48f3cd2a-244d-4849-888b-e4b9abd0ce9c/discovery/0.log"
Oct 30 01:37:33.534074 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:37:33.534033    5002 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7d9897b54c-2jzrh_49e9b6e9-fa30-4ea2-82e9-69e9e4ecab93/istio-proxy/0.log"
Oct 30 01:37:34.329824 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:37:34.329789    5002 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7d9897b54c-hb9g9_9224697d-3f44-4128-82de-642f6cdda3e3/istio-proxy/0.log"
Oct 30 01:37:34.905981 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:37:34.905932    5002 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7d9897b54c-pdgh4_c341e49a-5e59-414d-a432-e338bef60081/istio-proxy/0.log"
Oct 30 01:37:35.674168 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:37:35.674139    5002 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-85f7b79cf7-h2tpp_a3d234b3-76d4-47d5-98a4-dafd83477b17/router/0.log"
Oct 30 01:37:42.860806 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:37:42.860770    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbjdq" containerName="extractor"
Oct 30 01:37:48.224919 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:37:48.224887    5002 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-k2k4j_3f63d6f0-ed39-4ac0-90e7-74fd716b58f8/manager/0.log"
Oct 30 01:37:48.449017 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:37:48.448985    5002 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-5cfb77bf8c-lm8qn_a9373505-073b-4c82-92ee-ca6365ea95dc/manager/0.log"
Oct 30 01:37:48.702396 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:37:48.702349    5002 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-cgxg8_9c8859ce-8c25-4422-a91e-d9b56618cca8/manager/0.log"
Oct 30 01:37:49.161047 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:37:49.160971    5002 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-4gc22_19afed17-6147-4955-86b1-3440be3f780a/server/0.log"
Oct 30 01:37:49.632024 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:37:49.631991    5002 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-observability-584f99c788-h48gx_a2a28b50-085d-43ec-bcd1-22d8d5a94cf9/manager/0.log"
Oct 30 01:37:49.886157 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:37:49.886078    5002 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6bcdb85766-9j9s2_e7944126-c185-43db-b8ba-04cbefec39fb/copy-manifests/0.log"
Oct 30 01:37:49.919261 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:37:49.919228    5002 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6bcdb85766-9j9s2_e7944126-c185-43db-b8ba-04cbefec39fb/manager/0.log"
Oct 30 01:37:51.653259 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:37:51.653229    5002 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-gpfw7_cfa9a094-01c2-4c82-aeb9-37389bd63c9d/revoke-keys/0.log"
Oct 30 01:37:52.430051 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:37:52.430026    5002 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-xbgcp_5bafadaf-051a-49ca-a0f4-9f3609fcb1b9/revoke-keys/0.log"
Oct 30 01:37:53.236656 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:37:53.236626    5002 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-c8tcz_5b9a563b-7e8d-4965-8372-7b5c6306905d/revoke-keys/0.log"
Oct 30 01:37:55.316884 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:37:55.316852    5002 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-jrv82_d21d8c35-34a7-42ba-89a9-b9754b6b339b/kuadrant-console-plugin/0.log"
Oct 30 01:37:55.788848 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:37:55.788820    5002 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-7d8z4_661ea448-7d8a-40ac-951b-383599d5d61f/limitador/0.log"
Oct 30 01:37:56.755932 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:37:56.755899    5002 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-j5x7c_48f3cd2a-244d-4849-888b-e4b9abd0ce9c/discovery/0.log"
Oct 30 01:37:57.530604 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:37:57.530571    5002 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7d9897b54c-2jzrh_49e9b6e9-fa30-4ea2-82e9-69e9e4ecab93/istio-proxy/0.log"
Oct 30 01:37:58.317645 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:37:58.317609    5002 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7d9897b54c-hb9g9_9224697d-3f44-4128-82de-642f6cdda3e3/istio-proxy/0.log"
Oct 30 01:37:58.910463 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:37:58.910428    5002 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7d9897b54c-pdgh4_c341e49a-5e59-414d-a432-e338bef60081/istio-proxy/0.log"
Oct 30 01:37:59.668750 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:37:59.668718    5002 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-85f7b79cf7-h2tpp_a3d234b3-76d4-47d5-98a4-dafd83477b17/router/0.log"
Oct 30 01:38:12.860647 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:38:12.860609    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbjdq" containerName="extractor"
Oct 30 01:38:42.860652 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:38:42.860558    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbjdq" containerName="extractor"
Oct 30 01:39:12.861240 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:39:12.861192    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbjdq" containerName="extractor"
Oct 30 01:39:23.674725 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:39:23.674693    5002 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-tz6g8_80603987-09a1-4ca1-b8b4-3c9ac8d0efc6/global-pull-secret-syncer/0.log"
Oct 30 01:39:23.860017 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:39:23.859990    5002 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-cqjlz_9822c484-fd06-4e5e-94b4-ef0f5ade9a13/connection-checker/0.log"
Oct 30 01:39:23.983278 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:39:23.983252    5002 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-f2b4d_139583a1-f622-49ab-8c1e-af168fcdf6c4/connection-checker/0.log"
Oct 30 01:39:24.111514 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:39:24.111477    5002 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-fbdvl_783c7a98-5ebf-44dd-8da9-797e71c28169/connection-checker/0.log"
Oct 30 01:39:24.618455 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:39:24.618419    5002 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-m5qgr_0d1b347f-c22e-489b-986d-d3b3e01fa36b/konnectivity-agent/0.log"
Oct 30 01:39:24.735355 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:39:24.735322    5002 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-128-7.ec2.internal_74c81293f9236d86a7e367354c3aa34d/haproxy/0.log"
Oct 30 01:39:31.523144 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:39:31.523115    5002 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-jrv82_d21d8c35-34a7-42ba-89a9-b9754b6b339b/kuadrant-console-plugin/0.log"
Oct 30 01:39:31.835213 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:39:31.835127    5002 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-7d8z4_661ea448-7d8a-40ac-951b-383599d5d61f/limitador/0.log"
Oct 30 01:39:33.382753 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:39:33.382715    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-dhfxw/perf-node-gather-daemonset-t2h8w"]
Oct 30 01:39:33.383155 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:39:33.383109    5002 cpu_manager.go:414] "containerMap: removing container" podUID="5bafadaf-051a-49ca-a0f4-9f3609fcb1b9" containerName="revoke-keys"
Oct 30 01:39:33.383155 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:39:33.383121    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5bafadaf-051a-49ca-a0f4-9f3609fcb1b9" containerName="revoke-keys"
Oct 30 01:39:33.383231 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:39:33.383196    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="5bafadaf-051a-49ca-a0f4-9f3609fcb1b9" containerName="revoke-keys"
Oct 30 01:39:33.386344 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:39:33.386326    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-dhfxw/perf-node-gather-daemonset-t2h8w"
Oct 30 01:39:33.405249 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:39:33.405212    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-dhfxw/perf-node-gather-daemonset-t2h8w"]
Oct 30 01:39:33.418975 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:39:33.418942    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5654e6d2-2e23-4398-b820-21340ddcdeb8-sys\") pod \"perf-node-gather-daemonset-t2h8w\" (UID: \"5654e6d2-2e23-4398-b820-21340ddcdeb8\") " pod="openshift-must-gather-dhfxw/perf-node-gather-daemonset-t2h8w"
Oct 30 01:39:33.419131 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:39:33.418988    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sj28h\" (UniqueName: \"kubernetes.io/projected/5654e6d2-2e23-4398-b820-21340ddcdeb8-kube-api-access-sj28h\") pod \"perf-node-gather-daemonset-t2h8w\" (UID: \"5654e6d2-2e23-4398-b820-21340ddcdeb8\") " pod="openshift-must-gather-dhfxw/perf-node-gather-daemonset-t2h8w"
Oct 30 01:39:33.419131 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:39:33.419093    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5654e6d2-2e23-4398-b820-21340ddcdeb8-proc\") pod \"perf-node-gather-daemonset-t2h8w\" (UID: \"5654e6d2-2e23-4398-b820-21340ddcdeb8\") " pod="openshift-must-gather-dhfxw/perf-node-gather-daemonset-t2h8w"
Oct 30 01:39:33.419269 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:39:33.419196    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5654e6d2-2e23-4398-b820-21340ddcdeb8-lib-modules\") pod \"perf-node-gather-daemonset-t2h8w\" (UID: \"5654e6d2-2e23-4398-b820-21340ddcdeb8\") " pod="openshift-must-gather-dhfxw/perf-node-gather-daemonset-t2h8w"
Oct 30 01:39:33.419269 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:39:33.419234    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5654e6d2-2e23-4398-b820-21340ddcdeb8-podres\") pod \"perf-node-gather-daemonset-t2h8w\" (UID: \"5654e6d2-2e23-4398-b820-21340ddcdeb8\") " pod="openshift-must-gather-dhfxw/perf-node-gather-daemonset-t2h8w"
Oct 30 01:39:33.493124 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:39:33.493092    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-dhfxw\"/\"default-dockercfg-tt658\""
Oct 30 01:39:33.495454 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:39:33.495424    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-dhfxw\"/\"openshift-service-ca.crt\""
Oct 30 01:39:33.495454 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:39:33.495444    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-dhfxw\"/\"kube-root-ca.crt\""
Oct 30 01:39:33.520050 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:39:33.520018    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5654e6d2-2e23-4398-b820-21340ddcdeb8-proc\") pod \"perf-node-gather-daemonset-t2h8w\" (UID: \"5654e6d2-2e23-4398-b820-21340ddcdeb8\") " pod="openshift-must-gather-dhfxw/perf-node-gather-daemonset-t2h8w"
Oct 30 01:39:33.520187 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:39:33.520076    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5654e6d2-2e23-4398-b820-21340ddcdeb8-lib-modules\") pod \"perf-node-gather-daemonset-t2h8w\" (UID: \"5654e6d2-2e23-4398-b820-21340ddcdeb8\") " pod="openshift-must-gather-dhfxw/perf-node-gather-daemonset-t2h8w"
Oct 30 01:39:33.520187 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:39:33.520101    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5654e6d2-2e23-4398-b820-21340ddcdeb8-podres\") pod \"perf-node-gather-daemonset-t2h8w\" (UID: \"5654e6d2-2e23-4398-b820-21340ddcdeb8\") " pod="openshift-must-gather-dhfxw/perf-node-gather-daemonset-t2h8w"
Oct 30 01:39:33.520187 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:39:33.520135    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5654e6d2-2e23-4398-b820-21340ddcdeb8-sys\") pod \"perf-node-gather-daemonset-t2h8w\" (UID: \"5654e6d2-2e23-4398-b820-21340ddcdeb8\") " pod="openshift-must-gather-dhfxw/perf-node-gather-daemonset-t2h8w"
Oct 30 01:39:33.520187 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:39:33.520150    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5654e6d2-2e23-4398-b820-21340ddcdeb8-lib-modules\") pod \"perf-node-gather-daemonset-t2h8w\" (UID: \"5654e6d2-2e23-4398-b820-21340ddcdeb8\") " pod="openshift-must-gather-dhfxw/perf-node-gather-daemonset-t2h8w"
Oct 30 01:39:33.520369 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:39:33.520150    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5654e6d2-2e23-4398-b820-21340ddcdeb8-proc\") pod \"perf-node-gather-daemonset-t2h8w\" (UID: \"5654e6d2-2e23-4398-b820-21340ddcdeb8\") " pod="openshift-must-gather-dhfxw/perf-node-gather-daemonset-t2h8w"
Oct 30 01:39:33.520369 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:39:33.520156    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sj28h\" (UniqueName: \"kubernetes.io/projected/5654e6d2-2e23-4398-b820-21340ddcdeb8-kube-api-access-sj28h\") pod \"perf-node-gather-daemonset-t2h8w\" (UID: \"5654e6d2-2e23-4398-b820-21340ddcdeb8\") " pod="openshift-must-gather-dhfxw/perf-node-gather-daemonset-t2h8w"
Oct 30 01:39:33.520369 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:39:33.520217    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5654e6d2-2e23-4398-b820-21340ddcdeb8-sys\") pod \"perf-node-gather-daemonset-t2h8w\" (UID: \"5654e6d2-2e23-4398-b820-21340ddcdeb8\") " pod="openshift-must-gather-dhfxw/perf-node-gather-daemonset-t2h8w"
Oct 30 01:39:33.520369 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:39:33.520241    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5654e6d2-2e23-4398-b820-21340ddcdeb8-podres\") pod \"perf-node-gather-daemonset-t2h8w\" (UID: \"5654e6d2-2e23-4398-b820-21340ddcdeb8\") " pod="openshift-must-gather-dhfxw/perf-node-gather-daemonset-t2h8w"
Oct 30 01:39:33.545514 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:39:33.545491    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sj28h\" (UniqueName: \"kubernetes.io/projected/5654e6d2-2e23-4398-b820-21340ddcdeb8-kube-api-access-sj28h\") pod \"perf-node-gather-daemonset-t2h8w\" (UID: \"5654e6d2-2e23-4398-b820-21340ddcdeb8\") " pod="openshift-must-gather-dhfxw/perf-node-gather-daemonset-t2h8w"
Oct 30 01:39:33.698739 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:39:33.698704    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-dhfxw/perf-node-gather-daemonset-t2h8w"
Oct 30 01:39:33.997755 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:39:33.997725    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-dhfxw/perf-node-gather-daemonset-t2h8w"]
Oct 30 01:39:33.998886 ip-10-0-128-7 kubenswrapper[5002]: W1030 01:39:33.998859    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod5654e6d2_2e23_4398_b820_21340ddcdeb8.slice/crio-21f5fec4741ff551caef2862778b12dc76720be062bfa990c8709903f606bc3e WatchSource:0}: Error finding container 21f5fec4741ff551caef2862778b12dc76720be062bfa990c8709903f606bc3e: Status 404 returned error can't find the container with id 21f5fec4741ff551caef2862778b12dc76720be062bfa990c8709903f606bc3e
Oct 30 01:39:34.525844 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:39:34.525803    5002 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-884576c57-kjsj6_9aba3099-fe4c-4846-9a34-25db2c2fe593/cluster-monitoring-operator/0.log"
Oct 30 01:39:34.600409 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:39:34.600350    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-dhfxw/perf-node-gather-daemonset-t2h8w" event={"ID":"5654e6d2-2e23-4398-b820-21340ddcdeb8","Type":"ContainerStarted","Data":"1d4419536caca6546345a3c6aa693e122e0c664bd155b9128e4fffabe9bcbbbe"}
Oct 30 01:39:34.600607 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:39:34.600421    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-dhfxw/perf-node-gather-daemonset-t2h8w" event={"ID":"5654e6d2-2e23-4398-b820-21340ddcdeb8","Type":"ContainerStarted","Data":"21f5fec4741ff551caef2862778b12dc76720be062bfa990c8709903f606bc3e"}
Oct 30 01:39:34.600607 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:39:34.600557    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-dhfxw/perf-node-gather-daemonset-t2h8w"
Oct 30 01:39:34.644225 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:39:34.644163    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-dhfxw/perf-node-gather-daemonset-t2h8w" podStartSLOduration=1.644149423 podStartE2EDuration="1.644149423s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 01:39:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:39:34.643598479 +0000 UTC m=+4070.260031359" watchObservedRunningTime="2026-10-30 01:39:34.644149423 +0000 UTC m=+4070.260582302"
Oct 30 01:39:35.622331 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:39:35.622301    5002 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-hxbpg_4320a582-7ff0-456a-aa56-800a11ea6333/node-exporter/0.log"
Oct 30 01:39:35.745111 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:39:35.745081    5002 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-hxbpg_4320a582-7ff0-456a-aa56-800a11ea6333/kube-rbac-proxy/0.log"
Oct 30 01:39:35.868099 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:39:35.868068    5002 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-hxbpg_4320a582-7ff0-456a-aa56-800a11ea6333/init-textfile/0.log"
Oct 30 01:39:40.432096 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:39:40.432064    5002 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-86c478bb88-zqdbk_d7c5e976-596e-4cb5-81a9-b16262b3c85b/networking-console-plugin/0.log"
Oct 30 01:39:40.616157 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:39:40.616128    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-dhfxw/perf-node-gather-daemonset-t2h8w"
Oct 30 01:39:41.498653 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:39:41.498622    5002 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-g7c2w_5dc56e7e-f3c1-43eb-a21b-8be6bb4706cf/console-operator/1.log"
Oct 30 01:39:41.502826 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:39:41.502801    5002 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-g7c2w_5dc56e7e-f3c1-43eb-a21b-8be6bb4706cf/console-operator/2.log"
Oct 30 01:39:42.860250 ip-10-0-128-7 kubenswrapper[5002]: E1030 01:39:42.860213    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbjdq" containerName="extractor"
Oct 30 01:39:44.067459 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:39:44.067425    5002 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-58c9f5f46d-246g2_cea1b649-9c31-4434-9448-7f6652496830/volume-data-source-validator/0.log"
Oct 30 01:39:47.049955 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:39:47.049918    5002 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-btj6v_251cf359-f9f5-44ae-8ba9-42bf0cabb1ab/dns/0.log"
Oct 30 01:39:47.176681 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:39:47.176649    5002 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-btj6v_251cf359-f9f5-44ae-8ba9-42bf0cabb1ab/kube-rbac-proxy/0.log"
Oct 30 01:39:47.305120 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:39:47.305045    5002 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-6gjtz_078fd0e8-1393-45d0-baee-bbcd88095b2b/dns-node-resolver/0.log"
Oct 30 01:39:48.760492 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:39:48.760455    5002 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-j5x7c_48f3cd2a-244d-4849-888b-e4b9abd0ce9c/discovery/0.log"
Oct 30 01:39:49.057755 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:39:49.057668    5002 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7d9897b54c-2jzrh_49e9b6e9-fa30-4ea2-82e9-69e9e4ecab93/istio-proxy/0.log"
Oct 30 01:39:49.190979 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:39:49.190940    5002 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-8b6d6_14285dfa-dd38-490e-a486-a1fcfc3d0214/node-ca/0.log"
Oct 30 01:39:49.339127 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:39:49.339042    5002 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7d9897b54c-hb9g9_9224697d-3f44-4128-82de-642f6cdda3e3/istio-proxy/0.log"
Oct 30 01:39:49.562522 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:39:49.562490    5002 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7d9897b54c-pdgh4_c341e49a-5e59-414d-a432-e338bef60081/istio-proxy/0.log"
Oct 30 01:39:49.864521 ip-10-0-128-7 kubenswrapper[5002]: I1030 01:39:49.864478    5002 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-85f7b79cf7-h2tpp_a3d234b3-76d4-47d5-98a4-dafd83477b17/router/0.log"